Open infrastructure for building cloud software factories
Joined June 2021
- Tweets3.3K
- Following3
- Followers59.2K
- Likes3.9K
Pinned Tweet
Introducing Scorers: Agents that grade your agents.
Use LLM-as-a-judge to grade past coding agent sessions on:
• Quality
• Efficiency
• Compliance
• Or any custom dimension
Scores feed into performance measurements and automatic self-improvement for software factories
Warp retweeted
We made a skill for assessing the risk of merging a PR: /merge-risk
This addresses the code review bottleneck by helping devs trust the agent for lower risk changes.
It grades risk across 5 categories: alignment, impact, verification, rollback cost, and error detection.
- Alignment: does code follow the PR description and related tickets?
- Impact: what is the blast radius, and does it touch sensitive code paths?
- Verification: does the change include end-to-end verification? This means computer use, screen recordings, before/after screenshots... going beyond unit test coverage.
- Rollback cost: once it's shipped, how trivial is it to revert? @mattpocockuk calls this a "two-way door" vs. a "one-way door."
- Error detection: are errors reported, logged, or simply swallowed?
Try it out:
> npx skills add warpdotdev/common-skills --skill merge-risk
The real reason for software factories? Bringing together all of your conversation data. Don't leave it locked away on your laptop.
We should stop keeping all of our agent conversations local. They should be stored in a shared place.
This lets you do some powerful things:
- Schedule self-improvement agents to review runs and pull out failure modes worth fixing
- Replay past conversations against different models for benchmarking
I met a team of devs zipping up Claude and Codex sessions from their computers and shipping them to a shared S3 bucket. That's... a start 😄 But really, all agent sessions should happen on a single source (like a cloud runner) so that conversation traces are stored together by default.
It's why Warp Factories uses cloud sandboxes and makes sessions available via API. We also added an inbox view to see which conversations need your attention, which is quite convenient:
Interested in trying Warp Factories? Request early access: warp.dev/get-started 🔖
Warp retweeted
Building Evals on Production Data: 🦄 AI That Works #79 xitter.cf/i/broadcasts/1oJMvNZWv…
Warp retweeted
At @aiDotEngineer, I presented our skill called /skill-doctor. It's a skill to improve your agent skills.
It aggregates your past agent sessions, grades their quality, and propose improvements to fix common failures.
Great introduction to software factories if you've been wondering how they work
Here's my take on what a software factory really is
This is my talk from @aiDotEngineer NYC, and I decided to record a version for everyone to watch on X.
tl;dr - software engineering is becoming factory engineering, and there's new work (especially factory benchmarking) that engineers should start focusing on.
Warp retweeted
Getting software factories working is going to be one of the more interesting software engineering challenges in the next few years.
Software engineering is becoming factory engineering. Organizations that can make their factories robust, dependable, and self-improving will be able to ship more at better cost and have a competitive advantage.
If you're interested in software factories, but you're not sure where to start, this is a good read:
Until next time @aiDotEngineer!
We benchmarked some agent orchestration pipelines at @aiDotEngineer. Which would you pick?
ALT Five people standing in front of a Warp Factories booth. A "Foreman" label at the top has dashed red arrows pointing down to each person, who are labeled "Triage," "Spec," "Implementation," "Verification," and "Code review."
ALT Two people at a Warp Factories booth. One holds a laptop showing an analytics dashboard and a small purple plush toy, and the other holds up a Warp sticker. Above them, labels reading "Worker" and "Code reviewer" have arrows pointing back and forth between them.
Warp retweeted
An interesting engineering challenge this next year will be closing the development loop we've started.
We have small loops already (code review, automatic CI fixes, computer use verification), but we need to close the larger loop: let agents suggest what to build next.
Warp retweeted
Software factories can be a real culture shift. The biggest change for us was moving from individual agents on everyone's machines to one central agent setup (a factory) that we can build as a team.
I wrote some learnings on how to navigate that.
If you're building a software factory, this may be your north star:
3. Have agents suggest roadmap items from customer feedback.
For example, you can use @GrainHQ to record transcripts on customer discovery calls, and have coding agents pull out insights.
Here, an agent goes through prior customer interviews with a scheduled automation and looks for commonalities in where customers get stuck. From there, you can fully close the loop by connecting your taskboard via MCP to drive the next agent task.
More on integrations: docs.warp.dev/factories/conn… 🔖
ALT A digital workspace displays a new request to create a Notion item based on customer feedback, listing various topics and suggestions.
If you want to set up these automations, Warp Factories is a good place to start: warp.dev/get-started 🔖
Warp retweeted
We've found a few key pieces to successful software factories, both from our internal testing and from helping companies deploy their own:
Factories-as-code: you should configure every part of the factory (agent orchestration, automations, permissions, sandbox configuration, etc) as code. This enables testing of different factory configurations to see which are most efficient, highest quality, etc.
Multi-model & multi-harness: you should make sure that your factories are able to use the latest models, both frontier and open-weight, and use different coding agent harnesses like Claude Code and Codex.
Data ownership: you should make sure you store and own all data coming out of your factory. This is the raw material for improving its operations.
Join @BHolmesDev and @sergical to build a software factory that fixes its own bugs
I'll be joining @sergical at @sentry on October 22nd to talk about connecting error monitors to coding agents!
It's how we've closed the loop in our own software factory to have agents assign themselves work automatically. The flow we're using right now:
- Connect to an alert webhook like Sentry
- When errors happen, start a cloud agent run with the alert and the repo as context
- Have the agent ping humans for code review (or better: with an HTML summary of the problem and the solution)
- Merge and keep monitoring
This closes the loop on reducing software bugs. We'll show you how to deploy it in-code yourself!
ALT Two cycles shown side by side. On the left, "Prompting agents yourself," a human tracks bugs by hand. The loop goes from new work to align, implement, and deploy, then stops at a red question mark labeled "No agent watching the deploy." On the right, "Closing the loop," an "Error monitor watches for bugs" step after deploy sends a green arrow back to "New work assigned," so the cycle restarts on its own.