Warp

Every company selling coding agents in 2026 promises a productivity gain and almost none of them will tell you how to measure it. That is the gap Warp has built into.

Its factories are fleets of agents defined in a configuration file, triggered from the tools your team already uses, and scored on every run against your own codebase.

The measurement is not a dashboard bolted on afterwards. It is the product, and the automation is what the measurement makes safe to trust.

Warp: From Terminal To Software Factory

• Factories as code — repos, agents, models, permissions and checkpoints in a factory.yaml checked into your repository.
• Evals on your own work, plus benchmarks comparing models on your tasks rather than public leaderboards.
• Self-improvement loops: observer agents open pull requests against the factory config itself.
• Open at four layers — any MCP-capable agent, any model, your compute or theirs, your data storage.
• 800,000-plus developers on the terminal, with Factories in closed early access.

What Is A Software Factory As Code?

Warp's own definition is precise: a fleet of agents wired to your development lifecycle, triggered by an issue, a Slack message or a schedule, and coordinated from triage through review to a mergeable pull request. It is defined as code in a factory.yaml plus supporting agent files, checked into your repository like any other configuration.

The as-code part is what separates this from a chat interface with tools attached.

A configuration file in version control gets reviewed, diffed, reverted and audited. When an agent does something unexpected, the definition that produced it is in git with an author and a date, and that is the difference between a system you can operate and one you can only hope about.

It also makes the automation portable between repositories and teams, which a UI-configured agent never is.

The design choice underneath is the one worth noticing: work arrives from Slack, Teams, Linear, Jira, GitHub and GitLab, and status flows back to wherever it came from. Nobody has to adopt a new interface, which is the most reliable predictor of whether engineering tooling survives its pilot — the same insight behind Conductor keeping agent orchestration inside the repository.

A factory in version control gets reviewed, diffed and reverted. A UI-configured agent gets none of those, which is why it never scales past one team.

The Twenty To Thirty Percent Answer

Warp's FAQ states that most organisations start with around 20 to 30 percent of pull requests fully automated, beginning with simple tasks, and that this rises as models improve and the factory self-improves.

That is a remarkable thing to put on a marketing site, and it is the single strongest credibility signal on the page.

Every competing vendor has an incentive to imply a far higher number, and any engineering leader who has actually deployed coding agents knows that 20 to 30 percent of pull requests — dependency bumps, flaky test triage, small well-specified fixes — is both achievable and genuinely valuable.

It also sets the right expectation internally. A team told to expect 80 percent automation concludes the tool failed; a team told to expect a quarter of routine pull requests measures a real win.

The recommended starting point follows the same honesty — a single low-risk workflow like dependency bumps or flaky test triage before expanding. That is exactly how this should be adopted, and exactly what most organisations will be tempted to skip.

Evals, Benchmarks And Self-Improvement Loops

Three measurement mechanisms run alongside the automation: evals scoring runs against your own work, benchmarks comparing models on your tasks with a cost figure attached to each, and self-improvement loops where memory updates and prompt changes are tracked with their effect on score.

The benchmark-on-your-own-tasks piece is the most valuable and the least common.

Public model leaderboards measure performance on public problems. Your codebase has its own conventions, its own framework choices and its own peculiarities, and a model that leads a leaderboard can be worse on your work than one that does not.

Attaching a cost to each benchmarked run turns that into an economic decision rather than a quality one. Warp's own case study describes using this to cut cost per pull request substantially by choosing models per pipeline stage, which is precisely the optimisation nobody can perform without measurement.

The self-improvement loop closes it: observer agents score runs, then open pull requests against the factory configuration. Improvements arrive as reviewable changes rather than silent drift, which is the right architecture — the same discipline that makes evaluation tools like Promptfoo useful rather than decorative.

• Start with one low-risk workflow, not broad coverage.
• Run benchmarks on your own tasks before choosing a model.
• Read cost per pull request as an economic metric, not a vanity one.
• Review observer-agent pull requests like any other config change.
• Set expectations at a quarter of routine pull requests, not most of them.

Why Warp Calls Itself Open Infrastructure

Openness is claimed at four distinct layers: any MCP-capable coding agent including Claude Code, Codex and Cursor; any model, frontier or open-weight, chosen per pipeline stage; Warp's cloud or self-hosted in your own VPC; and data storage described as pluggable, zero-retention, in your VPC.

Naming four layers separately is more useful than a general no-lock-in claim, because each one fails differently.

Agent-level openness means the platform is not betting on which coding harness wins, which matters in a field where the leader changes every few months. Model-level openness protects against a provider's pricing or deprecation schedule.

Compute and data openness are the two that decide enterprise procurement. Self-hosting inside your own VPC under your existing retention rules removes the single largest objection to agentic coding — that your source code is being processed somewhere you cannot see.

The site is explicit that Factories works without the Warp terminal, and that nobody at your company has to adopt the terminal to use it. Separating those products rather than using one to sell the other is a straightforward, credible position.

Governance And Proving The Work

Two features address the thing that actually worries engineering leaders about agent fleets, which is not capability but control.

The first is a single control plane showing every agent run across every source — orchestration, Claude Code, Slack, Codex — with the model used, the initiator and the elapsed time, plus cost broken into compute, platform and inference.

That matters because agentic coding sprawls invisibly. Different teams adopt different harnesses, spend appears across several bills, and nobody can answer what the organisation is spending or what it got. Centrally configured agent access and centrally managed permissions are the antidote.

The second is verification through computer use: every factory agent captures a screenshot or video so you can check its work before shipping a pull request.

That is a better answer to agent trust than any amount of confidence scoring. An agent claiming it fixed a page is an assertion; an agent showing you the page working is evidence, and it moves review from reading a diff to checking an outcome. Release gates in tools like Testsigma make the same move from assertion to evidence.

Running Agents Against A Real Codebase

Agent adoption fails on trust and on cost, usually in that order, and both are measurement problems rather than technology problems.

This sequence follows the order Warp's own documentation recommends, which is the honest one.

1. Pick one low-risk workflow — dependency bumps or flaky test triage.
2. Define it in factory.yaml and check it into the repository.
3. Wire triggers to the tools your team already uses, not a new interface.
4. Run evals on your own work before trusting any output.
5. Benchmark two or three models on your tasks and compare cost per result.
6. Expand coverage only after the first workflow runs unattended for a month.

Warp: Platform And Adoption