← All writing

AI Engineering

Strict Pass Rate: Stripe Evals Show the Last-Mile Gap

Mani G 3 min read

Checklist with six green checks and one red cross beside a credit card and a robot arm, showing a coding agent missing one required check

Stripe published a benchmark on November 19 that I think every API company should copy. It is called Stripe evals, and the number worth staring at is the strict pass rate. Coding agents met 78.1% of Stripe's integration criteria on average. Count only the runs that passed every required check, and that drops to 29.4%.

Same agents. Same runs. Two very different stories.

What a strict pass rate actually measures

Most benchmarks hand out partial credit. Stripe's headline score works that way too. Each scenario has a rubric of MUST criteria (required for a valid integration) and SHOULD criteria (recommendations), borrowed from the RFC 2119 wording you see in spec documents. A run that hits 6 of 8 criteria scores 75%.

The strict pass rate is binary. A run passes only if every MUST criterion passes. Miss one and the case is a zero.

That sounds harsh until you remember how payments work. Nobody gets partly paid by a checkout flow that is 90% right. Stripe itself says the strict number "is more indicative of developer success."

How the benchmark works

The setup is smarter than the usual "here is a task, run the tests" approach:

  • 69 scenarios: 24 one-time payments, 26 subscriptions and 19 Connect (marketplace and platform) cases.
  • 14 model and harness pairs across Claude Code, Codex and OpenCode.
  • Every case runs twice with Stripe's agent plugin and twice without, so 276 runs per agent.
  • A second agent plays the developer. It holds a private list of two to four needs (six at most) and turns them into follow-up questions one at a time, the way a real person does when they only half know what they want.
  • GPT-6.1 Sol acts as both roleplayer and judge. Stripe ran a probe with Opus as the judge to check for bias and says the scores moved in lockstep, with Opus simply more lenient across the board.

Each criterion gets its own judge call, so one bad call doesn't bleed into the others.

The last-mile problem in one example

Stripe's own example is the one I'd show any team shipping agent-written payment code. In a fulfillment scenario, Opus 5.5 checked only that PaymentIntent.status === 'succeeded'. It never checked whether the order was still eligible for fulfillment. So an order could ship after a refund or a dispute.

That code would look fine in review. It would pass a happy-path test. It would also give away free stuff.

Stripe sums up the pattern plainly: many models build "almost correct implementations and have difficulty with the last mile."

The plugin helps, but read the fine print

All 14 configurations scored higher with the Stripe plugin (skills plus an MCP server with docs search and an implementation planner), by roughly 3 to 8 points. Web searches dropped 96%. Outdated parameters in API calls fell from 7% to 1.3%. Fair play to them.

The strict numbers stay modest, though. The best one in the table is GPT-6 Astra on Codex, which went from 41.9% to 52.9% with the plugin. So the top setup still misses a required check in close to half its runs.

Stripe also lists what the benchmark doesn't measure. It grades the specification, not execution, so "a well-shaped but non-functional integration can score full marks." It skips long-term maintenance too. I respect a vendor that publishes its own caveats.

Why the average is the wrong number

My take: for anything that moves money, the average score is a vanity metric. If you're picking an agent setup for billing or checkout work, ask for the strict number, or write your own MUST list and grade against it. Stripe released the harness, called Forge, so you can run the same kind of test on your own cases.

Picture an agent that nails 9 of 10 required checks on your refund flow. Would you merge it, or is 90% just a polite word for broken?