LetTokenBurnTestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications
A benchmark of 165 open-source applications measuring agents' ability to generate strong test suites from natural-language specifications using mutation scores.
$ npm i -g lettokenburn-cli$ ltb bench info testsuitebench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 55.5 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| GPT-6 Astra | 50.1 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Gemini 3.8 Flash | 37.9 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| GPT-5.6 Sol | 37.7 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| DeepSeek V4 Flash | 28.6 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| GPT-5.6 Luna | 19.3 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
TestSuiteBench comprises 165 applications with detailed specifications, reference implementations, and requirement-targeted mutants. It evaluates coding agents on generating complete test suites solely from natural-language descriptions.
Scoring uses validity (tests passing reference) and mutation score (fraction of mutants killed). Frontier agents struggle, achieving mutation scores below 60%, revealing a gap where agents can build software they cannot verify.