LetTokenBurnSpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
A benchmark of 291 tasks evaluating if coding agents can construct faithful validation harnesses (checks and scenarios) from natural language specifications in real-world repositories.
$ npm i -g lettokenburn-cli$ ltb bench info specguardbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5.6 Sol | 81.8 | codex | Independent | medium | Oct 12, 2026 | arxiv.org |
| GLM-5.3 | 74.2 | opencode | Independent | medium | Oct 12, 2026 | arxiv.org |
| DeepSeek V4 Pro | 66.7 | opencode | Independent | medium | Oct 12, 2026 | arxiv.org |
| Claude Opus 5 | 65.3 | claude code | Independent | medium | Oct 12, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
SpecGuardBench comprises 291 tasks from 13 real-world repositories, covering 8 verification logic types. Agents must generate executable checks and test scenarios based on natural language specs.
Scoring measures faithfulness (accepting valid/rejecting invalid code) and thoroughness. Best agents achieved <63% faithfulness, struggling to decouple checks from reference implementations.