LetTokenBurnIncident-Arena: Getting agents to the last nine of reliability
A human-built benchmark evaluating AI coding agents on production incident response tasks using real-world open-source software deployed in ephemeral Kubernetes clusters.
$ npm i -g lettokenburn-cli$ ltb bench info incident-arena
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-6 Astra | 0.6 | codex | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Opus 5.5 | 0.5 | claude code | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Fable 5.1 | 0.4 | claude code | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-6 Sol | 0.4 | codex | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6 | 0.4 | codex | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Opus 5 | 0.4 | claude code | Independent | medium | Sep 30, 2026 | arxiv.org |
| GLM-5.3 | 0.3 | claude code | Independent | medium | Sep 30, 2026 | arxiv.org |
| Grok-4.7 | 0.3 | grok build | Independent | medium | Sep 30, 2026 | arxiv.org |
| Muse Spark 1.3 | 0.3 | muse code | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Content & Construction: Incident-Arena consists of 20 carefully selected, human-authored tasks grounded in live-deployed open-source applications (e.g., Saleor, FrappeERP) and a custom Slack substrate. Each task deploys the application to an ephemeral Kubernetes cluster, injects parameterizable faults across configuration/runtime/image layers, and applies sustained load profiles.
Scoring & Findings: The benchmark utilizes a novel LLM-free dual-gate verifier measuring 'Outcome Gate' (recovery under traffic) and 'Safety Gate' (durability/data preservation). Across 3 application substrates with ~4.0M tokens per trial, the best model configuration achieved a Pass@1 score of 64.3%, while failures ranged from diagnosis errors to unsafe regressions.