LetTokenBurnEnd2End-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
A multilingual benchmark evaluating coding agents' ability to build complete, installable software repositories end-to-end from natural-language specifications and empty workspaces.
$ npm i -g lettokenburn-cli$ ltb bench info e2e-swe
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 67.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Claude Fable 5 | 65.3 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| GPT-5.6 Sol | 61.2 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Claude Opus 4.8 | 51.9 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Muse Spark 1.3 | 45.2 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Claude Sonnet 5 | 43.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
E2E-SWE consists of 186 tasks spanning 11 programming languages. Each task is constructed by a human engineer collaborating with an LLM to define user-facing functionality via hidden tests while leaving internal architecture open for the agent.
Tasks are evaluated using pass@1, measuring the percentage of fully resolved repositories in one attempt across frontier models (scores range from 11.7% to 67.7%). The benchmark emphasizes solvability and system-level reasoning over localized editing.