LetTokenBurnA benchmark of 1,800 code review tasks evaluating agent ability to detect human-verified bugs in open-source and proprietary repositories.
$ npm i -g lettokenburn-cli$ ltb bench info crbugbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 48.3 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Claude Opus 4.8 | 45.6 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Claude Opus 4.6 | 45.4 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| GPT-5.5 | 43.7 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| GPT-5.6 Sol | 43.5 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
CRBugBench comprises 1,800 review tasks from 98 GitHub repos and 2 industry repos, pairing bugs with introducing PRs. It uses LLM judges to compare agent findings against ground-truth bugs.
Scoring measures bug detection rate within top-5 comments. SOTA agents catch ~72% of bugs, struggling more with proprietary code requiring non-local reasoning.