LetTokenBurnEvidence from Nonpublic Vulnerabilities and Subsequent Attacks
Evaluates AI agents on exploit generation and vulnerability repair across 254 nonpublic tasks in five software environments using deterministic graders.
$ npm i -g lettokenburn-cli$ ltb bench info nonpublic-vulnerabilities-benchmark
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 0.6 | claude code | Independent | medium | Oct 5, 2026 | arxiv.org |
| GLM-5.3 | 0.6 | opencode | Independent | medium | Oct 5, 2026 | arxiv.org |
| Qwen3.8 Max 0902 | 0.5 | opencode | Independent | medium | Oct 5, 2026 | arxiv.org |
| Kimi K3 | 0.5 | opencode | Independent | medium | Oct 5, 2026 | arxiv.org |
| Daybreak Blue alias | 0.1 | codex | Independent | medium | Oct 5, 2026 | arxiv.org |
| GPT-5.6 Sol | 0.1 | codex | Independent | medium | Oct 5, 2026 | arxiv.org |
| Gemini 3.8 Flash | 0.0 | opencode | Independent | medium | Oct 5, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The benchmark comprises 254 execution records across five nonpublic software environments, including privately disclosed unpatched vulnerabilities and cryptographic challenges. It pairs attack and repair tasks to measure capability gaps.
Scoring uses researcher-developed deterministic graders with partial-credit scores in [0,1]. Findings show repair scores exceed attack scores in two environments but fall below in three, with significant subsequent attack success rates.