LetTokenBurnProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?
A black-box benchmark with 80 cross-component vulnerability-exploitation environments across 11 protocol domains, measuring LLM agents' ability to exploit protocol inconsistencies without source code access.
$ npm i -g lettokenburn-cli$ ltb bench info protoarena
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 4.8 | 48.8 | openhands | Independent | medium | Oct 13, 2026 | arxiv.org |
| Kimi K3 | 37.5 | openhands | Independent | medium | Oct 13, 2026 | arxiv.org |
| DeepSeek V4 Pro | 35.0 | openhands | Independent | medium | Oct 13, 2026 | arxiv.org |
| GLM-5.3 | 32.5 | openhands | Independent | medium | Oct 13, 2026 | arxiv.org |
| Qwen3.8 Max | 26.3 | openhands | Independent | medium | Oct 13, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
ProtoArena comprises 80 controlled environments spanning 11 protocol domains and 7 attack categories. Agents interact with running systems via scoped interfaces to achieve effect-level objectives without target source code access.
Scoring uses a hybrid verifier combining rule-based checks and LLM review of traffic/logs. Claude Opus 4.8 achieved the highest success rate of 48.8% under guided conditions and 12.5% in black-box settings.