HarnessSecurity-Bench
A benchmark of 23 coding tasks across five attack surfaces evaluating security mechanisms and utility in six coding agent harnesses.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info harnesssecurity-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
6 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Code | 54.2 | harnesssecurity-bench | Independent | medium | Oct 6, 2026 | arxiv.org |
| Codex CLI | 54.2 | harnesssecurity-bench | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini CLI | 54.2 | harnesssecurity-bench | Independent | medium | Oct 6, 2026 | arxiv.org |
| GitHub Copilot | 54.2 | harnesssecurity-bench | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gptme | 54.2 | harnesssecurity-bench | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen Code | 54.2 | harnesssecurity-bench | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Comprises 23 coding tasks in Python, JS, C#, C++, Go, and Ruby, augmented with malicious resources across five attack surfaces. It uses deterministic oracles to measure task utility and attack effects.
Evaluates nine security mechanisms across six harnesses via 2,500 trials. Key findings show auto-approve raises attack success to 95.6%, while isolation reduces it significantly but harms utility.