LetTokenBurn
Sign in
Auto86%Harbor: pendingCodingAgentic / Tool Use

ParanoiaEval

ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding

A benchmark of 200 evidence-controlled repository-level task pairs evaluating coding agents' risk-treatment capabilities and unnecessary defensive work.

PaperRepo

How to run

ltb · paranoiaeval
$ npm i -g lettokenburn-cli
$ ltb bench info paranoiaeval

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

7 sourced entries
HarnessSourceConfidenceLink
Claude Sonnet 4.611.2N/AIndependentmediumOct 6, 2026arxiv.org
Claude Haiku 4.517.7N/AIndependentmediumOct 6, 2026arxiv.org
5.6-Terra18.2N/AIndependentmediumOct 6, 2026arxiv.org
5.6-Luna19.2N/AIndependentmediumOct 6, 2026arxiv.org
5.6-Sol21.0N/AIndependentmediumOct 6, 2026arxiv.org
Claude Sonnet 522.2N/AIndependentmediumOct 6, 2026arxiv.org
Claude Opus 558.7N/AIndependentmediumOct 6, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

ParanoiaEval comprises 200 task pairs from 50 repositories, grounded in the ATMA risk management framework. Each pair differs only by treatment-defining evidence to isolate unnecessary defensive behaviors.

It uses a human-calibrated agentic judge to measure risk-treatment violations and evidence responsiveness. Experiments reveal unnecessary risk treatment occurs in 11.2%–58.7% of runs, decoupled from task capability.