LetTokenBurn
Sign in
Auto86%Harbor: pendingAgentic / Tool UseInstruction Following

DelegationBench

DelegationBench: Measuring When AI Agents Should Ask Before Acting

A benchmark of 156 scenarios with matched pairs testing if AI agents correctly choose to act, ask permission, ask for info, or refuse based on context features like stakes and reversibility.

PaperRepo

How to run

ltb · delegationbench
$ npm i -g lettokenburn-cli
$ ltb bench info delegationbench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

10 sourced entries
HarnessSourceConfidenceLink
Gemini 3.5 Flash Lite73.4N/AIndependentmediumOct 4, 2026arxiv.org
Gemini 3.6 Flash71.9N/AIndependentmediumOct 4, 2026arxiv.org
Claude Sonnet 568.7N/AIndependentmediumOct 4, 2026arxiv.org
Claude Opus 4.865.5N/AIndependentmediumOct 4, 2026arxiv.org
GPT-5.6-terra60.4N/AIndependentmediumOct 4, 2026arxiv.org
GPT-5.6 Luna60.0N/AIndependentmediumOct 4, 2026arxiv.org
Qwen3.6 27B58.7N/AIndependentmediumOct 4, 2026arxiv.org
GPT-5.6 Sol58.1N/AIndependentmediumOct 4, 2026arxiv.org
GPT-oss 120B54.5N/AIndependentmediumOct 4, 2026arxiv.org
gpt-oss-20b50.6N/AIndependentmediumOct 4, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

DelegationBench contains 156 short scenarios where models choose from four responses (act, ask permission, ask info, refuse). Most scenarios form matched pairs varying single features like reversibility or stakes to test sensitivity.

Scoring evaluates agreement with human labels, revealing gaps between judgment and action. Key findings show simple rules often outperform models in responsiveness, and prompt phrasing significantly alters behavior.