LetTokenBurn
Sign in
Auto86%Harbor: pendingAgentic / Tool Use

ToolBottleneckBench

ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets

A benchmark measuring tool-agent performance under controlled pressures (catalog, menu, workflow, encoding) and fixed harness budgets using 36 synthetic enterprise task families.

PaperRepo

How to run

ltb · toolbottleneckbench
$ npm i -g lettokenburn-cli
$ ltb bench info toolbottleneckbench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

12 sourced entries
HarnessSourceConfidenceLink
Claude Opus 5.597.2toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
Claude Opus 594.4toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
Claude Sonnet 5.594.4toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
Claude Sonnet 593.1toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
GLM-5.369.8toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
Gemini 3.8 Flash64.9toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
Qwen3.7 Flash51.4toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
GLM-5.3 Flash37.5toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
DeepSeek V4 Pro 081331.9toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
Qwen3.8 Flash30.6toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
Claude Haiku 5.54.5toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org
GPT-6 Luna0.0toolbottleneckbenchIndependentmediumOct 15, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

ToolBottleneckBench evaluates tool agents on 36 synthetic enterprise task families across accounting, email, file, and calendar domains. It isolates performance impacts by varying single pressures like visible tool menus or registered catalogs while keeping harness budgets fixed.

Scoring uses an exact grader that records the first unmet obligation and any triggering harness limit. Results indicate that visible menu size significantly penalizes success due to truncation, while most failures end at harness limits rather than model errors.