LetTokenBurn
Sign in
Auto86%Harbor: pendingCodingAgentic / Tool Use

AgentSWE

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

A benchmark of 25 tasks measuring coding agents' ability to build, edit, and optimize other agents against natural-language requirements using held-out cases.

PaperRepo

How to run

ltb · agentswe
$ npm i -g lettokenburn-cli
$ ltb bench info agentswe

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

5 sourced entries
HarnessSourceConfidenceLink
DS-V4-Pro19.1N/AIndependentmediumOct 9, 2026arxiv.org
DS-V4-Flash15.1N/AIndependentmediumOct 9, 2026arxiv.org
GPT-5.6 Sol13.7N/AIndependentmediumOct 9, 2026arxiv.org
GPT-5.55.8N/AIndependentmediumOct 9, 2026arxiv.org
Qwen3.6 Max Preview2.3N/AIndependentmediumOct 9, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

AgentSWE comprises 25 commissions across Creation, Editing, and Optimization stages. Builders develop agents from requirements and illustrative cases, then submit them for evaluation on held-out cases.

Scoring uses requirement-derived rubrics with hard pass/fail gates or native objectives. Results show a significant gap between demonstrated performance and actual delivery, with strong builders averaging low scores on held-out tests.