LetTokenBurn
Sign in
Auto86%Harbor: pendingSafety & AlignmentAgentic / Tool UseCoding

Live-Harness-Bench

Live-Harness-Bench: A live and extensible benchmark of reproducible harness bugs in agentic systems

A scalable benchmark containing 200 executable harness bug instances collected from open-source agentic systems to evaluate software agents' ability to repair real-world failures.

PaperRepo

How to run

ltb · live-harness-bench
$ npm i -g lettokenburn-cli
$ ltb bench info live-harness-bench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

3 sourced entries
HarnessSourceConfidenceLink
DeepSeek V3.213.8live-harness-benchIndependentmediumSep 29, 2026arxiv.org
GPT-4.1-mini9.3live-harness-benchIndependentmediumSep 29, 2026arxiv.org
Kimi K2.58.4live-harness-benchIndependentmediumSep 29, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Content & Construction: Live-Harness-Bench is constructed by applying AgentBug-Smith, an automated multi-agent system that continuously discovers and reproduces real-world harness bugs. It currently contains 200 reproducible instances covering diverse components like context management and orchestration.

Scoring & Findings: The benchmark evaluates state-of-the-art software agents on harness bug repair tasks. Key findings reveal limited capabilities in current models, with AgentBug-Smith achieving significantly higher reproduction success rates (10.67% - 27.56%) compared to general-purpose techniques.