LetTokenBurn
Sign in
Auto86%Harbor: pendingCodingAgentic / Tool Use

HealBench

Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrails

A benchmark of 265 runtime errors from 18 real-world repositories evaluating LLM agents' ability to heal crashes using cross-file context and live state.

PaperRepo

How to run

ltb · healbench
$ npm i -g lettokenburn-cli
$ ltb bench info healbench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

3 sourced entries
HarnessSourceConfidenceLink
GPT-5.6-terra38.1codexIndependentmediumSep 30, 2026arxiv.org
GLM-5.232.8mini-swe-agentIndependentmediumSep 30, 2026arxiv.org
DeepSeek V4 Flash20.0codexIndependentmediumSep 30, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Content & Construction: HealBench is a repository-level benchmark containing 265 runtime errors collected from executable tasks in SWE-Bench, SWE-Bench Pro, and R2E-Gym. It spans a median of 37 files per instance and provides a unified framework allowing agents to explore repositories, inspect live state, and submit healing code.

Scoring & Key Findings: The benchmark evaluates execution completion (best: 38.11%) and target-test success (28.68%). It also introduces HealGuard for safety analysis; while detecting all unsafe cases in controlled tests, it flags 17.4% of passing executions where healing state changes may reach protected operations.