LetTokenBurn
Sign in
Auto71%Harbor: pendingLong ContextAgentic / Tool Use

LongHarness Bench

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

A benchmark evaluating the effectiveness and efficiency of language model harnesses on four long-context tasks requiring diverse retrieval strategies and adaptive reasoning.

PaperRepoHomepage

How to run

ltb · longharness-bench
$ npm i -g lettokenburn-cli
$ ltb bench info longharness-bench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

5 sourced entries
HarnessSourceConfidenceLink
GPT-5.6 Sol60.0directIndependentmediumSep 29, 2026arxiv.org
Gemini 3.8 Flash10.0directIndependentmediumSep 29, 2026arxiv.org
GLM-5.39.5directIndependentmediumSep 29, 2026arxiv.org
Kimi K2.60.0directIndependentmediumSep 29, 2026arxiv.org
Qwen3.8 27B0.0directIndependentmediumSep 29, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Content & Construction: The benchmark comprises four diverse long-context tasks (Constraint Solving Search, Equivalent Program Pair Search, Program Execution Tracing, Outlier Memo Detection) that require challenging lexical/semantic search and multi-step adaptive reasoning. It evaluates frontier models combined with agentic harnesses like OpenCode, mini-swe-agent, RLM, and ReAct.

Scoring & Key Findings: Performance is measured by macro-average accuracy (best 68%) and computational cost per instance to distinguish efficiency tradeoffs. Results reveal that the same model exhibits markedly different efficiency under different harnesses, with some costing up to 12.4x more for similar accuracy.