Self-reported vs reproduced: why BenchHub exists, and how to reproduce a benchmark in 10 minutes

Most benchmark numbers you see in a launch post were measured by the company that trained the model, on a harness it chose, sometimes on a subset it chose. That is not a scandal; it is just one data point. BenchHub exists to put a second and a third data point next to it, and to make producing one cheap enough that anyone with a coding-agent subscription can do it.

Self-reported, independent, reproduced

Every score in the registry carries one of three labels, and a link to where it came from:

• Self-reported: published by the model vendor in a model card, blog post or technical report.
• Independent: measured by a third party, such as an official benchmark leaderboard or an evaluation lab.
• Reproduced: aggregated from runs that people pushed with the ltb CLI, pooled per model, benchmark and harness into a median and a run count.

Right now the registry holds 489 self-reported and 5,550 independent scores, and 156 model and benchmark pairs have both. The reproduced column is the thin one: 0 reproduced scores in the Oct 13, 2026 snapshot. That is the gap this post is about closing.

What is in the registry

6,820

benchmarks indexed

718

runnable with Harbor

386

verified records

6,039

scores, each with a source URL

Each benchmark has a page with its paper, dataset, license, metric, sample count, capability axes and every known score. 718 of them can be turned into Harbor tasks by the CLI today, either through a generated adapter (rows pulled from Hugging Face, one task per row) or because the benchmark already ships Harbor tasks. Browse them in the directory or see the model by benchmark grid on the leaderboard.

Reproduce a benchmark in about ten minutes

The walkthrough uses MMLU-Pro because its license allows redistribution, so the generated tasks include reference solutions and the oracle check works. A 20-task first pass on a machine with Docker already running takes roughly ten minutes, most of it spent by your agent. Larger samples take proportionally longer.

0. Install

You need Node.js 20 or newer, Docker, and Harbor (a Python tool, installed with uv).

$ npm i -g lettokenburn-cli
$ uv tool install harbor
$ ltb --version && harbor --version && docker info > /dev/null && echo ready

1. Discover

search and info read the public static registry; no account is needed. info prints the facts, license, top known scores and the exact commands for that benchmark.

$ ltb bench search mmlu
$ ltb bench search --axis coding --harbor-only --limit 10
$ ltb bench info mmlu-pro

2. Pull

pull samples rows deterministically (same --seed, same rows) and writes one Harbor task directory per row: an instruction, a Dockerfile, a verifier and, when the license allows, a reference solution.

$ ltb bench pull mmlu-pro --n 20
# -> ./mmlu-pro/<slug>-<row>/  task.toml  instruction.md  environment/  tests/  solution/

3. Sanity check with the oracle

Before spending any tokens, run the oracle agent. It applies the reference solution to every task, so it should score 100. If it does not, the problem is in the task packaging and you have found a bug worth reporting, not a model weakness.

$ ltb bench run ./mmlu-pro -a oracle
# expected: score 100. Anything lower means the packaging is broken, not the model.

4. Run with a real agent

Now point Harbor at the agent and model you already pay for. Claude Code, Codex, OpenHands and Terminus are supported; the model is passed through with -m. Your keys and your subscription stay on your machine.

$ ltb bench run ./mmlu-pro -a claude-code -m anthropic/claude-opus-4-8 -n 4

When the job finishes, the CLI writes ltb-run-summary.json: the score (mean reward × 100), task count, attempts, per-task rewards, Harbor and CLI versions, and a SHA-256 of the results file.

5. Push

$ ltb bench push ./jobs --notes "first pass, 20 tasks, seed 7"

Only that summary is uploaded; transcripts and model outputs are not. Anonymous pushes are allowed; with an API key from your profile the run is attributed to you. The command prints a link to the run page, and the run joins the Reproduced column for that model and benchmark.

Honest limits

1. 1Metadata is collected automatically. Benchmark records are gathered from 12 public channels (paper indexes, Hugging Face, GitHub, evaluation frameworks and leaderboards) and checked by an LLM reviewer. Fields such as license, sample count or release date can be missing or wrong.
2. 2Quality tiers say how much to trust a record. Each record is Verified (reviewed, or confirmed by independent channels and mostly complete), Auto (published by the reviewer, reasonably complete) or Incomplete, and the page lists what is missing. Only 386 of 6,820 records are verified today.
3. 3Community runs are just starting. The reproduced column is close to empty, which is exactly why every run counts.
4. 4Scores always carry a source. Missing cells are never estimated. If a cell disagrees with its source, email [email protected] with the benchmark, model, value and URL.

Why a subscription, and why now

Flat-rate coding-agent plans reset every cycle whether you use them or not. Spending some of that capacity on a reproduction produces a public, sourced data point that did not exist before. It is the most useful thing spare tokens can do, and it is the reason this site is called LetTokenBurn.