BenchHub user guide

BenchHub docs

BenchHub is the benchmark registry on LetTokenBurn. The loop is always the same: discover a benchmark, run it locally with your own subscription using the ltb CLI and Harbor, submit the reproduced score, and compare it with what was self-reported. This guide covers each step, how the registry is built, and where its data is still incomplete.

Browse benchmarks CLI quick reference Leaderboard Agent API

1. Discover
Each benchmark page lists paper, data, license, metric, known scores and a quality tier. Chapter 03.
2. Run it locally
ltb bench pull writes Harbor tasks; ltb bench run runs them with your agent. Chapter 04.
3. Submit
ltb bench push uploads a score summary, never model outputs. Chapter 05.
4. Compare
Your run joins the Reproduced column next to self-reported and independent scores. Chapter 06.

01. What BenchHub is 02. Core concepts 03. Discover a benchmark 04. Run it locally 05. Submit a reproduced score 06. Compare: leaderboard and graph 07. Data quality and corrections 08. Agent integration 09. FAQ

Chapter 01

What BenchHub is

BenchHub is a registry of AI benchmarks with one job: make any benchmark findable, runnable and independently reproducible. Think of it as a package index for evaluations rather than for code.

The loop

Find a benchmark in the directory: paper, dataset, license, metric, known scores, related benchmarks.
ltb bench pull turns it into standard Harbor task directories on your disk.
ltb bench run executes those tasks on your machine with the agent and model subscription you already pay for.
ltb bench push uploads a score summary. It shows up as a Reproduced score next to the vendor's own number.

What "Let Token Burn" means now

Most AI subscriptions come with quota that expires unused. BenchHub gives that quota a public purpose: burn it reproducing benchmarks, and the output is independent evaluation data that anybody can inspect. The platform never touches your API keys, never runs your jobs for you, and never resells tokens. Compute stays on your machine.

Who it is for

Researchers and students who need the data, the paper, the license and a working harness for a benchmark in one place.
Model and agent developers who want to see where a model has not been measured, and how far community reproductions sit from self-reported numbers.
Agent operators who have idle coding-agent capacity and would rather spend it on something with a public record.

Chapter 02

Core concepts

A handful of terms appear on every page. Once these are clear, the directory, the detail pages and the leaderboard read the same way.

Benchmark

One record per named evaluation set, identified by a lowercase slug derived from its name (MMLU-Pro → mmlu-pro, SWE-bench Verified → swe-bench-verified). Versions that differ materially get their own slug (terminal-bench-2). A record can carry the summary, capability axes, industries, task format, metric and scale, sample count, difficulty, saturation, paper, repo, dataset, license and Harbor status. These fields are collected automatically and not every record has all of them; the quality tier and the list of missing fields on each page tell you which (Chapter 07).

Capability axis

Each benchmark is tagged with one or more axes, primary axis first: Knowledge & Reasoning, Math, Coding, Agentic / Tool Use, Multimodal, Long Context, Instruction Following, Safety & Alignment, Abstract Reasoning, Multilingual. The directory filters by axis, the heatmap groups columns by primary axis, and the composite score averages inside an axis before weighting across axes.

Task format

How one item is answered and verified: multiple choice, short answer, open-ended QA, code generation, code repair, agentic environment, human preference, classification or metric-based generation. Format decides which verifier the Harbor adapter can use; multiple choice and short answer are checked by exact or normalized matching, SWE-style repair needs unit tests, open-ended QA needs a rubric.

Harbor task

A self-contained directory in the format used by Harbor (the runner behind Terminal-Bench 2.0): task.toml, instruction.md, environment/Dockerfile, tests/test.sh and optionally solution/. The agent reads the instruction inside a container, the verifier writes a reward. Chapter 04 shows the exact files ltb bench pull writes.

Run

One execution of a benchmark subset with one agent and one model. The CLI reads Harbor's per-task rewards, computes score = mean(reward) × 100, and stores the result as ltb-run-summary.json. That file, not the transcripts, is what gets pushed.

Confidence

Every score cell has a confidence of high, medium or low. Low is used for bulk-imported numbers whose source URL points to a leaderboard rather than the exact cell, and for anything a reviewer has not re-checked. The UI draws low-confidence cells pale with a dashed border. The number still links to its source; it is never hidden.

Three kinds of score

Self-reported

Published by the model vendor in a model card, launch blog or technical report. Useful, but the vendor chose the harness, the prompt and the subset.

Independent

Measured by a third party: an official benchmark leaderboard, Artificial Analysis, LMSYS, or a published reproduction in a paper.

Reproduced

Aggregated from runs that BenchHub users pushed with the CLI. Same benchmark, same model, same harness runs are pooled into median, mean, min, max and a run count.

The glyph appears inside heatmap cells. The gap between S and R on the same benchmark is the most useful thing on the site; it is also why the platform exists.

Composite score, in prose

The leaderboard shows a single composite per model. It is computed in the open, with the algorithm shared between the crawler and the website, and it works like this:

Keep live benchmarks. Only benchmarks with a dated score within 730 days of the newest score, and with at least 3 models scored, count. Otherwise old models measured only on old, easy benchmarks would outrank current ones.
Rank within each benchmark. A model's score on a benchmark becomes its percentile rank among all models scored there (100 = best, 0 = worst; lower-is-better metrics are inverted). Ranks keep an easy benchmark from outweighing a hard one.
Pick the best per benchmark. In any harness mode a model's best score across harnesses counts. In grouped mode only scores from one named harness are pooled, producing one row per model + harness.
Average inside each axis. Benchmarks are weighted equally within their primary axis, except saturated benchmarks which count at 0.5 because they no longer separate frontier models.
Weight across axes. Axis averages are combined with the configured weights: Knowledge & Reasoning 1, Math 1, Coding 1.5, Agentic / Tool Use 1.5, Multimodal 0.75, Long Context 0.75, Instruction Following 0.75, Safety & Alignment 0.5, Abstract Reasoning 0.75, Multilingual 0.5. Coding and agentic axes are deliberately heavier; safety and multilingual lighter.
Coverage threshold. A model must cover at least 5 distinct benchmarks across at least 3 axes. Below that the composite is shown as N/A and the row is marked "insufficient". Missing cells are never estimated or back-filled.

Weights are configuration, not truth. The leaderboard prints the active weights under the table so you can decide whether you agree with them.

Chapter 03

Discover a benchmark

The directory and the detail pages are built from a static snapshot generated by the crawler, so they load instantly. Every score links to its source; benchmark metadata is collected automatically and some fields may be missing, which each record states openly.

Directory filters

/benchmarks/ offers full-text search over name, slug and summary, plus filters for capability axis, industry, task format, difficulty, saturation, license (redistributable vs link-only) and Harbor status. Sort by release date, sample count or number of known scores. Filters are reflected in the URL so a filtered view can be shared.

Harbor status badge

Harbor adapter: the CLI can generate task directories from the source dataset.
Harbor native: the benchmark already ships Harbor tasks; the CLI points at them or clones the repo.
No Harbor adapter: index-only for now. ltb bench info still prints the links and any external harness instructions.

Anatomy of a detail page (/b/<slug>/)

Header: name, one-line summary, quality tier, axis and industry badges, Harbor status, difficulty, saturation.
Facts: metric and scale, task format, sample count, modalities, languages, release date, scoring method.
Links: paper (arXiv), repo, dataset or Hugging Face id, official leaderboard, homepage.
License: SPDX id when known, link to the license text, and whether redistribution is allowed.

How to run: the exact ltb bench commands for this benchmark, adapted to its Harbor status.
Scores: every known model score with source type, harness, confidence and a link to the source.
Reproduced by the community: live aggregates from pushed runs, fetched from the API.
Related benchmarks: edges from the graph with their relation type.

Redistributable vs link-only

A benchmark is redistributable when its license clearly allows mirroring task data (MIT, Apache-2.0, CC-BY and similar). Everything else, including an unknown license, is treated as link-only: BenchHub stores metadata and links, and the CLI fetches rows from the original host on your machine at pull time, for local evaluation only. The pull command prints a notice and does not write a solution/ directory for link-only benchmarks.

Citation

Each page shows the paper title, authors, year and venue when known. If you publish results obtained through BenchHub, cite the benchmark's original authors, not this site. Mentioning that the run was reproduced via Harbor tasks generated by ltb is welcome but secondary.

Chapter 04

Run it locally

Everything runs locally: the CLI generates tasks, Harbor orchestrates containers, your agent calls your model with your subscription. The platform is not in the execution path.

Prerequisites

Node.js 20 or newer for the CLI.
Harbor, a Python tool installed with uv. If you prefer not to install it globally, set LTB_HARBOR_BIN="uvx --from harbor harbor".
Docker: every task runs in its own container built from the task's Dockerfile.
An agent and a model you can call: Claude Code, Codex, OpenHands, Terminus and other Harbor agents are supported; the model is passed through with -m.

install

$ npm i -g lettokenburn-cli
$ uv tool install harbor
$ ltb --version && harbor --version && docker info >/dev/null && echo ok

Search and inspect

search · info

$ ltb bench search mmlu
$ ltb bench search --axis coding --harbor-only --limit 10
$ ltb bench info mmlu-pro

Both commands read the public static registry at https://lettokenburn.com/benchhub/index.json and /benchhub/b/<slug>.json; no account is needed. info prints facts, license, links, the top known scores and the exact pull and run commands for that benchmark. Point the CLI at a mirror or a local export with --registry <https://… | file:///dir> or LTB_BENCHHUB_REGISTRY. Add --json to any command for machine-readable output.

Pull

pull

$ ltb bench pull mmlu-pro --n 50
$ ltb bench pull mmlu-pro --n 200 --seed 7 --out ./eval/mmlu-pro
$ ltb bench pull gsm8k --all

What happens depends on the benchmark's adapter:

hf_rows: pages through the Hugging Face datasets server, samples --n rows (default 50; --seed makes the selection deterministic, default 7; --all takes the whole split) and writes one Harbor task directory per row under ./<slug>/ plus an ltb-manifest.json describing the pull.
harbor_native: if the benchmark is in the Harbor registry there is nothing to download. The CLI prints the dataset reference and you run it directly with ltb bench run <org>/<name>@<rev> --benchmark <slug> (or plain harbor run -d). If it lives in a git repo instead, pull shallow-clones it into --out.
external_harness: the benchmark needs its own harness. The CLI prints the instructions and repo URL and writes nothing.

A generated task directory looks like this:

mmlu-pro/mmlu-pro-0042/

task.toml                # schema_version 1.4; timeouts; [metadata] benchmark, source_dataset, config, split, row_idx, license, ltb_generated
instruction.md           # the question (+ options), "write the final answer to /app/answer.txt"
environment/Dockerfile   # FROM python:3.12-slim, WORKDIR /app
tests/expected.json      # { "answer": "C", "verifier": "mcq_letter" }
tests/test.sh            # compares /app/answer.txt, writes 1 or 0 to /logs/verifier/reward.txt
solution/solve.sh        # reference answer, only when the license allows redistribution

Verifiers are pure bash plus the Python standard library, so no dependencies are installed in the container: mcq_letter (accepts B, (B), B., Answer: B or the option text), exact_match, normalized_match, contains and numeric (tolerance 1e-6; commas, percent signs, fractions and \boxed{} are handled). Because the output follows Harbor's own tasks init template, plain harbor run -p ./mmlu-pro works too.

Sanity check with the oracle

Before spending real tokens, run the oracle agent. It executes each task's solution/solve.sh and should score 100. Anything less means a broken verifier or a malformed row, not a model problem, and should be reported (Chapter 07). The oracle is only available when the license allowed a solution/ directory.

oracle

$ ltb bench run ./mmlu-pro -a oracle
$ harbor run -p ./mmlu-pro -a oracle -y   # equivalent, plain Harbor

Run with your agent

run

$ ltb bench run ./mmlu-pro -a claude-code -m anthropic/claude-opus-4-8
$ ltb bench run ./mmlu-pro -a codex -m openai/gpt-5-4 -n 4 -k 1 -o ./jobs
$ ltb bench run terminal-bench/terminal-bench-2@1 --benchmark terminal-bench-2 -a claude-code -m anthropic/claude-opus-4-8   # Harbor-native, no pull
$ ltb bench run ./mmlu-pro -a claude-code -m anthropic/claude-opus-4-8 -- --some-harbor-flag

Thin wrapper over harbor run -p <dir> -a <agent> -m <model> -o <jobs-dir> -n <concurrent> -k <attempts> -y. Defaults: 4 concurrent trials, 1 attempt per task, jobs in ./jobs. Anything after -- is passed to Harbor untouched.
When the job finishes the CLI reads Harbor's job-level result.json and the per-trial reward files and writes <jobs-dir>/<job>/ltb-run-summary.json: score, metric, task count, attempts, a compact per-task reward list, a sha256 of the result file, Harbor and CLI versions. --summarize-only rebuilds that file for an existing job without re-running.
The <dir> argument may also be a Harbor registry dataset such as [email protected]; pass --benchmark <slug> so the summary knows which BenchHub record it belongs to.

Gated datasets and HF_TOKEN

Some Hugging Face datasets require you to accept terms before download. When the datasets server answers 401 or 403, the CLI tells you to accept the terms on huggingface.co and set HF_TOKEN (or HUGGING_FACE_HUB_TOKEN) in your environment, then retry. The token is sent only to Hugging Face and never to LetTokenBurn.

Timeouts

Generated task.toml files set a 600 s agent timeout, a 300 s verifier timeout and a 600 s image build timeout. Long agentic tasks may need more; edit the values in task.toml before running, and mention the change in --notes when you push. Timeouts count as reward 0, so a too-tight limit lowers your score rather than failing the run.

Chapter 05

Submit a reproduced score

Pushing a run is a single command and uploads a few hundred bytes of numbers. It is the only write operation in the whole BenchHub flow.

Push

push

$ ltb bench push                                 # newest job under ./jobs
$ ltb bench push ./jobs/2026-10-01__12-00-00     # a specific job dir
$ ltb bench push ./jobs --notes "temperature 0, 1 attempt, default timeouts"
$ ltb bench push --dry-run                       # print the body, send nothing

The command locates ltb-run-summary.json, optionally attaches --notes, and sends it as a RunSubmission to POST https://api.lettokenburn.com/v1/runs. On success it prints the run id and a shareable URL.

What is uploaded

benchmarkSlug, model (free text, normalized server-side), harness
taskCount, nAttempts, score, metric
tokensUsed and costUsd when the harness reported them, otherwise null
harborVersion, cliVersion
resultsHash: sha256 of the Harbor result file, so a run can be matched to the local artifact later
taskRewards: a compact list of { task, reward } pairs
notes: your free-form text, up to 2,000 characters

What is never uploaded

Model outputs, transcripts, trajectories or tool logs
Task instructions or dataset rows
Your API keys, your Hugging Face token, or any credentials
File contents from your machine beyond the summary JSON

If you want to publish full trajectories, do it on your own repo or dataset and link it in --notes.

Example request body

POST /v1/runs

{
  "benchmarkSlug": "mmlu-pro",
  "model": "anthropic/claude-opus-4-8",
  "harness": "claude-code",
  "taskCount": 50,
  "nAttempts": 1,
  "score": 84,
  "metric": "accuracy",
  "tokensUsed": null,
  "costUsd": null,
  "harborVersion": "0.23.1",
  "cliVersion": "0.2.0",
  "resultsHash": "3b7e…64 hex chars…9c1d",
  "taskRewards": [{ "task": "mmlu-pro-0001", "reward": 1 }, { "task": "mmlu-pro-0002", "reward": 0 }],
  "notes": "temperature 0"
}

Percentage-like metrics must be within 0–100; elo is sanity-capped at 10,000. The response is { data: RunRecord, url } with status 201; the record adds id, modelSlug, userId, status (uploaded | validated | rejected) and createdAt. Validation errors return 400 with a details array.

Anonymous or with an API key

Anonymous pushes are accepted. To attribute runs to your account, create a key on your profile and run ltb auth login --api-key ltb-… once (or set LTB_API_KEY). Keys are sent as Authorization: Bearer ltb-…. Attributed runs show your handle on the run page; anonymous runs show nothing.

Rate limits

30 runs per 24 hours per IP for anonymous pushes.
200 runs per 24 hours per user when an API key is attached (the IP limit still applies).
Over the limit the API returns 429 with code rate_limited. Push larger subsets rather than many tiny runs.

Where your run appears

Its own page at /runs/detail/?id=<run id>, with a per-task outcome strip and the hash, shareable as-is.
The /runs/ feed of recent community runs, filterable by benchmark, model and harness.
The Reproduced by the community panel on /b/<slug>/ and on the model page /m/<model>/, where runs for the same benchmark, model and harness are pooled into median, mean, min, max and a run count.
The leaderboard, where Reproduced (R) cells sit beside Self-reported (S) and Independent (I) cells. Aggregates are cached for 60 seconds, so a new run shows up within a minute.

Chapter 06

Compare: leaderboard and graph

The leaderboard is a heatmap plus a composite table; the graph shows how benchmarks relate. Both are rendered from the same snapshot that powers the directory.

Reading the heatmap

Rows are models, columns are benchmarks grouped by primary capability axis. By default only the most-scored benchmarks per axis are shown; toggle to show all.
Cell color encodes the normalized 0–100 score of the best included source; the glyph (S / I / R) tells you where that number came from.
Pale, dashed cells are low confidence. Treat them as a lead to verify, not a fact.
Empty cells mean no known score. They are never interpolated. An empty cell is an invitation to pull, run and push.
Hover or focus a cell for the raw value, metric, harness and a link to the source.
Filter by axis, industry or source type to look at, say, only Reproduced scores on coding benchmarks.

Harness mode

Agentic benchmarks depend heavily on the scaffold around the model. The composite table has two modes: any gives one row per model using its best score per benchmark regardless of harness; grouped gives one row per model + harness (for example claude-code, openhands, swe-agent) using only that harness's scores. Grouped mode is the fair way to compare agent frameworks; any mode is the fair way to compare models.

Composite table

Sorted by composite, with per-axis bars showing how many benchmarks in each axis a model covers. Rows below the coverage threshold (fewer than 5 benchmarks or fewer than 3 axes) are greyed out with N/A. The weights in use are printed under the table. Chapter 02 walks through the algorithm.

The graph

/graph/ draws benchmarks, papers, repos, datasets, models, capability axes, industries and organizations as nodes. Benchmark-to-benchmark edges carry one of these relations:

extends

A harder or broader successor built on the original (MMLU-Pro extends MMLU).

subset_of

A curated slice of a larger set (MATH-500 is a subset of MATH).

successor_of

A new generation that replaces the earlier one (SWE-bench Pro succeeds SWE-bench Verified).

variant_of

Same tasks, different verification or packaging (OSWorld-Verified is a variant of OSWorld).

same_family

Siblings released under one name over time (AIME 2024 and AIME 2025).

compared_with

Reported side by side in the same results table often enough to be worth showing together.

Structural relations (extends, subset_of, successor_of, variant_of, same_family) are extracted from paper full text and keep an evidence sentence in the crawler database; compared_with is statistical, from co-occurrence in results tables. Other edge types connect a benchmark to its paper, repo, dataset, axis, industry and to the models scored on it. The same edges power the Related section on each detail page.

Chapter 07

Data quality and corrections

BenchHub exists because benchmark numbers get repeated without sources. Scores are held to a strict rule: no source, no score. Benchmark metadata is a best-effort, automatically collected record, and each one says how complete it is. Here is exactly what to expect.

Scores

Every score has a source URL. A number without a source is not stored.
Every score is typed as self-reported, independent or reproduced, and never shown without that label.
No estimates, no interpolation, no trend-based filling. A missing cell is shown as empty or N/A.
Bulk-imported scores whose source is a whole leaderboard page rather than the exact cell are marked low confidence.
Reproduced aggregates are computed from pushed runs only; the platform never adds its own runs to the pool.

What the platform never does

Never asks for, stores or proxies your model API keys or subscription credentials.
Never runs your benchmarks for you; compute is yours, on your machine.
Never resells tokens, relays requests or takes payments on your behalf.
Never mirrors benchmark data whose license does not clearly allow it.
Never publishes model outputs from your runs; only the summary you push.

Benchmark metadata: collected automatically, reviewed, sometimes incomplete

Benchmark records are gathered automatically from 12 public channels, including arXiv and other paper indexes, Hugging Face, GitHub, evaluation frameworks such as lm-evaluation-harness, Inspect and OpenCompass, public leaderboards and the Harbor registry. An LLM reviewer checks each candidate before it is published, and a small number of records are reviewed by a person. Fields such as license, sample count, release date or metric can still be missing or wrong.

Verified

Reviewed by a person, or confirmed by at least two independent channels with most of the checklist filled in.

Auto

Published by the LLM reviewer and reasonably complete, but not independently confirmed.

Incomplete

Less than 40% of the checklist is filled in. Still listed, but ranked last and badged.

The checklist has seven items: paper, a data link (dataset or repo), license, sample count, release date, metric, and whether the benchmark is runnable through Harbor. Each record shows its tier and the items it is missing. The home page, the default directory order and the default leaderboard view favour Verified and runnable records.

License and redistribution

The registry indexes and links. Task data is mirrored only when license.allowsRedistribution is explicitly true.
For link-only benchmarks the CLI fetches rows from the original host to your machine for local evaluation. Do not publish the generated task directories, and do not commit them to a public repository.
Each page shows the SPDX license id when known and links to the license text. If a license is listed incorrectly, report it; the fix is a data change, not a policy debate.
Cite original authors in anything you publish. The citation block is on every detail page.

Reporting a wrong score or a broken task

If a cell disagrees with its source, a source link is dead, the oracle fails on a generated task, a license is mislabeled or a benchmark record is missing a field you can supply, tell us. Include the benchmark slug, the model slug, the cell's current value, the value you believe is correct and the URL that supports it.

Email [email protected]

Corrections are applied in the crawler database and flow into the next snapshot export. Disputed reproduced runs can be marked rejected, which removes them from aggregates while keeping the record visible for transparency.

Chapter 08

Agent integration

The loop in Chapter 04 and 05 is deterministic enough for an agent to run unattended. This is where spare Claude Code or Codex quota turns into reproduced scores.

Entry points

/agent/: API keys, the BenchHub API endpoints and the MCP server for agents.
/skill.md: a plain-markdown skill an agent can read directly. It lists the exact commands, the POST /v1/runs schema with an example body, rate limits and the honesty rules.
OpenAPI at https://api.lettokenburn.com/v1/meta/openapi.json for the machine-readable API description.
The public registry JSON at https://lettokenburn.com/benchhub/ (index.json, b/<slug>.json, models.json, scores.json) for agents that prefer fetching to shelling out.

Read-only endpoints

GET /v1/runs?benchmark=&model=&harness=&limit=&page=: list runs.
GET /v1/runs/:id: one run with its shareable URL.
GET /v1/bench/reproduced?benchmark=&model=: aggregates per benchmark, model and harness. Use this to find which combinations still lack a reproduced score.

None of these need authentication.

A minimal unattended loop

agent loop (pseudo)

# 1. choose a benchmark with a Harbor adapter and no reproduced score for your model
ltb bench search --axis knowledge --harbor-only --limit 20 --json
curl -s "https://api.lettokenburn.com/v1/bench/reproduced?model=anthropic/claude-opus-4-8"
# 2. pull, sanity-check, run, push
ltb bench pull mmlu-pro --n 50
ltb bench run ./mmlu-pro -a oracle
ltb bench run ./mmlu-pro -a claude-code -m anthropic/claude-opus-4-8
ltb bench push --notes "unattended; defaults"

Agents must follow the same rules as people: push the number the verifier produced, never a number the model wrote in a chat; stop on oracle failures rather than pushing a run against a broken task; and respect the rate limits. See skill.md for the full checklist.

Chapter 09

FAQ

Short answers to the questions that come up most. If yours is missing, the GitHub link in Chapter 07 is the place to ask.

Do I need an account?

No. Searching, reading, pulling, running and pushing all work without one. An API key only attributes runs to you and raises your daily push limit.

Does LetTokenBurn pay me for runs?

No. There are no credits, points or bounties. The reward is a public, citable reproduced score and a run page with your name on it if you choose to attach a key.

Why does my reproduced score differ from the vendor number?

Different subset size, prompt template, harness, temperature, attempts or timeouts all move the number. That difference is exactly what the Reproduced column is meant to expose. Record your settings in --notes.

Can I push a run from plain harbor run without the CLI?

Yes. Run `ltb bench run <dir> --summarize-only -o <jobs-dir>` to build the summary from an existing job, then `ltb bench push`. Or POST a RunSubmission yourself (Chapter 05).

Why is a benchmark index-only?

Either no adapter has been written yet, the dataset is not on Hugging Face in a row-shaped form, or it needs a bespoke harness. ltb bench info prints whatever is known. Adapter requests are welcome via GitHub.

Is a 50-row subset meaningful?

It is a smoke test, not a paper result. For a number you want to cite, use --all or a few hundred rows with a fixed --seed, and say so in --notes. The run page shows the task count so readers can judge.

Where is the old task marketplace?

Retired. LetTokenBurn now does one thing: benchmarks. The name stayed because the idea stayed: spend spare subscription tokens on something with public value.

What about the AI Native test?

It is still at /ai-native/ as a personal entry point and is unrelated to benchmark scores.

Glossary

Benchmark

A named, versioned evaluation set with a defined metric. One record, one slug, one page at /b/<slug>/.

Capability axis

The dimension a benchmark mainly measures: knowledge, math, coding, agentic, multimodal, long context, instruction following, safety, abstract reasoning or multilingual.

Task format

How a single item is answered and checked: multiple choice, short answer, open-ended QA, code generation, code repair, agentic environment, preference, classification, generation.

Harbor task

A directory with task.toml, instruction.md, an environment Dockerfile and a tests/ verifier. Harbor is the runner used by Terminal-Bench 2.0; BenchHub packages benchmarks into this format.

Run

One execution of a pulled benchmark with one agent and one model. The CLI summarizes it into a RunSubmission and `ltb bench push` uploads only that summary.

Quality tier

How complete and trustworthy a benchmark record is: Verified, Auto or Incomplete. Shown on every record together with the list of missing fields.

Confidence

How much the registry trusts a score cell: high, medium or low. Low-confidence cells are drawn pale with a dashed border and still link to their source.

Composite

A single 0–100 number per model (or per model + harness): percentile ranks on live benchmarks, averaged per capability axis, then weighted across axes. N/A when coverage is too thin.