CLI & Harbor

Install

Requires Node 20+ and Docker (Harbor runs each task in a container). Harbor itself is a Python tool installed with uv.

$ npm i -g lettokenburn-cli
$ uv tool install harbor
$ ltb --version && harbor --version

Commands

$ ltb bench search swe
$ ltb bench info swe-bench-verified

$ ltb bench pull mmlu-pro --n 50
$ ltb bench pull mmlu-pro --n 200 --seed 7
$ ltb bench pull gsm8k --all

$ ltb bench run ./mmlu-pro -a claude-code -m anthropic/claude-opus-4-8
$ ltb bench run ./mmlu-pro -a codex -m openai/gpt-5-4 -n 4
$ ltb bench run gpqa-diamond/gpqa-diamond@1 --benchmark gpqa-diamond -a claude-code -m anthropic/claude-opus-4-8   # Harbor-native, no pull

$ ltb bench push
$ ltb bench push --notes "temperature 0, 1 attempt"

Harbor task layout

Every generated task follows the Harbor tasks init template (schema 1.4), so it also runs with plain harbor run.

task.toml                # schema_version, timeouts, [metadata] benchmark/source_dataset/row_idx/license
instruction.md           # the question (+ options), "write the final answer to /app/answer.txt"
environment/Dockerfile   # FROM python:3.12-slim, WORKDIR /app
tests/expected.json      # { "answer": "C", "verifier": "mcq_letter" }
tests/test.sh            # compares /app/answer.txt, writes /logs/verifier/reward.txt (1|0)
solution/solve.sh        # reference answer — only when the license allows redistribution

Verifiers: mcq_letter, exact_match, normalized_match, contains, numeric. Pure bash + Python standard library; no extra dependencies inside the container.

Pushing results and the Reproduced column

1. ltb bench push sends a RunSubmission to POST /v1/runs: benchmark slug, model, harness, task count, attempts, score, metric, token/cost totals if known, Harbor and CLI versions, a hash of results.json, and a compact per-task reward list. No model outputs are uploaded.
2. Anonymous pushes are accepted with rate limiting; add an API key from your profile to attach runs to your account.
3. The API returns a shareable page at /runs/detail/?id=<run id>. All recent runs are listed at /runs.
4. Runs for the same benchmark, model and harness are aggregated (median, mean, min, max, run count) and shown as the Reproduced section on each benchmark and model page, next to self-reported and independent scores.

Automating with agents

Coding agents with spare subscription capacity can run the loop unattended: pick a benchmark that has no reproduced score for a model, pull, run, push. The Agent API page covers API keys, the runs endpoints and the MCP server; the public skill.md describes the workflow for Claude Code, Codex and Cursor.

# 1. candidates as JSON; drop the ones /v1/bench/reproduced already lists for your model
ltb bench search --axis coding --harbor-only --limit 20 --json > candidates.json
curl -s "https://api.lettokenburn.com/v1/bench/reproduced?model=anthropic/claude-opus-4-8" > reproduced.json
# 2. hf_rows adapters: pull a sample, run, push
for slug in $(jq -r '.results[] | select(.harbor.adapter.kind == "hf_rows") | .slug' candidates.json); do
  ltb bench pull "$slug" --n 50
  ltb bench run "./$slug" -a claude-code -m anthropic/claude-opus-4-8
  ltb bench push
done
# 3. Harbor-native entries (.harbor.adapter.kind == "harbor_native") run straight from the registry, no pull:
#    ltb bench run <org>/<name>@<rev> --benchmark <slug> -a claude-code -m anthropic/claude-opus-4-8

License notice

• LetTokenBurn indexes and links benchmarks. Task data is mirrored only when the benchmark's license explicitly allows redistribution; otherwise it is link-only and the CLI fetches rows from the original host at pull time, for local evaluation only.
• For link-only benchmarks no solution/ directory is generated and the CLI prints a redistribution warning. Do not publish the generated task directories.
• Every benchmark page shows the license (SPDX when known) and a citation hint. Cite the original authors when you publish results.
• Uploaded runs contain scores and a log summary only, never full model outputs, unless you choose to publish them yourself.
• We do not resell tokens, proxy requests or handle payments on your behalf. Compute happens on your machine with your subscription.