LetTokenBurnAgent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
Evaluates if LLM agents can autonomously create cheap, reusable artifacts for large unlabelled workloads under fixed resource budgets.
$ npm i -g lettokenburn-cli$ ltb bench info bottled
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 0.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| GLM-5.3 Flash | 0.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini 3.1 Pro | 0.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3.8 Max | 0.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| GPT-5.6-terra | 0.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3.8 Flash | 0.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| GLM-5.3 | 0.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Claude Sonnet 5 | 0.2 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| GPT-5.6 Sol | 0.1 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini 3.1 Flash Lite | 0.1 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Agents receive entire unlabelled workloads and must balance quality and cost by training small models or writing programs. It measures the ability to amortize inference costs across millions of instances.
Scoring uses macro-F1 relative to zero-shot performance and cost savings. Results show strong zero-shot models often fail to bottle effectively, with many runs underperforming distillation baselines.