AgenticBBO-Bench
Harbor: pendingA cross-domain benchmark evaluating LLM agents on five black-box optimization tasks (synthetic, HPO, DB tuning, chip design, molecular) under a unified finite-budget protocol.
- Samples
- N/A
- Released
- 2026
- Scores
- 7
A2Z GameSpec-Bench
A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?
A benchmark of 100 long-form Game Design Documents (GDDs) evaluating coding agents' ability to generate playable games that faithfully satisfy interdependent requirements across logic, rendering, and play.
How to run
ltb · a2z-gamespec-bench
$ npm i -g lettokenburn-cli$ ltb bench info a2z-gamespec-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
9 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 77.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Opus 5 | 73.9 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-6 Astra | 71.6 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.5 | 63.2 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6 Sol | 61.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Opus 4.8 | 56.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GLM-5.3 | 55.6 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V4 Pro | 50.4 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi-K2.7 | 39.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Content & Construction: The dataset consists of 50 'Small' designs with compact scope and 50 'Big' designs with extensive content. Each GDD is converted into a fixed Dependency-Aware Contract containing rules, constraints, and prerequisite relations to evaluate end-to-end game development.
Scoring & Findings: Faithfulness (GDD Fidelity) is measured by checking if the generated game satisfies contract requirements via source-code inspection, visual assessment, and adaptive playtesting. Key findings show that high compilability does not guarantee fidelity; requirement-specific feedback improves GDD Fidelity by 10.9% relative to self-revision.