BrickBench
Harbor: pendingA benchmark of 300 text prompts for agentic LEGO-set design, evaluating physical validity, semantic alignment, and design quality across three complexity settings.
- Samples
- 300
- Released
- 2026
- Scores
- 11
SDDBench
Ship Design Decision Benchmark
A benchmark with 5,000 records and 43,496 typed questions evaluating natural-language-driven constrained ship-form editing decisions.
How to run
ltb · sddbench
$ npm i -g lettokenburn-cli$ ltb bench info sddbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 92.9 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| DeepSeek V4.1 Flash | 92.8 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| GPT-5.6 Sol | 92.4 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Jev | 91.0 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Qwen3 Max (2025-09-23) | 75.6 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Gemma3 | 67.0 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Llama3.2 | 63.9 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
SDDBench is a subset of the Ship Design Decision Dataset containing 5,000 cleaned records. It features 43,496 typed questions designed to evaluate models on translating natural language requests into discrete ship design actions.
Evaluation metrics include question accuracy, FFD exact match, and calibration scores. The Chip model achieved 95.90% question accuracy and 99.32% FFD exact match on this benchmark.