LetTokenBurnArchitectureIQ: On the Measure of Training Intuition
A benchmark measuring model intuition by asking LLMs and humans to predict which training recipe yields the best metric for a synthetic dataset.
$ npm i -g lettokenburn-cli$ ltb bench info architectureiq
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-6 Astra | 76.4 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Opus 5 | 76.0 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Fable 5 | 72.2 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V4 Flash | 70.6 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V4 Pro | 68.4 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6 Luna | 67.8 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| Gemini 3.1 Pro | 66.8 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| GLM-5.2 | 47.2 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi K3 | 46.8 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Sonnet 5 | 44.0 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6 Sol | 44.0 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-4o | 40.0 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
| Llama-3.1-70B-Instruct | 38.0 | architectureiq | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Content & Construction: ArchitectureIQ is a fully controlled, verifiable probe consisting of 500 questions. Each question presents one synthetic dataset and several candidate training recipes rendered as executable code. The ground truth winner is established by running these programs across random seeds.
Scoring & Findings: Models are evaluated on their ability to predict the best recipe before execution (accuracy). Frontier models achieve ~76% accuracy, outperforming humans (~66%) but falling short of perfection. Key findings include that intuition is empirical rather than structured, not maximally condensed into a knowledge base, and insensitive to dataset properties.