ArchitectureIQ
Harbor: pendingA benchmark measuring model intuition by asking LLMs and humans to predict which training recipe yields the best metric for a synthetic dataset.
Auto86%Coding
- Samples
- 500
- Released
- 2026
- Scores
- 13
Actionable Indeterminacy Benchmark
Solver-grounded benchmark for actionable indeterminacy in LLM preference reasoning
Evaluates LLMs' ability to decide between acting, asking for clarification, or relaxing constraints in preference reasoning tasks using solver-verified labels across four domains.
How to run
ltb · actionable-indeterminacy-benchmark
$ npm i -g lettokenburn-cli$ ltb bench info actionable-indeterminacy-benchmark
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
4 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-6 Astra | 92.6 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Gemini 3.1 Pro | 91.5 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Claude Sonnet 5 | 87.3 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| GPT-oss 120B | 83.5 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The benchmark spans object allocation, meeting scheduling, apartment choice, and stable matching. It uses matched pairs where the source remains constant but intervention necessity changes, with exact solvers determining ground truth.
Scoring separates decision correctness, matched-pair reliability (Intervention Flip Correctness), and fully correct responses including valid actions or repairs. Findings show models struggle to recognize when intervention is unnecessary despite identifying uncertainty.