HakemBench
Harbor: pendingA Turkish benchmark with 2,346 items and 4,275 questions measuring typed decisions (choice, yes/no, score) across seven tracks including legal, education, and safety.
- Samples
- 2,346
- Released
- 2026
- Scores
- 9
TypedBench
TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models
TypedBench evaluates System One decision models on calibration, framing sensitivity, and cost using seven policy-labelled programmatic generators and nine evaluation suites.
How to run
ltb · typedbench
$ npm i -g lettokenburn-cli$ ltb bench info typedbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
8 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Jev 1.13 | 1.0 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Gemma 4 31B, constrained | 0.9 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Kev 9B | 0.9 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5-9B, constrained | 0.8 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Kev 4B | 0.8 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Kev -0.8B | 0.6 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Laya typed-dec. | 0.5 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Laya english | 0.5 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
TypedBench uses seven programmatic generators to create contamination-resistant items with derived labels, testing typed probability outputs across choice, score, and noul primitives. It assesses accuracy, calibration error relative to noise floors, selective prediction, and realized cost under asymmetric matrices.
The benchmark evaluates hosted and open-weight models, revealing that while decoders are stable, they are slow; hosted models are wording-sensitive and underconfident. It exposes failure modes in policy adherence and probability quality rather than producing a single leaderboard score.