Unanswerability Benchmark
Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification Signals
A turn-labeled multi-turn benchmark with 423 conversations and 1,661 labeled turn-states designed to evaluate latent underspecification signals and unanswerability detection in dialogue.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info unanswerability-benchmark
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
4 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemma4-12B-it | 0.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Granite4.1-8B | 0.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| qwen3.5-9B | 0.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Ministral3-8B | 0.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
The benchmark comprises 423 conversations with constraints revealed incrementally, featuring 1,661 turn-states audited by annotators. It evaluates the transferability of unanswerability probes across domains and structures using a simulated user harness.
Scoring relies on probe AUROC and gate firing precision. Results show robust transfer within unanswerability grounds (AUROC 0.77–0.97) but poor zero-shot transfer to multi-turn structures, where performance matches TF-IDF baselines.