LetTokenBurn
Sign in
Auto86%Harbor: pendingKnowledge & ReasoningSafety & Alignment

ArcticAbstain

Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science

A paired benchmark of 194 Arctic science MCQs measuring LLM abstention sensitivity by comparing answer-present vs. answer-absent conditions.

PaperRepo

How to run

ltb · arcticabstain
$ npm i -g lettokenburn-cli
$ ltb bench info arcticabstain

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

8 sourced entries
HarnessSourceConfidenceLink
ChatGPT 5.6 Terra0.8N/AIndependentmediumOct 7, 2026arxiv.org
ChatGPT 5.6 Sol1.1N/AIndependentmediumOct 7, 2026arxiv.org
Gemini 3.7 Flash7.3N/AIndependentmediumOct 7, 2026arxiv.org
Gemini 3.8 Flash8.4N/AIndependentmediumOct 7, 2026arxiv.org
Claude Fable 5.141.5N/AIndependentmediumOct 7, 2026arxiv.org
Claude Opus 544.4N/AIndependentmediumOct 7, 2026arxiv.org
Claude Sonnet 549.8N/AIndependentmediumOct 7, 2026arxiv.org
ChatGPT Astra68.6N/AIndependentmediumOct 7, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

ArcticQA contains 194 multiple-choice questions derived from primary Arctic research, with automated checks for answer support and distractor contradiction. ArcticAbstain pairs these into answer-present and answer-absent conditions to evaluate abstention behavior.

Scoring measures false abstention and false commitment rates across three trials per condition. Results show answer-present abstention ranges from 0-63%, with a 5.05% average increase when valid answers are removed.