LetTokenBurn
Sign in
Auto86%Harbor: pendingCodingKnowledge & Reasoning

ARCS

Ambiguity Resolution Corpus for SQL

A text-to-SQL benchmark with 311 tasks featuring naturally occurring ambiguities over real-world databases, measuring end-to-end execution accuracy.

PaperRepo

How to run

ltb · arcs
$ npm i -g lettokenburn-cli
$ ltb bench info arcs

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

12 sourced entries
HarnessSourceConfidenceLink
Gemini 3 Pro44.0N/AIndependentmediumOct 7, 2026arxiv.org
Gemini 2.5 Pro30.9N/AIndependentmediumOct 7, 2026arxiv.org
Claude Haiku 4.528.6N/AIndependentmediumOct 7, 2026arxiv.org
Gemini 2.5 Flash27.6N/AIndependentmediumOct 7, 2026arxiv.org
Gptoss 120B26.1N/AIndependentmediumOct 7, 2026arxiv.org
DeepSeek-V3.123.5N/AIndependentmediumOct 7, 2026arxiv.org
DeepSeek-R1-052823.1N/AIndependentmediumOct 7, 2026arxiv.org
Qwen3-235B-A22B-Thinking-250720.3N/AIndependentmediumOct 7, 2026arxiv.org
Kimi K2 Thinking Turbo19.3N/AIndependentmediumOct 7, 2026arxiv.org
Qwen 3 Coder 480B15.8N/AIndependentmediumOct 7, 2026arxiv.org
Gptoss 20B7.4N/AIndependentmediumOct 7, 2026arxiv.org
Qwen3 8B5.8N/AIndependentmediumOct 7, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

ARCS contains 311 realistic disambiguation tasks derived from unmodified real-world databases, annotated with all valid ambiguity points, interpretations, and SQL queries. It evaluates systems via a user simulator supporting structured disambiguation.

Scoring uses end-to-end execution accuracy. Experiments show gpt-6-sol achieves 51% accuracy, while no open-source model exceeds 27%, indicating significant challenges in handling ambiguity.