LetTokenBurnConversational Task Disambiguation over Tabular Data Benchmark Suite
AmbiTab is a benchmark suite of >8,000 ambiguous text-to-SQL tasks across six datasets, evaluating conversational disambiguation and oracle leakage via a unified representation.
$ npm i -g lettokenburn-cli$ ltb bench info ambitab
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
AmbiTab unifies six existing ambiguous text-to-SQL datasets (derived from BIRD, BIRD-Interact, Ambrosia, AmbiQT, TACO) into a common format specifying agent, oracle, and verifier access boundaries. It supports evaluating asking policies separately from solution generation.
Scoring uses judge-free leakage diagnostics, asking precision/recall, and task success rates. Experiments show trained asking policies improve disambiguation metrics across all datasets and task success on five, while quantifying oracle leakage effects.