RDL-Bench
RDL-Bench: A Benchmark for Task-Aware Question Answering over Relational Databases
A benchmark for task-aware QA over real multi-table relational databases, covering reasoning and prediction tasks across seven domains with denotation-level ground truth.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info rdl-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
13 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude-only | 0.7 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen32B-only | 0.6 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| CHESS | 0.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| XiYanSQL-32B | 0.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Claude-Schema | 0.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Claude-2Hop | 0.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen32B-Schema | 0.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen32B-2Hop | 0.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| DIN-SQL | 0.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| C3 | 0.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Llama8B-2Hop | 0.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Llama8B-Schema | 0.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| RESDSQL | 0.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
RDL-Bench evaluates natural language question answering over relational databases, distinguishing between reasoning-oriented (current state) and prediction-oriented (future outcomes) questions. It includes task metadata, schema info, and temporal context.
Scoring uses macro-average denotation score. Experiments show current methods achieve at most 0.49, indicating significant room for improvement in schema grounding and relational composition.