TIQ
Existing benchmarks for temporal QA focus on a single information source (either a KB or a text corpus), and include only few questions with implicit constraints.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info tiq
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
2 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4 | 28.6 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| InstructGPT | 23.6 | N/A | Independent | medium | Mar 4, 2022 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Existing benchmarks for temporal QA focus on a single information source (either a KB or a text corpus), and include only few questions with implicit constraints. we devise a new method for automatically creating temporal questions with implicit constraints, with systematic controllability of different aspects, including the relative importance of different source types (text, infoboxes, KB), fractions of prominent vs. long-tail entities, question complexity, and more.