ARC-Challenge
Harbor: pendingARC-Challenge — AI evaluation benchmark listed by opencompass.
Incomplete14%Knowledge & Reasoning
- Samples
- N/A
- Released
- N/A
- Scores
- 26
WebQuestions
The WebQuestions dataset is a question answering dataset using Freebase as the knowledge base and contains 6,642 question-answer pairs.
How to run
ltb · webquestions
$ npm i -g lettokenburn-cli$ ltb bench pull webquestions --n 50$ ltb bench run ./webquestions -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
License is link-only: ltb bench pull fetches rows from the original host for local evaluation only and does not write a solution/ directory. Do not redistribute the generated tasks.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
9 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| PaLM 540B | 43.5 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| GPT-3-175B | 41.5 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| PaLM 2-L | 28.2 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-M | 26.9 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| GPT-3-175B | 25.3 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| PaLM 540B | 22.6 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 2-S | 21.8 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| GPT-3-175B | 14.4 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| PaLM 540B | 10.6 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The WebQuestions dataset is a question answering dataset using Freebase as the knowledge base and contains 6,642 question-answer pairs. It was created by crawling questions through the Google Suggest API, and then obtaining answers using Amazon Mechanical Turk. The original split uses 3,778 examples for training and 2,032 for testing. All answers are defined as Freebase entities.
Example questions (answers) in the dataset include “Where did Edgar Allan Poe died?” (baltimore) or “What degrees did Barack Obama get?” (bachelor_of_arts, juris_doctor).
Source: Question Answering with Subgraph Embeddings Image Source: Berant et al