ARC-Challenge
Harbor: pendingARC-Challenge — AI evaluation benchmark listed by opencompass.
Incomplete14%Knowledge & Reasoning
- Samples
- N/A
- Released
- N/A
- Scores
- 26
WSC
Winograd Schema Challenge
The Winograd Schema Challenge was introduced both as an alternative to the Turing Test and as a test of a system’s ability to do commonsense reasoning.
How to run
ltb · wsc
$ npm i -g lettokenburn-cli$ ltb bench pull wsc --n 50$ ltb bench run ./wsc -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
16 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| PaLM 540B | 89.5 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 540B | 89.1 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 2-M | 88.1 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-L | 86.9 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 540B | 86.3 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 2-S | 84.6 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| GPT-3-175B | 80.1 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| UL2 20B | 79.9 | N/A | Independent | medium | May 10, 2022 | arxiv.org |
| GPT-2-XL 1.5B | 73.3 | N/A | Independent | medium | Apr 27, 2023 | arxiv.org |
| GPT-2-XL 1.5B | 70.7 | N/A | Independent | medium | Feb 14, 2019 | d4mucfpksywv.cloudfront.net |
| OPT 1.3B | 64.2 | N/A | Independent | medium | May 24, 2024 | arxiv.org |
| OPT-125M | 59.6 | N/A | Independent | medium | May 24, 2024 | arxiv.org |
| Pythia 12B | 54.8 | N/A | Independent | medium | Apr 3, 2023 | arxiv.org |
| Pythia 2.8B | 38.5 | N/A | Independent | medium | Apr 3, 2023 | arxiv.org |
| Pythia 12B | 36.5 | N/A | Independent | medium | Apr 3, 2023 | arxiv.org |
| Pythia 6.9B | 36.5 | N/A | Independent | medium | Apr 3, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The Winograd Schema Challenge was introduced both as an alternative to the Turing Test and as a test of a system’s ability to do commonsense reasoning. A Winograd schema is a pair of sentences differing in one or two words with a highly ambiguous pronoun, resolved differently in the two sentences, that appears to require commonsense knowledge to be resolved correctly. The examples were designed to be easily solvable by humans but difficult for machines, in principle requiring a deep understanding of the content of the text and the situation it describes.
The original Winograd Schema Challenge dataset consisted of 100 Winograd schemas constructed manually by AI experts. As of 2020 there are 285 examples available; however, the last 12 examples were only added recently. To ensure consistency with earlier models, several authors often prefer to report the performance on the first 273 examples only. These datasets are usually referred to as WSC285 and WSC273, respectively.
Source: https://arxiv.org/pdf/2004.13831.pdf Image Source: https://arxiv.org/pdf/1907.11983.pdf