ANLI
Harbor adapterAdversarial NLI (ANLI) is a dataset collected via an iterative, adversarial human-and-model-in-the-loop procedure.
Verified100%Knowledge & Reasoning
- Samples
- 169,265
- Released
- 2019
- Scores
- 12
WinoGrande
An Adversarial Winograd Schema Challenge at Scale
WinoGrande is a collection of 44k problems, inspired by Winograd Schema Challenge (Levesque, Davis, and Morgenstern 2011), but adjusted to improve the scale and robustness against the dataset-specific bias.
How to run
ltb · winogrande
$ npm i -g lettokenburn-cli$ ltb bench pull winogrande
External harness. pip install lm-eval && lm_eval --model <hf|vllm|openai-chat-completions> --model_args <model args> --tasks winogrande (lm-evaluation-harness task family 'winogrande') https://github.com/EleutherAI/lm-evaluation-harness
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
132 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Llama-3.1-405B-Instruct | 89.2 | N/A | Independent | medium | N/A | arxiv.org |
| Claude 3 Opus | 88.5 | N/A | Independent | medium | Mar 4, 2024 | anthropic.com |
| Claude 3 Opus | 88.5 | N/A | Self-reported | medium | N/A | assets.anthropic.com |
| GPT-4 | 87.5 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-4 0314 | 87.5 | N/A | Self-reported | medium | N/A | arxiv.org |
| GPT-4 32K 0314 | 87.5 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon-180B | 87.1 | N/A | Self-reported | medium | N/A | arxiv.org |
| UNICORN | 86.6 | N/A | Self-reported | medium | N/A | ojs.aaai.org |
| DeepSeek-V2 | 86.3 | N/A | Self-reported | medium | N/A | arxiv.org |
| DeepSeek-V3 | 85.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 540B | 85.1 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 540B | 85.1 | N/A | Self-reported | medium | N/A | jmlr.org |
| DeepSeek Coder V2 | 83.7 | N/A | Independent | medium | N/A | arxiv.org |
| Meta-Llama-3-70B-Instruct | 83.5 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-L | 83.0 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-L | 83.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| Qwen2.5-72B-Instruct | 82.3 | N/A | Independent | medium | N/A | arxiv.org |
| Llama-3.1-405B-Instruct | 82.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| GPT-3.5 Turbo | 81.6 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-3.5 Turbo 0613 | 81.6 | N/A | Independent | medium | N/A | arxiv.org |
| text-davinci-002 | 81.6 | N/A | Independent | medium | N/A | arxiv.org |
| Phi-3-medium-128k-instruct | 81.5 | N/A | Self-reported | medium | N/A | arxiv.org |
| Phi-3-small-8k-instruct | 81.5 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 540B | 81.1 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| Qwen2.5-Coder-32B-Instruct | 80.8 | N/A | Self-reported | medium | N/A | arxiv.org |
| Llama 2 70B | 80.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-M | 79.2 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| GLaM | 79.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-M | 79.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| Gemma 7B | 79.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| Megatron-Turing NLG 530B | 78.9 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon 11B | 78.3 | N/A | Self-reported | medium | N/A | arxiv.org |
| Nemotron-4 15B | 78.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-S | 77.9 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-S | 77.9 | N/A | Self-reported | medium | N/A | arxiv.org |
| text-davinci-001 | 77.7 | N/A | Independent | medium | N/A | arxiv.org |
| Mixtral 8x7B | 77.2 | N/A | Independent | medium | Jan 8, 2024 | arxiv.org |
| Mixtral-8x7B-Instruct-v0.1 | 77.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 65B | 77.0 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| PaLM 62B | 77.0 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM-cont 62B | 77.0 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| LLaMA 65B | 77.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 62B | 77.0 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 62B | 77.0 | N/A | Self-reported | medium | N/A | jmlr.org |
| PaLM-cont 62B | 77.0 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon 40B | 76.9 | N/A | Independent | medium | N/A | arxiv.org |
| Qwen2.5-Coder-14B | 76.8 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 2 34B | 76.7 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 2 34B | 76.7 | N/A | Self-reported | medium | N/A | arxiv.org |
| Falcon 40B | 76.4 | N/A | Self-reported | medium | N/A | arxiv.org |
Top 50 of 132 by score
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
WinoGrande is a collection of 44k problems, inspired by Winograd Schema Challenge (Levesque, Davis, and Morgenstern 2011), but adjusted to improve the scale and robustness against the dataset-specific bias. Formulated as a fill-in-a-blank task with binary options, the goal is to choose the right option for a given sentence which requires commonsense reasoning.
Available in EleutherAI lm-evaluation-harness as task family winogrande (1 tasks).