ANLI
Harbor adapterAdversarial NLI (ANLI) is a dataset collected via an iterative, adversarial human-and-model-in-the-loop procedure.
Verified100%Knowledge & Reasoning
- Samples
- 169,265
- Released
- 2019
- Scores
- 12
OpenBookQA
OBQA
OpenBookQA is a question-answering dataset modeled after open book exams for assessing human understanding of a subject.
How to run
ltb · openbookqa
$ npm i -g lettokenburn-cli$ ltb bench pull openbookqa --n 50$ ltb bench run ./openbookqa -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
62 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Phi-3-mini-4k-instruct | 88.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| Phi-3-small-8k-instruct | 88.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| Phi-3-medium-128k-instruct | 87.4 | N/A | Self-reported | medium | N/A | arxiv.org |
| GPT-3.5-Turbo-1106 | 86.0 | N/A | Independent | medium | N/A | arxiv.org |
| Mixtral-8x7B-Instruct-v0.1 | 85.8 | N/A | Independent | medium | N/A | arxiv.org |
| Meta-Llama-3-8B-Instruct | 82.6 | N/A | Independent | medium | N/A | arxiv.org |
| Mistral-7B-v0.1 | 79.8 | N/A | Independent | medium | N/A | arxiv.org |
| Gemma 7B | 78.6 | N/A | Independent | medium | N/A | arxiv.org |
| Phi-2 | 73.6 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 540B | 68.0 | N/A | Self-reported | medium | N/A | jmlr.org |
| GPT-3-175B | 65.4 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| text-davinci-001 | 65.4 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon-180B | 64.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| GLaM | 63.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 65B | 60.2 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| Llama 2 70B | 60.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 65B | 60.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 33B | 58.6 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 2 7B | 58.6 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 33B | 58.6 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-L | 58.5 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| LLaMA 2 34B | 58.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| OPT 66B | 58.0 | N/A | Independent | medium | Mar 30, 2023 | arxiv.org |
| GPT-3-175B | 57.6 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| PaLM 2-S | 57.4 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-M | 57.4 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 7B | 57.2 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 7B | 57.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 2 13B | 57.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| Falcon 40B | 56.6 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 13B | 56.4 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 13B | 56.4 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-M | 56.2 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-S | 56.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 540B | 53.4 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 540B | 53.4 | N/A | Independent | medium | N/A | arxiv.org |
| MPT 30B | 52.0 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon 7B | 51.6 | N/A | Independent | medium | N/A | arxiv.org |
| MPT 7B | 51.4 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 62B | 50.4 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 62B | 50.4 | N/A | Independent | medium | N/A | arxiv.org |
| BLOOM 176B | 47.2 | N/A | Independent | medium | Mar 30, 2023 | arxiv.org |
| Meta-Llama-3-8B-Instruct | 45.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| GPT-NeoX 50B | 44.2 | N/A | Independent | medium | Mar 30, 2023 | arxiv.org |
| LLaMA 7B | 44.2 | N/A | Independent | medium | N/A | arxiv.org |
| Xgen 7B 8K Base | 40.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| RedPajama-INCITE-7B-Base | 40.0 | N/A | Independent | medium | N/A | arxiv.org |
| OPT-13b | 39.8 | N/A | Independent | medium | N/A | arxiv.org |
| Dolly V2 12B | 39.2 | N/A | Independent | medium | N/A | arxiv.org |
| GPT-NeoX 20B | 38.8 | N/A | Independent | medium | N/A | arxiv.org |
Top 50 of 62 by score
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
OpenBookQA is a question-answering dataset modeled after open book exams for assessing human understanding of a subject. It consists of 5,957 multiple-choice elementary-level science questions (4,957 train, 500 dev, 500 test), which probe the understanding of a small “book” of 1,326 core science facts and the application of these facts to novel situations. For training, the dataset includes a mapping from each question to the core science fact it was designed to probe. Answering OpenBookQA questions requires additional broad common knowledge, not contained in the book. The questions, by design, are answered incorrectly by both a retrieval- based algorithm and a word co-occurrence algorithm.
Available in EleutherAI lm-evaluation-harness as task family openbookqa (1 tasks).