Natural Questions
Harbor: pendingNatural Questions — AI evaluation benchmark listed by opencompass.
Auto57%Knowledge & Reasoning
- Samples
- N/A
- Released
- 2019
- Scores
- 15
BoolQ
Exploring the Surprising Difficulty of Natural Yes/No Questions
Reading comprehension dataset that queries for complex, non-factoid information, and require difficult entailment-like inference to solve.
How to run
ltb · boolq
$ npm i -g lettokenburn-cli$ ltb bench pull boolq --n 50$ ltb bench run ./boolq -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
162 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| T5-11B | 91.2 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 2-L | 90.9 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-L | 90.9 | N/A | Independent | medium | N/A | arxiv.org |
| T5-3B | 89.9 | N/A | Independent | medium | N/A | arxiv.org |
| Inflection-1 | 89.7 | N/A | Independent | medium | N/A | arxiv.org |
| Palmyra X | 89.6 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| StableBeluga2 | 89.4 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon-180B | 89.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| GPT-4o-mini | 88.7 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 540B | 88.7 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 2-M | 88.6 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| Llama 2 70B | 88.6 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| PaLM 2-M | 88.6 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 2-S | 88.1 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-S | 88.1 | N/A | Independent | medium | N/A | arxiv.org |
| text-davinci-003 | 88.1 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| text-davinci-002 | 87.7 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| InternLM 20B | 87.5 | N/A | Independent | medium | N/A | arxiv.org |
| Mistral-7B-v0.1 | 87.4 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| LLaMA 65B | 87.1 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| GPT-3.5 Turbo 0613 | 87.0 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| Qwen-14B | 86.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 33B | 86.1 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| gemini-1.5-flash-001 | 85.8 | N/A | Independent | medium | N/A | arxiv.org |
| Gemma 2 9B IT | 85.7 | N/A | Independent | medium | N/A | arxiv.org |
| Cohere Command beta | 85.6 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| T5-Large | 85.4 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 65B | 85.3 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 65B | 85.3 | N/A | Self-reported | medium | N/A | arxiv.org |
| Llama 2 70B | 85.0 | N/A | Independent | medium | Jul 18, 2023 | arxiv.org |
| Llama 2 70B | 85.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| MPT 30B | 85.0 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| Megatron-Turing NLG 530B | 84.8 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 62B | 84.8 | N/A | Independent | medium | N/A | arxiv.org |
| Phi-3.5-MoE-instruct | 84.6 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM-cont 62B | 83.9 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 2 34B | 83.7 | N/A | Independent | medium | Jul 18, 2023 | arxiv.org |
| Chinchilla-70B | 83.7 | N/A | Independent | medium | Mar 29, 2022 | arxiv.org |
| Chinchilla | 83.7 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 2 34B | 83.7 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 2 34B | 83.7 | N/A | Independent | medium | N/A | arxiv.org |
| Vicuna 13B V1.1 | 83.5 | N/A | Independent | medium | N/A | arxiv.org |
| Gemma 7B | 83.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| Mistral-7B-Instruct-v0.2 | 83.2 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 33B | 83.1 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| Falcon 40B | 83.1 | N/A | Independent | medium | N/A | arxiv.org |
| GLaM 64B/64E | 83.1 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 33B | 83.1 | N/A | Self-reported | medium | N/A | arxiv.org |
| Falcon-Instruct | 82.9 | N/A | Independent | medium | N/A | crfm.stanford.edu |
| Jurassic-2 Jumbo | 82.9 | N/A | Independent | medium | N/A | crfm.stanford.edu |
Top 50 of 162 by score
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Reading comprehension dataset that queries for complex, non-factoid information, and require difficult entailment-like inference to solve.