Natural QA
Harbor: pendingQuestion answering from wikipedia.
Incomplete29%Knowledge & Reasoning
- Samples
- N/A
- Released
- N/A
- Scores
- 0
Natural Questions
Natural Questions — AI evaluation benchmark listed by opencompass.
How to run
ltb · natural-questions
$ npm i -g lettokenburn-cli$ ltb bench pull natural-questions
External harness. pip install opencompass && opencompass --models <model_cfg> --datasets nq_gen (OpenCompass dataset config dir 'nq') https://github.com/open-compass/opencompass
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
15 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| LLaMA 65B | 39.9 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| PaLM 540B | 39.6 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 2-L | 37.5 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| Chinchilla | 35.5 | N/A | Independent | medium | Mar 29, 2022 | arxiv.org |
| LLaMA 65B | 35.0 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| Llama 2 70B | 33.0 | N/A | Independent | medium | Jul 18, 2023 | arxiv.org |
| PaLM 2-M | 32.0 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| LLaMA 65B | 31.0 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| GPT-3-175B | 29.9 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| PaLM 540B | 29.3 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| Mistral 7B | 28.8 | N/A | Independent | medium | Oct 10, 2023 | arxiv.org |
| Gopher | 28.2 | N/A | Independent | medium | Dec 8, 2021 | arxiv.org |
| PaLM 2-S | 25.3 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| LLaMA 33B | 24.9 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| PaLM 540B | 21.2 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The Natural Questions corpus is a question answering dataset containing 307,373 training examples, 7,830 development examples, and 7,842 test examples. Each example is comprised of a google.com query and a corresponding Wikipedia page. Each Wikipedia page has a passage (or long answer) annotated on the page that answers the question and one or more short spans from the annotated passage containing the actual answer. The long and the short answer annotations can however be empty. If they are both empty, then there is no answer on the page at all. If the long answer annotation is non-empty, but the short answer annotation is empty, then the annotated passage answers the question but no explicit short answer could be found. Finally 1% of the documents have a passage annotated with a short answer that is “yes” or “no”, instead of a list of short spans.
Source: A BERT Baseline for the Natural Questions Image Source: https://paperswithcode.com/paper/natural-questions-a-benchmark-for-question/