ARC-Challenge
Harbor: pendingARC-Challenge — AI evaluation benchmark listed by opencompass.
Incomplete14%Knowledge & Reasoning
- Samples
- N/A
- Released
- N/A
- Scores
- 26
LAMBADA
LAMBADA is a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task.
How to run
ltb · lambada
$ npm i -g lettokenburn-cli$ ltb bench pull lambada
External harness. pip install lm-eval && lm_eval --model <hf|vllm|openai-chat-completions> --model_args <model args> --tasks lambada_openai (lm-evaluation-harness task family 'lambada') https://github.com/EleutherAI/lm-evaluation-harness
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
56 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| PaLM 540B | 89.7 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| Megatron-Turing NLG 530B | 87.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-L | 86.9 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| GLaM 64B/64E | 86.6 | N/A | Self-reported | medium | N/A | arxiv.org |
| GPT-3-175B | 86.4 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| text-davinci-001 | 86.4 | N/A | Independent | medium | N/A | arxiv.org |
| GLaM 137B | 84.2 | N/A | Self-reported | medium | N/A | arxiv.org |
| PaLM 2-M | 83.7 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 540B | 81.8 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 2-S | 80.7 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| Falcon-180B | 79.8 | N/A | Self-reported | medium | N/A | arxiv.org |
| GLaM 8B/64E | 79.0 | N/A | Self-reported | medium | N/A | arxiv.org |
| Llama 2 70B | 78.9 | N/A | Independent | medium | N/A | arxiv.org |
| Inflection-1 | 78.5 | N/A | Independent | medium | N/A | arxiv.org |
| PaLM 540B | 77.9 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| PaLM 540B | 77.9 | N/A | Independent | medium | N/A | arxiv.org |
| Chinchilla | 77.7 | N/A | Independent | medium | Mar 29, 2022 | arxiv.org |
| LLaMA 65B | 77.7 | N/A | Independent | medium | N/A | arxiv.org |
| Chinchilla | 77.4 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon 40B | 77.3 | N/A | Self-reported | medium | N/A | arxiv.org |
| LLaMA 33B | 77.2 | N/A | Independent | medium | N/A | arxiv.org |
| GLaM 8B | 77.1 | N/A | Self-reported | medium | N/A | arxiv.org |
| Megatron-Turing NLG 530B | 76.6 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 2 13B | 76.5 | N/A | Independent | medium | N/A | arxiv.org |
| GPT-3-175B | 76.2 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| GPT-3-175B | 76.2 | N/A | Independent | medium | N/A | arxiv.org |
| OPT-175B | 75.6 | N/A | Independent | medium | Jan 2, 2023 | arxiv.org |
| LLaMA 13B | 75.2 | N/A | Independent | medium | N/A | arxiv.org |
| Falcon 7B | 74.9 | N/A | Self-reported | medium | N/A | arxiv.org |
| Gopher | 74.5 | N/A | Self-reported | medium | N/A | arxiv.org |
| Gopher | 74.5 | N/A | Independent | medium | N/A | arxiv.org |
| Baichuan-2-13B-Base | 74.0 | N/A | Independent | medium | N/A | arxiv.org |
| Baichuan-2-7B-Base | 73.3 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 2 7B | 73.3 | N/A | Independent | medium | N/A | arxiv.org |
| LLaMA 7B | 73.3 | N/A | Independent | medium | N/A | arxiv.org |
| GPT-3 13B | 72.5 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| InternLM 20B | 71.8 | N/A | Independent | medium | N/A | arxiv.org |
| StableBeluga2 | 71.3 | N/A | Independent | medium | N/A | arxiv.org |
| Qwen-14B | 71.1 | N/A | Self-reported | medium | N/A | arxiv.org |
| Pythia 12B | 70.5 | N/A | Independent | medium | Apr 3, 2023 | arxiv.org |
| GPT-3 6.7B | 70.3 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| MPT 7B | 70.0 | N/A | Independent | medium | N/A | arxiv.org |
| GPT-J-6B | 69.7 | N/A | Independent | medium | N/A | paperswithcode.com |
| Qwen 7B | 67.9 | N/A | Self-reported | medium | N/A | arxiv.org |
| Pythia 6.9B | 67.3 | N/A | Independent | medium | Apr 3, 2023 | arxiv.org |
| GPT-3 2.7B | 67.1 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
| InternLM 7B | 67.0 | N/A | Independent | medium | N/A | arxiv.org |
| GLaM 1.7B/64E | 64.3 | N/A | Self-reported | medium | N/A | arxiv.org |
| GPT-2 1.5B | 63.2 | N/A | Independent | medium | Feb 14, 2019 | d4mucfpksywv.cloudfront.net |
| GLaM 1.7B | 63.0 | N/A | Self-reported | medium | N/A | arxiv.org |
Top 50 of 56 by score
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
LAMBADA is a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse.
Available in EleutherAI lm-evaluation-harness as task family lambada (2 tasks).