The COLOSSEUM
Harbor: pendingTo realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions.
Auto57%Coding
- Samples
- N/A
- Released
- 2024
- Scores
- 0
The Pile
The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.
How to run
ltb · the-pile
$ npm i -g lettokenburn-cli$ ltb bench pull the-pile
External harness. pip install lm-eval && lm_eval --model <hf|vllm|openai-chat-completions> --model_args <model args> --tasks pile_arxiv (lm-evaluation-harness task family 'pile') https://github.com/EleutherAI/lm-evaluation-harness
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
10 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-3 | 0.7 | N/A | Independent | medium | Oct 5, 2022 | arxiv.org |
| Gemma-2 2B | 0.7 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
| Llama-3.2 1B | 0.7 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
| Phi-3 3.8B | 0.7 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
| Phi-3 7B | 0.7 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
| Gemma 2 9B IT | 0.7 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
| Phi-3 14B | 0.7 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
| Llama-3.2 3B | 0.6 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
| GLM-130B | 0.6 | N/A | Independent | medium | Oct 5, 2022 | arxiv.org |
| Gemma-2 27B | 0.6 | N/A | Independent | medium | Oct 10, 2024 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. To score well on Pile BPB (bits per byte), a model must be able to understand many disparate domains including books, github repositories, webpages, chat logs, and medical, physics, math, computer science, and philosophy papers.
Available in EleutherAI lm-evaluation-harness as task family pile (22 tasks).