Verified100%Harbor adapterKnowledge & Reasoning Math

MMLU

Massive Multitask Language Understanding

MMLU evaluates English language models on knowledge and reasoning across 57 diverse subjects using a multiple-choice format with over 14k samples.

Paper Repo Dataset Leaderboard Homepage

How to run

ltb · mmlu

$ npm i -g lettokenburn-cli
$ ltb bench pull mmlu --n 50
$ ltb bench run ./mmlu -a claude-code -m anthropic/claude-opus-4-8
$ ltb bench push

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

209 sourced entries

		Harness	Source	Confidence		Link
GPT-4o	88.1	N/A	Independent	medium	N/A	arxiv.org
Claude 3.5 Sonnet	87.3	N/A	Independent	medium	N/A	crfm.stanford.edu
DeepSeek-V3	87.2	N/A	Independent	medium	N/A	crfm.stanford.edu
DeepSeek-V3	87.1	N/A	Self-reported	medium	N/A	arxiv.org
GPT-4 o1	87.0	N/A	Independent	medium	Oct 3, 2024	arxiv.org
gemini-1.5-pro-002	86.9	N/A	Independent	medium	N/A	crfm.stanford.edu
Llama 3.1	86.6	N/A	Independent	medium	Dec 13, 2024	arxiv.org
GPT-4 0314	86.4	N/A	Self-reported	medium	N/A	arxiv.org
Llama-3.3-70B-Instruct	86.3	N/A	Independent	medium	N/A	arxiv.org
gemini-1.5-pro-001	85.9	N/A	Self-reported	medium	N/A	storage.googleapis.com
Qwen2.5-72B-Instruct	85.3	N/A	Independent	medium	N/A	arxiv.org
Phi-4	84.8	N/A	Self-reported	medium	N/A	arxiv.org
Claude 3 Opus	84.6	N/A	Independent	medium	N/A	crfm.stanford.edu
Llama-3.1-405B-Instruct	84.5	N/A	Independent	medium	N/A	crfm.stanford.edu
GPT-4	83.9	N/A	Independent	medium	N/A	arxiv.org
Gemini Ultra	83.7	N/A	Independent	medium	N/A	paperswithcode.com
Gemini 1.5 Pro 001 Feb24	82.7	N/A	Independent	medium	N/A	crfm.stanford.edu
GPT-4-0613	82.4	N/A	Independent	medium	N/A	crfm.stanford.edu
Qwen2 72B Instruct	82.4	N/A	Independent	medium	N/A	crfm.stanford.edu
amazon.nova-pro-v1:0	82.0	N/A	Independent	medium	N/A	crfm.stanford.edu
Gemini 1.5 Pro 001 Feb24	81.9	N/A	Self-reported	medium	N/A	storage.googleapis.com
GPT-4o-mini	81.8	N/A	Independent	medium	N/A	arxiv.org
Gemini 1.5 Pro	81.0	N/A	Independent	medium	N/A	crfm.stanford.edu
Llama-3.2-90B-Vision-Instruct	80.3	N/A	Independent	medium	N/A	crfm.stanford.edu
Llama-3.1-70B-Instruct	80.1	N/A	Independent	medium	N/A	crfm.stanford.edu
Mistral Large 2407	80.0	N/A	Independent	medium	N/A	crfm.stanford.edu
Qwen2.5 14B Instruct	79.9	N/A	Independent	medium	N/A	arxiv.org
Gemini 2.0 Flash Experimental	79.7	N/A	Independent	medium	N/A	crfm.stanford.edu
Meta-Llama-3-70B-Instruct	79.3	N/A	Independent	medium	N/A	crfm.stanford.edu
Yi-large	79.3	N/A	Independent	medium	N/A	crfm.stanford.edu
Qwen2.5-Coder-32B-Instruct	79.1	N/A	Self-reported	medium	N/A	arxiv.org
Claude 3 Sonnet	79.0	N/A	Independent	medium	Mar 4, 2024	anthropic.com
PaLM 2	78.6	N/A	Independent	medium	N/A	crfm.stanford.edu
Palmyra X V3	78.6	N/A	Independent	medium	N/A	crfm.stanford.edu
Claude 2	78.5	N/A	Self-reported	medium	N/A	www-cdn.anthropic.com
DeepSeek-V2	78.4	N/A	Self-reported	medium	N/A	arxiv.org
Jamba 1.5 Large	78.2	N/A	Independent	medium	N/A	crfm.stanford.edu
Phi-3-medium-128k-instruct	78.0	N/A	Self-reported	medium	N/A	arxiv.org
gemini-1.5-flash-001	77.9	N/A	Independent	medium	N/A	crfm.stanford.edu
Gemini 1.5 Flash 0514	77.8	N/A	Independent	medium	N/A	crfm.stanford.edu
Mixtral-8x22B-Instruct-v0.1	77.8	N/A	Independent	medium	N/A	crfm.stanford.edu
Solar Pro	77.6	N/A	Independent	medium	N/A	crfm.stanford.edu
Phi-3-medium-128k-instruct	77.5	N/A	Independent	medium	N/A	crfm.stanford.edu
Qwen1.5 72B	77.5	N/A	Independent	medium	N/A	paperswithcode.com
Qwen1.5	77.4	N/A	Independent	medium	N/A	crfm.stanford.edu
amazon.nova-lite-v1:0	77.0	N/A	Independent	medium	N/A	crfm.stanford.edu
Claude 1.3	77.0	N/A	Self-reported	medium	N/A	www-cdn.anthropic.com
Yi 34B	76.3	N/A	Independent	medium	N/A	arxiv.org
Claude 3 Sonnet	75.9	N/A	Independent	medium	N/A	crfm.stanford.edu
Gemma-2 27B	75.7	N/A	Independent	medium	N/A	crfm.stanford.edu

Top 50 of 209 by score

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs

About

Massive Multitask Language Understanding (MMLU) is a comprehensive benchmark designed to assess the capabilities of large language models. It consists of approximately 14,000 multiple-choice questions spanning 57 academic disciplines and logical reasoning tasks.

The dataset is scored using exact-match accuracy on these standardized questions. MMLU has become an industry standard for measuring general knowledge and reasoning proficiency in English-speaking AI systems.

Facts

Metric: accuracy
Scale: 0–100
Scoring: exact-match accuracy
Format: Multiple choice
Samples: 14,042
Difficulty: Medium
Saturation: Saturated
Languages: en
Modalities: text
Industries: Education, Science & R&D, General
Released: Sep 7, 2020
HF dataset: cais/mmlu

Data completeness

Verified100%

Verified: Human-reviewed, or confirmed by 2+ independent sources with most fields filled.

All key fields present.

Confirmed by 9 source channels. Fields are extracted automatically and can be wrong.

Report a correction ([email protected])

License

MITredistributable

Citation. Cite “Measuring Massive Multitask Language Understanding” (2020), arXiv:2009.03300.

Index record

Review status: reviewed

Extraction confidence: 100%

Updated: Oct 1, 2026

JSON record

How to run

Scores

Reproduced by the community

About

Related benchmarks

Aider Polyglot

AlgoREval

ANLI

ARC-AGI-2

ARC-Challenge

ARC-Easy