Model
Claude 3.5 Sonnet
Anthropic · closed weights
Knowledge & Reasoning
Knowledge & Reasoning| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| Balrog | 32.6 | average_progress | N/A | Independent | medium | balrogai.com |
| EnigmaEval | 0.9 | accuracy | N/A | Independent | medium | labs.scale.com |
| GPQA Diamond | 55.3 | accuracy | N/A | Independent | high | epoch.ai |
| HLE | 4.1 | accuracy | N/A | Independent | low | epoch.ai |
| LiveBench | 59.0 | global_average | N/A | Independent | medium | livebench.ai |
| MapEval-Textual | 66.3 | accuracy_(%_) | N/A | Independent | medium | arxiv.org |
| MMLU | 87.3 | exact_match | N/A | Independent | medium | crfm.stanford.edu |
Math
Math| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| FrontierMath | 1.0 | accuracy | N/A | Independent | medium | paperswithcode.com |
Coding
Coding| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| Aider Polyglot | 51.6 | percent_correct | N/A | Independent | medium | aider.chat |
| GSO | 4.6 | score_opt@1 | OpenHands | Independent | medium | gso-bench.github.io |
| HumanEval | 90.8 | pass@1 | N/A | Independent | medium | paperswithcode.com |
| Web-Bench | 23.0 | pass@1 | N/A | Independent | medium | arxiv.org |
| WebApp1k-Duo-React | 67.9 | pass@1 | N/A | Independent | medium | arxiv.org |
| WebApp1K-React | 88.1 | pass@1 | N/A | Independent | medium | arxiv.org |
Agentic / Tool Use
Agentic / Tool Use| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| TheAgentCompany | 24.0 | resolved_rate | N/A | Independent | medium | github.com |
Multimodal
Multimodal| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| GeoBench | 62.0 | acw_country_% | N/A | Independent | medium | geobench.org |
| MapEval-Visual | 61.6 | accuracy_(%_) | N/A | Independent | medium | arxiv.org |
| Video-MME | 60.0 | overall_(no_subtitles) | N/A | Independent | medium | video-mme.github.io |
| Vinoground | 32.8 | text_score | N/A | Independent | medium | paperswithcode.com |
Instruction Following
Instruction Following| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| Intent-Eval | 91.7 | accuracy | N/A | Independent | medium | arxiv.org |
Safety & Alignment
Safety & Alignment| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| Cybench | 17.5 | unguided_%_solved | N/A | Independent | medium | cybench.github.io |
| Cybench | 17.5 | unguided_performance | N/A | Independent | medium | arxiv.org |
Reproduced by the community
Loading community runs…