ARC-Challenge
Harbor: pendingARC-Challenge — AI evaluation benchmark listed by opencompass.
Incomplete14%Knowledge & Reasoning
- Samples
- N/A
- Released
- N/A
- Scores
- 26
How to run
ltb · math
$ npm i -g lettokenburn-cli$ ltb bench pull math --n 50$ ltb bench run ./math -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
46 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini 2.0 Flash Experimental | 89.7 | N/A | Independent | medium | N/A | paperswithcode.com |
| Qwen2.5-Math-72B-Instruct | 85.9 | N/A | Independent | medium | Sep 18, 2024 | arxiv.org |
| Qwen2-Math-72B-Instruct | 84.0 | N/A | Independent | medium | Jul 15, 2024 | arxiv.org |
| Qwen2.5-Math-7B-Instruct | 83.6 | N/A | Independent | medium | Sep 18, 2024 | arxiv.org |
| Qwen2.5-Math-1.5B-Instruct | 75.8 | N/A | Independent | medium | Sep 18, 2024 | arxiv.org |
| Gemini Ultra | 53.2 | N/A | Independent | medium | Dec 19, 2023 | arxiv.org |
| DeepSeekMath-7B-KPMath-Plus | 48.8 | N/A | Independent | medium | Mar 4, 2024 | arxiv.org |
| Llemma-34B-KPMath-Plus | 48.6 | N/A | Independent | medium | Mar 4, 2024 | arxiv.org |
| Mistral-7B-KPMath-Plus | 46.8 | N/A | Independent | medium | Mar 4, 2024 | arxiv.org |
| GPT-4 | 42.5 | N/A | Independent | medium | Mar 22, 2023 | arxiv.org |
| Llama2-13B-KPMath-Plus | 41.0 | N/A | Independent | medium | Mar 4, 2024 | arxiv.org |
| PaLM 2 | 34.3 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| Minerva 540B mCoT | 33.6 | N/A | Independent | medium | Nov 16, 2022 | arxiv.org |
| Minerva 540B | 33.6 | N/A | Independent | medium | Jun 29, 2022 | arxiv.org |
| WizardMath-7B-V1.1 | 33.0 | N/A | Independent | medium | Aug 18, 2023 | arxiv.org |
| Gemini Pro | 32.6 | N/A | Independent | medium | Dec 19, 2023 | arxiv.org |
| Qwen2idae-16x14B | 29.9 | N/A | Independent | medium | Jan 5, 2024 | arxiv.org |
| Mixtral 8x7B | 28.4 | N/A | Independent | medium | Jan 8, 2024 | arxiv.org |
| Minerva 62B | 27.6 | N/A | Independent | medium | Jun 29, 2022 | arxiv.org |
| MetaMath 70B | 26.0 | N/A | Independent | medium | Sep 21, 2023 | arxiv.org |
| WizardMath-70B-V1.0 | 22.7 | N/A | Independent | medium | Aug 18, 2023 | arxiv.org |
| MetaMath 13B | 22.5 | N/A | Independent | medium | Sep 21, 2023 | arxiv.org |
| MetaMath 7B | 19.4 | N/A | Independent | medium | Sep 21, 2023 | arxiv.org |
| davinci-002 175B | 19.1 | N/A | Independent | medium | Jun 29, 2022 | arxiv.org |
| LLaMA 33B-maj1@k | 15.2 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| Minerva 8B | 14.1 | N/A | Independent | medium | Jun 29, 2022 | arxiv.org |
| WizardMath-13B-V1.0 | 14.0 | N/A | Independent | medium | Aug 18, 2023 | arxiv.org |
| Mistral 7B | 13.1 | N/A | Independent | medium | Oct 10, 2023 | arxiv.org |
| Mistral 7B | 12.7 | N/A | Independent | medium | Jan 8, 2024 | arxiv.org |
| WizardMath-7B-V1.0 | 10.7 | N/A | Independent | medium | Aug 18, 2023 | arxiv.org |
| LLaMA 65B | 10.6 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 13B-maj1@k | 8.8 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| PaLM 540B mCoT | 8.8 | N/A | Independent | medium | Nov 16, 2022 | arxiv.org |
| PaLM 540B | 8.8 | N/A | Independent | medium | Jun 29, 2022 | arxiv.org |
| LLaMA 33B | 7.1 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 7B-maj1@k | 6.9 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| GPT-2 | 6.9 | N/A | Independent | medium | Mar 5, 2021 | arxiv.org |
| GPT-3 13B | 5.6 | N/A | Independent | medium | Mar 5, 2021 | arxiv.org |
| GPT-3-175B | 5.2 | N/A | Independent | medium | Nov 16, 2022 | arxiv.org |
| GPT-3-175B | 5.2 | N/A | Independent | medium | Mar 5, 2021 | arxiv.org |
| PaLM 62B | 4.4 | N/A | Independent | medium | Jun 29, 2022 | arxiv.org |
| LLaMA 13B | 3.9 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| GPT-3 13B | 3.0 | N/A | Independent | medium | Mar 5, 2021 | arxiv.org |
| LLaMA 7B | 2.9 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| GPT-3 2.7B | 2.9 | N/A | Independent | medium | Mar 5, 2021 | arxiv.org |
| PaLM 8B | 1.5 | N/A | Independent | medium | Jun 29, 2022 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
MATH (MATH Dataset) — 竞赛级数学（AMC/AIME等）. Scoring: 准确率. Difficulty: 高. Imported from benchmark_research.md (2026-06).