LetTokenBurn$ npm i -g lettokenburn-cli$ ltb bench pull mbpp --n 50$ ltb bench run ./mbpp -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude 3 Opus | 86.4 | N/A | Independent | medium | Mar 4, 2024 | anthropic.com |
| Claude 3 Haiku | 80.4 | N/A | Independent | medium | Mar 4, 2024 | anthropic.com |
| GPT-4 | 80.0 | N/A | Independent | medium | Jan 25, 2024 | arxiv.org |
| Claude 3 Sonnet | 79.4 | N/A | Independent | medium | Mar 4, 2024 | anthropic.com |
| Claude | 71.4 | N/A | Independent | medium | Jul 24, 2023 | arxiv.org |
| GPT-3.5 Turbo | 70.8 | N/A | Independent | medium | Jan 25, 2024 | arxiv.org |
| DeepSeek-Coder-Instruct 33B | 70.0 | N/A | Independent | medium | Jan 25, 2024 | arxiv.org |
| GPT-3.5 Turbo | 67.6 | N/A | Independent | medium | Apr 11, 2023 | arxiv.org |
| StarCoder2-15B | 66.2 | N/A | Independent | medium | Feb 29, 2024 | arxiv.org |
| Code Llama - Python 70B | 65.5 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| DeepSeek-Coder-Instruct 6.7B | 65.4 | N/A | Independent | medium | Jan 25, 2024 | arxiv.org |
| Code Llama 70B | 62.4 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| code-davinci-002 175B | 61.4 | N/A | Independent | medium | Apr 11, 2023 | arxiv.org |
| Mixtral 8x7B | 60.7 | N/A | Independent | medium | Jan 8, 2024 | arxiv.org |
| Code Llama - Instruct 34B | 57.0 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| Code Llama - Python 34B | 56.2 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| StarCoder 15.5B | 52.7 | N/A | Independent | medium | May 9, 2023 | arxiv.org |
| GPT-3.5 Turbo | 52.2 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| WizardCoder 15B | 51.8 | N/A | Independent | medium | Jun 14, 2023 | arxiv.org |
| PaLM 2-S | 50.0 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| DeepSeek-Coder-Instruct 1.3B | 49.4 | N/A | Independent | medium | Jan 25, 2024 | arxiv.org |
| Code Llama - Instruct 13B | 49.4 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| Code Llama - Python 13B | 49.0 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| Qwen2idae-16x14B | 48.6 | N/A | Independent | medium | Jan 5, 2024 | arxiv.org |
| Code Llama - Python 7B | 47.6 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| Mistral 7B | 47.5 | N/A | Independent | medium | Oct 10, 2023 | arxiv.org |
| StarCoder 15.5B | 47.2 | N/A | Independent | medium | Apr 11, 2023 | arxiv.org |
| PaLM Coder 540B | 47.0 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| GPT-3.5 Turbo | 45.4 | N/A | Independent | medium | Nov 16, 2023 | arxiv.org |
| Llama 2 70B | 45.0 | N/A | Independent | medium | Jul 18, 2023 | arxiv.org |
| Code Llama - Instruct 7B | 44.4 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| phi-1.5-web 1.3B | 43.5 | N/A | Independent | medium | Sep 11, 2023 | arxiv.org |
| GPT-3.5 Turbo | 39.8 | N/A | Independent | medium | Nov 16, 2023 | arxiv.org |
| LLaMA 65B | 37.7 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| PaLM 540B | 36.8 | N/A | Independent | medium | Apr 5, 2022 | arxiv.org |
| SantaCoder 1.1B | 35.0 | N/A | Independent | medium | May 9, 2023 | arxiv.org |
| LLaMA 2 34B | 33.0 | N/A | Independent | medium | Jul 18, 2023 | arxiv.org |
| LLaMA 2 13B | 30.6 | N/A | Independent | medium | Jul 18, 2023 | arxiv.org |
| LLaMA 33B | 30.2 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| CodeGeeX-13B | 24.4 | N/A | Independent | medium | Mar 30, 2023 | arxiv.org |
| LLaMA 13B | 22.0 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 2 7B | 20.8 | N/A | Independent | medium | Jul 18, 2023 | arxiv.org |
| InCoder 6.7B | 19.4 | N/A | Independent | medium | Apr 12, 2022 | arxiv.org |
| LLaMA 7B | 17.7 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
MBPP (Mostly Basic Python Problems) — Python编程基础. Scoring: Score. Difficulty: 低. Imported from benchmark_research.md (2026-06).