LetTokenBurn
Sign in
Verified100%Harbor adapterKnowledge & Reasoning

SimpleQA Verified

SimpleQA Verified A 1,000-prompt factuality benchmark from Google DeepMind and Google Research, designed to reliably evaluate LLM parametric knowledge.

PaperDatasetLeaderboard

How to run

ltb · simpleqa-verified
$ npm i -g lettokenburn-cli
$ ltb bench pull simpleqa-verified --n 50
$ ltb bench run ./simpleqa-verified -a claude-code -m anthropic/claude-opus-4-8
$ ltb bench push

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

79 sourced entries
HarnessSourceConfidenceLink
GPT-6 Astra75.6N/AIndependenthighAug 30, 2026epoch.ai
GPT-6.1 Sol73.9N/AIndependenthighSep 29, 2026epoch.ai
Gemini 3.1 Pro73.5N/AIndependenthighAug 10, 2026epoch.ai
Claude Opus 5.572.2N/AIndependenthighSep 22, 2026epoch.ai
Claude Fable 5.170.8N/AIndependenthighSep 1, 2026epoch.ai
Claude Fable 570.7N/AIndependenthighAug 10, 2026epoch.ai
Gemini 3.8 Flash69.7N/AIndependenthighSep 2, 2026epoch.ai
GPT-5.6 Sol69.7N/AIndependenthighAug 10, 2026epoch.ai
Gemini 3.7 Flash69.2N/AIndependenthighAug 27, 2026epoch.ai
Gemini 3 Flash Preview66.8N/AIndependenthighAug 27, 2026epoch.ai
Gemini-3.5-flash66.2N/AIndependenthighAug 27, 2026epoch.ai
Gemini 3.6 Flash66.2N/AIndependenthighAug 27, 2026epoch.ai
GPT-5.563.0N/AIndependenthighAug 27, 2026epoch.ai
GPT-6 Sol60.7N/AIndependenthighSep 22, 2026epoch.ai
Muse Spark 1.260.3N/AIndependenthighAug 27, 2026epoch.ai
Claude Opus 559.9N/AIndependenthighAug 10, 2026epoch.ai
Muse Spark 1.157.8N/AIndependenthighAug 31, 2026epoch.ai
Grok-4.756.0N/AIndependenthighSep 22, 2026epoch.ai
Qwen3.7 Max55.8N/AIndependenthighAug 27, 2026epoch.ai
Claude Opus 4.853.0N/AIndependenthighAug 27, 2026epoch.ai
DeepSeek V4 Pro 081352.9N/AIndependenthighAug 27, 2026epoch.ai
Qwen3.6 Max Preview52.0N/AIndependenthighAug 27, 2026epoch.ai
Claude Opus 4.751.7N/AIndependenthighAug 27, 2026epoch.ai
Kimi K350.6N/AIndependenthighAug 27, 2026epoch.ai
GPT-550.1N/AIndependenthighAug 27, 2026epoch.ai
o349.4N/AIndependenthighAug 27, 2026epoch.ai
Grok-4.649.3N/AIndependenthighAug 27, 2026epoch.ai
Qwen3 Max (2025-09-23)48.7N/AIndependenthighAug 27, 2026epoch.ai
Grok 4.548.3N/AIndependenthighAug 27, 2026epoch.ai
GPT-5.148.0N/AIndependenthighAug 27, 2026epoch.ai
Qwen3.8 Max 090247.3N/AIndependenthighSep 2, 2026epoch.ai
Claude Opus 4.647.0N/AIndependenthighAug 27, 2026epoch.ai
DeepSeek V4 Pro47.0N/AIndependenthighAug 27, 2026epoch.ai
Claude Sonnet 5.546.5N/AIndependenthighSep 29, 2026epoch.ai
GPT-5.4 Pro46.3N/AIndependenthighAug 27, 2026epoch.ai
Qwen3.8 Max45.8N/AIndependenthighAug 27, 2026epoch.ai
claude-opus-4-5-20251101-32K45.7N/AIndependenthighAug 27, 2026epoch.ai
GPT-5.445.1N/AIndependenthighAug 27, 2026epoch.ai
Qwen3.6 Plus44.1N/AIndependenthighAug 27, 2026epoch.ai
GPT-5.6-terra43.2N/AIndependenthighAug 10, 2026epoch.ai
GPT-6 Luna41.4N/AIndependenthighSep 22, 2026epoch.ai
OpenAI/o1-2024-12-17-high41.1N/AIndependenthighAug 31, 2026epoch.ai
GLM-5.341.0N/AIndependenthighAug 28, 2026epoch.ai
GPT-5.6 Luna41.0N/AIndependenthighAug 10, 2026epoch.ai
Qwen3-235B-A22B-Thinking-250740.4N/AIndependenthighAug 27, 2026epoch.ai
Inkling40.3N/AIndependenthighAug 27, 2026epoch.ai
GPT-5.2-chat37.1N/AIndependenthighAug 27, 2026epoch.ai
Kimi K2.7 Code36.5N/AIndependenthighAug 27, 2026epoch.ai
Claude Sonnet 4.635.5N/AIndependenthighAug 10, 2026epoch.ai
Kimi K2.634.9N/AIndependenthighAug 10, 2026epoch.ai
Top 50 of 79 by score

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

SimpleQA Verified

A 1,000-prompt factuality benchmark from Google DeepMind and Google Research, designed to reliably evaluate LLM parametric knowledge.

▶ SimpleQA Verified Leaderboard on Kaggle▶ Technical Report▶ Evaluation Starter Code

Benchmark

SimpleQA Verified is a 1,000-prompt benchmark for reliably evaluating Large Language Models (LLMs) on short-form factuality and parametric knowledge. The authors from Google DeepMind and Google Research…