SimpleQA
Simple QA (OpenAI)
SimpleQA is a benchmark containing 4,326 samples designed to measure the short-form factuality and knowledge accuracy of large language models.
How to run
$ npm i -g lettokenburn-cli$ ltb bench run openai/simpleqa@1 --benchmark simpleqa -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Harbor-native: you can also call Harbor directly.
$ harbor run -d openai/simpleqa@1 -a claude-code -m anthropic/claude-opus-4-8
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
0 sourced entriesNo sourced scores recorded yet.
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Overview: SimpleQA evaluates how well LLMs answer short factual questions without hallucinating incorrect details.
Scoring & Importance: The dataset uses exact-match accuracy as its primary metric. It matters because it provides a rigorous test for the 'short-form factuality' axis, ensuring models remain grounded in reality when answering concise queries.