LetTokenBurn
Sign in
Auto57%Harbor: pendingMath

Prim

The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models - Prim Benchmark

A benchmark evaluating mathematical reasoning across four dimensions (Discovery, Generation, Digestion, Execution) using the Humanity's Last Exam dataset.

PaperHomepage

How to run

ltb · prim
$ npm i -g lettokenburn-cli
$ ltb bench info prim

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

6 sourced entries
HarnessSourceConfidenceLink
GPT-5.467.0N/AIndependentmediumOct 1, 2026arxiv.org
Qwen3.6 27B52.8N/AIndependentmediumOct 1, 2026arxiv.org
GPT-5.4-mini50.5N/AIndependentmediumOct 1, 2026arxiv.org
qwen3.5-27B47.8N/AIndependentmediumOct 1, 2026arxiv.org
GPT-5.4-nano44.5N/AIndependentmediumOct 1, 2026arxiv.org
gpt-oss-20b43.4N/AIndependentmediumOct 1, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Content & Construction: The paper introduces 'Prim', a novel benchmark built on the mathematics subset of Humanity’s Last Exam. It evaluates Large Language Models' structural understanding by probing their ability to discover mathematical primitives independently (Discovery), generate solutions directly (Generation), extract underlying structures from provided proofs (Digestion), and execute problems given explicit primitive guidance (Execution).

Scoring & Key Findings: Performance is measured via exact match on final answers. The benchmark reveals that solution accuracy masks distinct capability profiles, identifies 'Discovery' as the dominant bottleneck in mathematical reasoning, and shows that discovery-limited failures are particularly amenable to post-training repair.