PutnamBench
Harbor: pendingWe present a multilingual benchmark for formal-theorem proving of problems from the William Lowell Putnam Mathematical Competition.
Incomplete29%Math
- Samples
- N/A
- Released
- 2024
- Scores
- 0
claimcheck
claimcheck: Measuring and Improving Specification Faithfulness
A 3,234-row benchmark measuring spec formalization faithfulness across Dafny, Lean 4, Isabelle, and Rocq by pairing natural-language requirements with formal artifacts.
How to run
ltb · claimcheck
$ npm i -g lettokenburn-cli$ ltb bench info claimcheck
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entriesEvery number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The benchmark pairs natural-language requirements with formal artifacts in four languages, including meaning-preserving controls to prevent mutation detection shortcuts. Labels are established via execution, proof, or source benchmarks.
Scoring uses macro recall to measure a judge's ability to detect discrepancies between requirements and specifications. The proposed claimcheck tool improves recall by 4.2–6.0 points over informed baselines by separating informalization from judgment.