LetTokenBurn
Sign in
Verified86%Harbor adapterCoding

CRUXEval

CRUXEval is a benchmark of 800 Python functions paired with input-output pairs.

PaperDatasetHomepage

How to run

ltb · cruxeval
$ npm i -g lettokenburn-cli
$ ltb bench pull cruxeval --n 50
$ ltb bench run ./cruxeval -a claude-code -m anthropic/claude-opus-4-8
$ ltb bench push

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

0 sourced entries

No sourced scores recorded yet.

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

CRUXEval is a benchmark of 800 Python functions paired with input-output pairs. It evaluates a model's ability to reason about code execution in two directions: given a function and input, predict the output (cruxeval_output); and given a function and output, find a valid input (cruxeval_input). The benchmark includes chain-of-thought variants that prompt the model to reason step by step before answering.

Available in EleutherAI lm-evaluation-harness as task family cruxeval (6 tasks).