LetTokenBurn
Sign in
Verified71%Harbor: pendingKnowledge & ReasoningMath

ChemBench

Are large language models superhuman chemists?

ChemBench is designed to reveal limitations of current frontier models for use in the chemical sciences.

PaperDataset

How to run

ltb · chembench
$ npm i -g lettokenburn-cli
$ ltb bench pull chembench

External harness. pip install inspect-evals && inspect eval inspect_evals/chembench --model <provider/model> https://github.com/UKGovernmentBEIS/inspect_evals

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

0 sourced entries

No sourced scores recorded yet.

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

ChemBench is designed to reveal limitations of current frontier models for use in the chemical sciences. It consists of 2786 question-answer pairs compiled from diverse sources. Our corpus measures reasoning, knowledge and intuition across a large fraction of the topics taught in undergraduate and graduate chemistry curricula. It can be used to evaluate any system that can return text (i.e., including tool-augmented systems).