FREA
FREA: A Multi-Source Expert Benchmark for Reaction Feasibility Verification
Expert-labeled benchmark of 751 chemical reactions from four sources to evaluate reaction feasibility verifiers.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info frea
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
5 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 68.0 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| BARTSmiles | 65.9 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| GPT-6 Astra | 65.7 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Descriptor GBDT | 63.2 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Chemformer | 62.8 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
FREA comprises 751 reactions labeled by expert chemists as feasible or infeasible. Sources include retrosynthesis model proposals, zero-yield records, LLM edits, and generated negatives.
It evaluates verifier agreement with experts using balanced accuracy and AUROC. Findings show no single verifier leads across all sources, with forward models struggling on alternative disconnections.