CARAT
Controlled Ablation of structural Representations with Anti-shortcut Testing
A benchmark evaluating materials LLMs' ability to reason over crystal structures versus reciting answers from input text using eight matched evidence views and anti-shortcut testing.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info carat
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
2 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Pointer-SFT | 99.8 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Frozen | 32.3 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: CARAT evaluates Large Language Models on crystalline material properties by fixing questions, gold answers, and difficulty while varying structural evidence across eight matched views (e.g., chemical formula, raw CIF, GraphSpace). It constructs explicit site, neighbour, and orbit relations to isolate specific evidence contributions.
Scoring & Key Findings: The benchmark uses paired statistical layers with material-cluster bootstrap intervals. A key finding is that a 27B model shows an aggregate advantage of 17.3 percentage points over formula inputs on the hardest families when using GraphSpace, but this gap vanishes or changes significantly under masking and injection tests.