BrickBench
Harbor: pendingA benchmark of 300 text prompts for agentic LEGO-set design, evaluating physical validity, semantic alignment, and design quality across three complexity settings.
- Samples
- 300
- Released
- 2026
- Scores
- 11
CompMat-Bench
CompMat-Bench: Benchmarking AI Agents for Computational Materials Science
A benchmark of 94 tasks from 15 published materials science studies evaluating AI agents on input preparation and output analysis without running expensive simulations.
How to run
ltb · compmat-bench
$ npm i -g lettokenburn-cli$ ltb bench info compmat-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5.6 Sol | 90.4 | matclaw | Independent | medium | Sep 30, 2026 | arxiv.org |
| Gemini 3.8 Flash | 89.4 | matclaw | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V4.1 Flash | 66.0 | matclaw | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
CompMat-Bench comprises 94 tasks derived from peer-reviewed computational materials studies. It evaluates agents on preparing inputs and analyzing outputs for expensive simulations, using pre-computed ground truth to avoid runtime costs.
Scoring uses fixed rules with numerical tolerances, avoiding LLM judges. Agents are tested under four conditions varying methodological guidance and task span (single vs. workflow), revealing that scientific errors dominate failures over software usage issues.