MMLU
Harbor adapterMMLU evaluates English language models on knowledge and reasoning across 57 diverse subjects using a multiple-choice format with over 14k samples.
- Samples
- 14,042
- Released
- 2020
- Scores
- 209
LOGIC
LOGIC: An LLM Benchmark for Intent-Grounded Change Impact in Aerospace Electrical Systems
A 168-scenario benchmark evaluating LLMs on grounding engineering requests to specific changes in aerospace electrical systems before impact propagation.
How to run
ltb · logic
$ npm i -g lettokenburn-cli$ ltb bench info logic
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 0.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Llama-3.1-8B-Instruct | 0.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Mistral 7B | 0.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
LOGIC comprises 168 controlled scenarios (144 selection, 24 abstention) involving aerospace electrical design revisions with concurrent changes. It separates intent-grounded candidate selection from deterministic graph propagation.
Scoring uses F1 for candidate selection and accuracy for propagation/abstention. LLMs outperform lexical matching on relational paraphrases but struggle with abstention and conflicting evidence.