MAGMA-Bench
MAGMA-Bench: Evaluating Long-Horizon Embodied Agents under Dynamic Tasks and Execution Failures
An architecture-agnostic benchmark for persistent task reasoning in dynamic multi-robot environments, featuring 1200+ episodes, controlled perturbations, and heterogeneous robot coordination.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info magma-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
4 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini-robotic-er-2.0 | 79.3 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| gpt-oss-20b | 40.4 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Qwen3 4B | 10.6 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| LLama-3 8B | 0.0 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
MAGMA-Bench evaluates long-horizon embodied agents across 12 scenario families with 1-3 robots, separating task-level reasoning from perception. It includes 1200+ episodes with online perturbations like execution failures and goal interruptions.
The benchmark measures success rates under dynamic conditions, revealing that even strong models struggle with long horizons and multi-robot coordination. It releases 261,646 training examples and an expert-trace generation pipeline.