LetTokenBurnMASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
A benchmark evaluating LLM-based multi-agent collaboration mechanisms (Protocol, Memory, Routing) under partial observability constraints using deterministic metrics.
$ npm i -g lettokenburn-cli$ ltb bench info masbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | 26.1 | N/A | Independent | medium | Oct 3, 2026 | arxiv.org |
| DeepSeek V3.2 | 22.5 | N/A | Independent | medium | Oct 3, 2026 | arxiv.org |
| Gemini 2.5 Flash Lite | 17.0 | N/A | Independent | medium | Oct 3, 2026 | arxiv.org |
| Qwen Plus (2025-01-25) | 13.5 | N/A | Independent | medium | Oct 3, 2026 | arxiv.org |
| GPT-5 nano (2025-08-07) | 1.0 | N/A | Independent | medium | Oct 3, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
MASBench features three progressive task categories: Reasoning, Scheduling, and Game. It evaluates collaboration mechanisms like protocol, memory, and routing in partially observable environments.
Scoring uses deterministic metrics: performance score, communication cost, and communication efficiency. This avoids LLM-as-a-judge biases, ensuring reproducibility and fair comparison of collaboration outcomes and overhead.