MemCut
MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents
MemCut evaluates user-requested forgetting in LLM agents using 456 hand-designed secrets with annotated evidence trails in long-horizon conversations.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info memcut
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
14 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| DeepSeek Chat | 85.1 | nemori | Independent | medium | Oct 15, 2026 | arxiv.org |
| DeepSeek Chat | 88.8 | lightmem | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen Max (2025-01-25) | 90.4 | lightmem | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen Max (2025-01-25) | 91.7 | nemori | Independent | medium | Oct 15, 2026 | arxiv.org |
| MiMo | 92.3 | lightmem | Independent | medium | Oct 15, 2026 | arxiv.org |
| DeepSeek Chat | 93.0 | memorybank | Independent | medium | Oct 15, 2026 | arxiv.org |
| MiMo | 93.4 | nemori | Independent | medium | Oct 15, 2026 | arxiv.org |
| MiMo | 93.6 | memorybank | Independent | medium | Oct 15, 2026 | arxiv.org |
| GLM | 93.9 | lightmem | Independent | medium | Oct 15, 2026 | arxiv.org |
| GLM | 93.9 | nemori | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen Max (2025-01-25) | 95.8 | memorybank | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen Max (2025-01-25) | 97.8 | turn rag | Independent | medium | Oct 15, 2026 | arxiv.org |
| DeepSeek Chat | 98.2 | turn rag | Independent | medium | Oct 15, 2026 | arxiv.org |
| GLM | 99.3 | memorybank | Independent | medium | Oct 15, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
MemCut contains 456 human-designed secrets embedded in long multi-session conversations. Each sample includes annotated disclosures, indirect evidence trails, sufficient sets, and distractors to test if facts remain recoverable after deletion requests.
It uses native recovery (via system retrieval) and residual recovery (given surviving objects) metrics. Experiments show that deleting direct disclosures leaves significant residual recovery (39–76%), identifying candidate coverage as a bottleneck in existing memory designs.