Bemula
Bemula: A Benchmark for Memory Unlearning in LLM Agents
Bemula evaluates memory unlearning in LLM agents using 50 multi-session dialogues, 135 forget requests, and 1,742 test questions to measure selective forgetting.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info bemula
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
2 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | 28.5 | full context | Independent | medium | Oct 13, 2026 | arxiv.org |
| GPT-5.6 Luna | 23.1 | full context | Independent | medium | Oct 13, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Bemula comprises 50 multi-session dialogue instances with paired control versions, synthesized via a five-stage multi-agent pipeline. It includes 135 forget requests of six types and 1,742 test questions.
Scoring uses five probe suites (storage, removal, preservation, relearning, attack resistance) graded by an LLM judge against controls. Results show no system passes more than 1.5% of requests end-to-end.