LetTokenBurnPersonal Fact Memory benchmark
A controlled revision benchmark measuring pre-generation memory failures (staleness, misattribution, lateness) in language agents using prompt-level metrics.
$ npm i -g lettokenburn-cli$ ltb bench info pfm
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| qwen2:7b | 0.2 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Qwen2.5-7B-Instruct | 0.2 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The benchmark evaluates agent memory states before generation, focusing on temporal validity and identity resolution. It uses controlled revisions and public corpora like LongMemEval to test key assignment and retrieval.
Scoring relies on prompt-level metrics such as current-value recall, stale exposure, and clean-retrieval rates. Findings show LLM key assigners suffer from false merges, and dense retrieval offers no advantage over lexical methods when active stores are fixed.