PrefEval
Harbor: pendingLarge Language Models (LLMs) are increasingly deployed as chatbots, yet their ability to personalize responses to user preferences remains limited.
- Samples
- N/A
- Released
- 2025
- Scores
- 0
CICM
Controlled In-Context Memory
A diagnostic benchmark for tracking and using updated information in LLM conversations by distinguishing old-value reuse from other errors.
How to run
ltb · cicm
$ npm i -g lettokenburn-cli$ ltb bench info cicm
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
2 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4o | 79.7 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen2.5-7B-Instruct | 12.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Content & Construction: CICM is a comprehensive benchmark designed to test whether Large Language Models (LLMs) use the current state when earlier versions remain in context. It features programmatic ground truth, explicit update rules, and synthetic overwrite tasks alongside complex operational logs like warehouse operations.
Scoring & Findings: The benchmark classifies answers as 'current', 'old' (stale binding), or other variables to diagnose selection failures. Key findings include the phenomenon of 'attention drift,' where models favor old values over current ones, and a training-free intervention method that redirects attention to correct these errors.