Balrog
Harbor: pendingTracked by Epoch AI's Benchmarking Hub (results collected from external sources).
Incomplete29%Knowledge & Reasoning
- Samples
- N/A
- Released
- 2024
- Scores
- 39
TALK-Dem
Talking Attributes and Linguistic Knowledge in Dementia Benchmark
A benchmark evaluating LLM-driven robot task planning robustness against five dementia-associated communication patterns using 4,800 instructions derived from household tasks.
How to run
ltb · talk-dem
$ npm i -g lettokenburn-cli$ ltb bench info talk-dem
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
8 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 97.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Claude Opus 5 | 96.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Qwen2.5-72B-Instruct | 86.3 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Gemma 2 9B IT | 79.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Ministral-8B-Instruct-2410 | 73.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Llama-3.1-70B-Instruct | 69.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Llama-3.1-8B-Instruct | 66.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Qwen2.5-7B-Instruct | 28.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Content & Construction: TALK-Dem is the first benchmark for embodied task planning under verbal communication patterns associated with people living with dementia (PLWD). It constructs a dataset of 4,800 instructions by transforming 300 semantically diverse ALFRED household tasks into natural conversational requests and applying five specific dementia-associated patterns: Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion. These transformations are applied at three intensity levels to simulate real-world interaction challenges.
Scoring & Key Findings: The benchmark measures task success rates using the SayCan planner across various LLMs (both open-weight and closed-source). Experiments revealed that dementia-associated patterns cause substantial performance drops in open-weight models, with degradation up to 22.3 percentage points compared to ideal instructions. A proposed mitigation method called CARE improved average task success by 18.1 percentage points over vanilla prompting.