AgentDoxx
Harbor: pendingAn evaluation suite of 822 synthetic interview transcripts grounded in public info to measure LLM re-identification risks via web search.
- Samples
- 822
- Released
- 2026
- Scores
- 9
DSV-Mem
DSV-Mem: Evaluating Multimodal Memory in Professional Workflows for MLLM Agents
A benchmark with 1,000 questions across five categories evaluating dense stateful visual memory in professional workflows using expert-reviewed scenarios.
How to run
ltb · dsv-mem
$ npm i -g lettokenburn-cli$ ltb bench info dsv-mem
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
13 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini 3.7 Flash | 39.6 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini 3.1 Pro | 27.5 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Kimi K3 | 24.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini 3.6 Flash | 23.0 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| MiniMax-M3 | 13.1 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3.5 397B A17B | 11.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| qwen3.5-27B | 9.6 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| google/gemma-4-31B-it | 8.5 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini 3.5 Flash Lite | 8.2 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| qwen3.5-9B | 7.2 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Kimi K2.5 | 5.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| GLM-4.6V | 3.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3.5 4B | 3.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
DSV-Mem comprises 1,000 questions across five user-oriented categories (Current State, Past State, Derived State, Change History, Conflict/Refusal) featuring dense, evolving professional artifacts. It uses a generation harness decoupling state-transition synthesis from conversation filling.
Scoring relies on accuracy of state tracking and reconciliation. Key findings show even strongest baselines score below 45%, with state evolution being the dominant difficulty factor rather than OCR or arithmetic.