LetTokenBurn
Sign in
Auto86%Harbor: pendingInstruction FollowingCodingMath

Intent-Eval

Intent-Eval: Demystifying Intent in Multi-Turn Dialogue

A benchmark of 414 source tasks (3,312 instances) across Actions, Code, Database, and Math to evaluate model robustness to rejected or superseded user intents in multi-turn dialogue.

PaperRepoHomepage

How to run

ltb · intent-eval
$ npm i -g lettokenburn-cli
$ ltb bench info intent-eval

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

8 sourced entries
HarnessSourceConfidenceLink
Claude 3.5 Sonnet91.7N/AIndependentmediumOct 5, 2026arxiv.org
Gemini 1.5 Pro91.7N/AIndependentmediumOct 5, 2026arxiv.org
google/gemma-4-26B-A4B-it91.7N/AIndependentmediumOct 5, 2026arxiv.org
Llama-3.1-8B-Instruct91.7N/AIndependentmediumOct 5, 2026arxiv.org
Mistral Large 291.7N/AIndependentmediumOct 5, 2026arxiv.org
Qwen2.5-7B-Instruct91.7N/AIndependentmediumOct 5, 2026arxiv.org
Qwen3.6 27B91.7N/AIndependentmediumOct 5, 2026arxiv.org
Qwen3 8B91.7N/AIndependentmediumOct 5, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Adapted from LiC, BFCL, HumanEval, LiveCodeBench, Spider, and GSM8K, it creates four conditions: Original, Neutral, Retained (rejected change), and Revised (accepted change).

Scoring uses accuracy against reference answers. Models show significant degradation when handling rejected proposals, suffering from 'mentioned-as-in-effect' confusion where inactive content influences outputs.