DialFork
DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents
A benchmark of 36,278 dialogues across 24 tasks and 12 domains measuring procedural hallucination in conversational agents via controlled forking.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info dialfork
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemma4 31B | 82.8 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen3.6-35B | 78.9 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| gpt-oss-20b | 78.0 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
DialFork evaluates procedural hallucination by forking conformant dialogues at controlled turns and injecting four types of deviations. It spans 24 tasks across 12 domains with 36,278 samples validated by human annotators.
The benchmark defines three evaluation tasks using FlowKL and FlowScore metrics to measure deviation from intended conversational processes. FlowScore significantly outperforms semantic similarity baselines across seven encoders.