ToolHACE
ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses
A span-level benchmark for detecting post-tool-call hallucinations in tool-augmented LLMs, featuring five fine-grained categories including Answer Mismatch and Missing Tool.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info toolhace
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
6 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3.5 2B | 79.0 | N/A | Independent | medium | Oct 12, 2026 | arxiv.org |
| Qwen3.5-0.8B | 76.1 | N/A | Independent | medium | Oct 12, 2026 | arxiv.org |
| Kimi 2.6 | 67.8 | N/A | Independent | medium | Oct 12, 2026 | arxiv.org |
| GPT-4o | 65.8 | N/A | Independent | medium | Oct 12, 2026 | arxiv.org |
| ModernBERT-base | 65.5 | N/A | Independent | medium | Oct 12, 2026 | arxiv.org |
| ModernBERT-large | 64.7 | N/A | Independent | medium | Oct 12, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
ToolHACE is built on the ToolACE corpus, using synthetic edits and programmatic perturbations to create span-level annotations for hallucination detection. It covers five categories: Correct, Answer Mismatch, Overgeneration, Undergeneration, and Missing Tool.
The benchmark evaluates detectors via span-level classification performance. Results show that post-tool-call hallucinations are widespread and that detectors trained on tool-agnostic data transfer poorly, requiring specialized training.