ACEBench
Harbor: pendingACEBench is a comprehensive benchmark for evaluating LLM tool-usage ability.
- Samples
- 1,023
- Released
- 2025
- Scores
- 0
ToolRACERBench
ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation
A synthetic benchmark of 5.6K multi-turn conversation trajectories across six domains and 55 personas, designed to evaluate agentic robustness in function-calling scenarios including adversarial interactions.
How to run
ltb · toolracerbench
$ npm i -g lettokenburn-cli$ ltb bench info toolracerbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| ToolACE-MT | 84.9 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3 4B | 75.9 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| xLAM-2-3B-FC-R | 75.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen2.5 32B Instruct | 19.2 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3 4B Instruct 2507 | 8.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| ToolACE-8B | 6.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| xLAM2-3b-fc | 0.0 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
ToolRACERBench comprises 5.6K validated multi-turn conversations generated via a multi-agent simulation pipeline covering Happy, Unhappy, and Impossible paths. It spans six domains with diverse user personas to test agent resilience against non-cooperative behavior and tool failures.
The benchmark evaluates agentic accuracy and robustness using judge LLMs for validation. Models trained on this data show improved performance on external benchmarks like tau2-bench and ACEBench, demonstrating gains in handling realistic, failure-prone scenarios.