LetTokenBurn
Sign in
Auto71%Harbor: pendingAgentic / Tool UseInstruction Following

UserProxyBench

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

An evaluation layer over the tau-bench family that measures user proxy functional fidelity via task-grounded rubrics across Telecom, Airline, Retail, and Banking domains.

Paper

How to run

ltb · userproxybench
$ npm i -g lettokenburn-cli
$ ltb bench info userproxybench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

7 sourced entries
HarnessSourceConfidenceLink
Claude Sonnet 4.60.9N/AIndependentmediumSep 29, 2026arxiv.org
GPT-4.1-mini0.8N/AIndependentmediumSep 29, 2026arxiv.org
Gemini-3.5-flash0.8N/AIndependentmediumSep 29, 2026arxiv.org
Gemma-31B0.8N/AIndependentmediumSep 29, 2026arxiv.org
Gemma-26B0.8N/AIndependentmediumSep 29, 2026arxiv.org
Qwen3.6-35B0.7N/AIndependentmediumSep 29, 2026arxiv.org
GPT-5.4-mini0.7N/AIndependentmediumSep 29, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Content & Construction: The benchmark evaluates seven LLM-based user proxies on enterprise tasks where a frozen agent (GPT-5.5) interacts with the simulated user. It assesses adherence to private blueprints regarding information disclosure timing, goal deviation, and grounding.

Scoring & Findings: Scoring uses an independent verifier model to grade trajectories against atomic criteria derived from task guidelines. Key findings reveal that 24.4% of successful agent episodes contain user-contract violations (primarily premature disclosure), which significantly alters interaction dynamics without affecting the primary reward metric.