RealWorldShop
RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce
A session-level benchmark for conversational shopping agents evaluating state tracking, constraint updating, and grounded convergence over 3.28M products.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info realworldshop
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5 singh2025openai | 71.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi-K2.6 team2025kimi | 67.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek-V3.2 liu2025deepseek | 56.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.5-27B yang2025qwen3 | 55.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.5-35B-A3B yang2025qwen3 | 53.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Gemini-2.5-Flash comanici2025gemini | 38.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-4o hurst2024gpt | 38.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: The benchmark consists of a 3.28M-SKU product inventory, approximately 1,200 diverse structured shopping scenarios (covering ambiguous intents, bundles, multi-intents), and over 2,000 synthesized user profiles for action-controlled simulation.
Scoring & Findings: Evaluation uses role-play protocols to assess turn-level response quality, process quality, holistic outcomes, and early-exit behavior. Analysis reveals current systems struggle with state tracking under partial observability; the proposed RealShop_Agent improves success rates from 0.71 to 0.82.