VAmoS Energy
VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation - VAmoS Energy Benchmark
A benchmark evaluating voice agents on utility billing and payment assistance tasks involving multiple requests, stateful tools, regulatory policies, and background noise conditions.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info vamos-energy
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
14 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Grok Voice | 44.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Gemini 3.8 Live | 40.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| OpenAI GPT-Live 1 | 39.2 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| OpenAI Realtime 2.1 | 37.9 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| ElevenLabs | 36.4 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| OpenAI Realtime | 33.6 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| LiveKit | 29.4 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Gemini 3.1 Live | 25.6 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Mistral · Medium 3.5 + Voxtral | 22.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Gradium | 21.3 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Vapi | 20.5 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Deepgram | 19.8 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Hugging Face · gpt-oss-120b | 19.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Pipecat · GPT-4.1 mini | 17.3 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: The benchmark simulates residential utility calls where an agent handles two to four customer requests (e.g., bill questions, payments) using Stripe and Apache Fineract services. Tasks are constructed by appending valid requests while adhering to Pennsylvania's billing regulations.
Scoring & Findings: Success is measured via completion rates verified against explicit requirements; simulated callers can only hear spoken words but not inspect actions. Background noise (e.g., TV) significantly reduces success, with Grok Voice leading and others following.