AndroidWorld
Harbor: pendingAn environment for benchmarking Android UI agents which provides durable rewards for different task parameterizations across 20 apps.
- Samples
- N/A
- Released
- 2025
- Scores
- 0
GraceBench
GraceBench for Execution Stability Diagnosis
An online evaluation framework with 169 tasks across 10 mobile apps using UI Transition Graphs and video-grounded trajectories to diagnose GUI agent execution stability.
How to run
ltb · gracebench
$ npm i -g lettokenburn-cli$ ltb bench info gracebench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
11 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini-3.5-flash | 0.7 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| GUI-Owl-1.5-8B | 0.7 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Qwen3.8 Max | 0.6 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| UI-TARS-1.5-7B | 0.6 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Qwen3.8 Flash | 0.6 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Qwen3.8 27B | 0.6 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| GLM-5.3 Flash | 0.6 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| DeepSeek V4.1 Flash | 0.6 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Gemini 3.1 Flash Lite | 0.5 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| GPT-5.5 | 0.5 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| GPT-5.6 Sol | 0.4 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
GraceBench contains 169 tasks spanning 10 applications, constructing 10 UI Transition Graphs with 2,909 canonical states. It uses video-grounded trajectory evaluation to align dynamic UI observations with graph states for step-level diagnosis.
It measures deviation rate, loop burden rate, and recovery ability beyond final success. Findings show 51.24% of successful runs contain deviations or loops, revealing hidden instability in high-performing agents like GPT-5.6 Sol.