ScreenSpot
ScreenSpot Evaluation Benchmark ScreenSpot is an evaluation benchmark for GUI grounding, comprising over 1,200 instructions from various environments, including iOS, Android, macOS, Windows, and Web.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info screenspot
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen2-VL-7B | 42.1 | N/A | Independent | medium | Sep 18, 2024 | arxiv.org |
| Qwen-GUI | 28.6 | N/A | Independent | medium | Jun 17, 2024 | arxiv.org |
| Qwen-VL-Max | 5.2 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
ScreenSpot Evaluation Benchmark
ScreenSpot is an evaluation benchmark for GUI grounding, comprising over 1,200 instructions from various environments, including iOS, Android, macOS, Windows, and Web. Each data point includes annotated element types (Text or Icon/Widget). For more details and examples, please refer to our paper.
Test Sample Details
Each test sample includes:
- img_filename: The interface screenshot file.
- instruction: Human-provided instruction.
- bbox: The bounding box of the target element corresponding to the instruction.
- data_type: The type of the target element, either
"icon"or"text". - data_source: The interface platform, which could be iOS, Android, macOS, Windows, or Web (e.g., GitLab, Shop, Forum, Tool).