AgentSWE
Harbor: pendingA benchmark of 25 tasks measuring coding agents' ability to build, edit, and optimize other agents against natural-language requirements using held-out cases.
- Samples
- 25
- Released
- 2026
- Scores
- 5
How to run
ltb · osworld
$ npm i -g lettokenburn-cli$ ltb bench pull osworld
External harness. pip install inspect-evals && inspect eval inspect_evals/osworld --model <provider/model> (other tasks: osworld_small) https://github.com/UKGovernmentBEIS/inspect_evals
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
25 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5.4 | 75.0 | N/A | Self-reported | low | Jun 1, 2026 | os-world.github.io |
| GPT-5.4 Pro | 75.0 | N/A | Self-reported | low | Jun 1, 2026 | os-world.github.io |
| Claude Opus 4.6 | 72.7 | N/A | Self-reported | low | Jun 1, 2026 | os-world.github.io |
| Claude Sonnet 4.6 | 72.5 | N/A | Self-reported | low | Jun 1, 2026 | os-world.github.io |
| Claude Sonnet 4.6 | 72.1 | claude-sonnet-4-6 (100 steps) | Independent | medium | N/A | os-world.github.io |
| Claude Opus 4.5 (2025-11-01) | 66.3 | Claude Opus 4.5 | Self-reported | medium | Nov 25, 2025 | anthropic.com |
| Kimi K2.5 | 63.3 | Kimi-K2.5 | Independent | medium | Feb 3, 2026 | os-world.github.io |
| Claude Sonnet 4.5 (2025-09-29) | 62.9 | claude-sonnet-4-5-20250929 (100 steps) | Independent | medium | Oct 30, 2025 | os-world.github.io |
| Qwen 3.5 | 62.2 | N/A | Self-reported | low | Jun 1, 2026 | os-world.github.io |
| Claude Sonnet 4.5 (2025-09-29) | 58.1 | claude-sonnet-4-5-20250929 (50 steps) | Independent | medium | Oct 30, 2025 | os-world.github.io |
| Claude Sonnet 4 (2025-05-14) | 43.9 | claude-4-sonnet-20250514 (50 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Claude Sonnet 4.5 (2025-09-29) | 42.9 | claude-sonnet-4-5-20250929 (15 steps) | Independent | medium | Oct 30, 2025 | os-world.github.io |
| Claude Sonnet 4 (2025-05-14) | 41.4 | claude-4-sonnet-20250514 (100 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Claude 3.7 Sonnet Thinking (2025-02-19) | 35.8 | claude-3-7-sonnet-20250219 (50 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Claude 3.7 Sonnet Thinking (2025-02-19) | 35.6 | claude-3-7-sonnet-20250219 (100 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Computer Use Preview (2025-03-11) | 31.3 | computer-use-preview (50 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Claude Sonnet 4 (2025-05-14) | 31.2 | claude-4-sonnet-20250514 (15 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Computer Use Preview (2025-03-11) | 30.5 | computer-use-preview (100 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Claude 3.7 Sonnet Thinking (2025-02-19) | 27.1 | claude-3-7-sonnet-20250219 (15 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Computer Use Preview (2025-03-11) | 26.0 | computer-use-preview (15 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| o3 | 23.0 | o3 (100 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| o3 | 17.2 | o3 (50 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| o3 | 9.1 | o3 (15 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Qwen2.5-VL-72B-Instruct | 5.0 | qwen2.5-vl-72b-instruct (100 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
| Qwen2.5-VL-72B-Instruct | 4.4 | qwen2.5-vl-72b-instruct (15 steps) | Independent | medium | Jul 27, 2025 | os-world.github.io |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
OSWorld (OS World) — 369个GUI/计算机操作任务. Scoring: 成功率. Difficulty: 极高. Imported from benchmark_research.md (2026-06).