SWE-bench_Lite
Harbor: pendingSWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically.
Verified71%Coding
- Samples
- 300
- Released
- 2023
- Scores
- 0
SWE-bench Complex
SWE-bench Complex A contamination-free, complexity-focused evaluation set for AI coding agents.
How to run
ltb · swe-bench-complex
$ npm i -g lettokenburn-cli$ ltb bench pull swe-bench-complex
External harness. Requires the SWE-bench style execution harness; run with the official harness or an agent scaffold (mini-swe-agent, openhands). Harbor adapter pending. https://github.com/SWE-bench/SWE-bench
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
0 sourced entriesNo sourced scores recorded yet.
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
SWE-bench Complex
A contamination-free, complexity-focused evaluation set for AI coding agents. SWE-bench Complex is a curated dataset of 115 real-world GitHub issues from major Python open-source projects, designed specifically for studying code complexity in AI-generated patches. All tasks were merged between January–March 2026, guaranteeing they postdate the training cutoff of current frontier models.
Task-Text Judge Comparison
GPT-5.4 reviewed solver-visible…