LetTokenBurn
Sign in
Auto57%Harbor: pendingCodingMultimodal

SWE-bench_Multimodal

SWE-bench Multimodal Dataset Summary SWE-bench Multimodal is a dataset that tests systems' ability to resolve real-world GitHub issues in visual software domains.

PaperDataset

How to run

ltb · swe-bench-multimodal
$ npm i -g lettokenburn-cli
$ ltb bench pull swe-bench-multimodal

External harness. Requires the SWE-bench style execution harness; run with the official harness or an agent scaffold (mini-swe-agent, openhands). Harbor adapter pending. https://github.com/SWE-bench/SWE-bench

License is link-only: ltb bench pull fetches rows from the original host for local evaluation only and does not write a solution/ directory. Do not redistribute the generated tasks.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

0 sourced entries

No sourced scores recorded yet.

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

SWE-bench Multimodal

Dataset Summary SWE-bench Multimodal is a dataset that tests systems' ability to resolve real-world GitHub issues in visual software domains. Unlike the original SWE-bench, which is Python-only and text-only, every task instance here comes from a JavaScript or TypeScript repository and carries at least one image asset — a screenshot, a screen recording, a diagram, or a rendering of incorrect output. The dataset collects 612 Issue-Pull Request pairs from 17…