VulnBump
VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green
Evaluates agentic tools on 307 dependency upgrade tasks across 43 real repositories, requiring vulnerability remediation, successful builds, and passing tests.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info vulnbump
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Evo Remediation Agent | 90.2 | claude sonnet 5.5 | Independent | medium | Oct 12, 2026 | arxiv.org |
| Claude Code + Snyk scan | 77.5 | claude sonnet 5.5 | Independent | medium | Oct 12, 2026 | arxiv.org |
| Claude Code unaided | 59.6 | claude sonnet 5.5 | Independent | medium | Oct 12, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
VulnBump comprises 307 upgrade tasks covering 2,066 vulnerabilities in 43 repositories across six package managers. Tasks are derived from real Snyk production data to ensure realistic distribution.
Scoring requires passing four gates in a fresh container: dependency resolution, vulnerability rescan, build success, and test preservation. The best reported pass rate is 90.2% by Snyk’s Evo Remediation Agent.