LetTokenBurn
Sign in
Incomplete29%Harbor: pendingSafety & AlignmentCoding

PatchEval

PatchEval (verified dockerized subset)

Evaluates LLMs on automated vulnerability repair (AVR) using 230 real-world CVEs (2015-2025) across Python, JavaScript, and Go, each packaged in a Docker sandbox.

Paper

How to run

ltb · patcheval
$ npm i -g lettokenburn-cli
$ ltb bench pull patcheval

External harness. Externally registered Inspect eval. Install from https://github.com/bytedance/PatchEval and run `inspect eval patcheval_verified --model <provider/model>` (see the repo README). https://github.com/bytedance/PatchEval

License is link-only: ltb bench pull fetches rows from the original host for local evaluation only and does not write a solution/ directory. Do not redistribute the generated tasks.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

0 sourced entries

No sourced scores recorded yet.

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Evaluates LLMs on automated vulnerability repair (AVR) using 230 real-world CVEs (2015-2025) across Python, JavaScript, and Go, each packaged in a Docker sandbox. An agent inspects a vulnerable repository and submits a unified diff patch. Scoring runs fix-run.sh inside a clean evaluator sandbox, executing both security (PoC) tests and functionality (unit) tests. A patch is successful only if it passes both test types.