PickleBench
A benchmark of 1051 real-world Pickle-based pre-trained models (959 benign, 92 malicious) for evaluating security scanners against deserialization attacks.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info picklebench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entriesEvery number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
PickleBench comprises 959 benign and 92 malicious real-world Pickle-based pre-trained models, including extension registry attacks. It is designed to evaluate the coverage and precision of model security scanners.
The benchmark measures scanning coverage, false-negative rate, and false-positive rate. DITTO achieved an F1 score of 0.966 on this dataset, outperforming state-of-the-art scanners by minimizing false alerts while maintaining 100% detection coverage.