LetTokenBurnDo MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
A counterfactual benchmark auditing bias in image editing evaluation using 1,196 real samples and verified quality-preserving cues across four pipeline sites.
$ npm i -g lettokenburn-cli$ ltb bench info editjudgebias
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Content & Construction: The benchmark consists of 1,196 real editing triplets drawn from five public benchmarks and three independent content pools. It injects thirteen distinct visual or textual cues at four evaluation points: display protocol, judge prompt, image pixels, and edited content.
Scoring & Findings: Samples are scored via rubric ratings (1–10) in a scoring setting and pairwise preferences. The benchmark verifies that quality-preserving interventions move judges beyond their noise floors, revealing biases such as position effects, order reversals up to 60.9%, and reduced human agreement when cues involve the edit region.