PICABench
Harbor: pendingImage editing has achieved remarkable progress recently.
- Samples
- 105,085
- Released
- 2026
- Scores
- 0
Counterfactual Physical Consistency Benchmark
Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency
A benchmark of 5,500 synthetic images testing physical consistency (shadows, reflections, support) via counterfactual rendering with region-grounded labels.
How to run
ltb · counterfactual-physical-consistency-benchmark
$ npm i -g lettokenburn-cli$ ltb bench info counterfactual-physical-consistency-benchmark
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
4 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 95.3 | lora | Independent | medium | Oct 6, 2026 | arxiv.org |
| LLaVA-1.5-7B | 94.8 | lora | Independent | medium | Oct 6, 2026 | arxiv.org |
| InternVL3.5-8B-Instruct | 93.3 | lora | Independent | medium | Oct 6, 2026 | arxiv.org |
| LLaVA-1.5-7B | 48.4 | zero-shot | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The benchmark uses Mitsuba 3 to generate 5,500 images from 500 scene families, pairing clean scenes with controlled physical violations like missing shadows or unsupported objects. It provides category labels, masks, and metadata without manual annotation.
Evaluation measures category macro-F1 on held-out scenes and under intervention shifts. Baselines like LLaVA-1.5 achieve ~64% F1 in-distribution but drop significantly under shift, revealing reliance on rendering shortcuts.