OmniCapBench
Omni-Video Caption Benchmark
A deep-structured evaluation framework for fine-grained audio-visual captioning with 786 densely annotated videos, measuring entity references, visual shots, and audio events.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info omnicapbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3.5-Omni-Plus | 78.6 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5-Omni-Flash | 62.5 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3-Omni | 37.5 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
OmniCapBench comprises 786 videos decomposed into atomic units: 5,818 entities, 6,537 audio events, and 11,419 visual shots across Reference, Shot, and Event tracks. It evaluates fine-grained omnimodal comprehension including identity tracking and temporal grounding.
Scoring uses deterministic rules for structural integrity and localized LLMs for semantic checks. Experiments reveal MLLMs have strong local perception but weak long-horizon reasoning, with global scores masking local errors like identity drift.