FastBench
FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?
A benchmark of 306 QA pairs evaluating streaming VLMs on high-dynamic real-world video streams across eight domains and six capability dimensions.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info fastbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
8 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini-3.5-flash | 50.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| GPT-5.6 Sol | 46.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| JoyAI-VL-Interaction | 44.1 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3-VL | 32.9 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Aura | 32.1 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Doubao-Seed-2.0-Lite | 29.5 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| MOSS-VL-Realtime | 27.9 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| VideoChat3 | 20.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
FastBench contains 306 trajectory-grounded QA pairs spanning sports, gaming, wildlife, and transportation. It uses a pipeline with VLM generation, low-FPS filtering, and vision expert verification to ensure high-dynamic temporal dependency.
Scoring relies on accuracy of answers requiring fine-grained temporal perception. Top models like Gemini-3.5-Flash achieve only 50.7%, indicating the task remains unsolved and sensitive to frame sampling rates.