Video-MME
Harbor adapterVideo-MME stands for Video Multi-Modal Evaluation.
- Samples
- 2,700
- Released
- 2024
- Scores
- 51
SME
Standard Multimodal Explanation
SME is a new dataset for Multi-modal Explanation for Visual Question Answering comprising 1,028,230 samples, with 1,656 visual objects requiring detection in explanations.
How to run
ltb · sme
$ npm i -g lettokenburn-cli$ ltb bench info sme
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
4 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4-1106-Vision-Preview | 45.5 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| Gemini 1.5 Pro | 41.9 | N/A | Independent | medium | Mar 8, 2024 | arxiv.org |
| Qwen-VL-Max | 24.3 | N/A | Independent | medium | Aug 24, 2023 | arxiv.org |
| GLM-4V | 14.4 | N/A | Independent | medium | Nov 6, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
SME is a new dataset for Multi-modal Explanation for Visual Question Answering comprising 1,028,230 samples, with 1,656 visual objects requiring detection in explanations. To our knowledge, this is the first dataset where the explanations are in standard English with additional visual grounding tokens.