AlgoPuzzleVQA
Harbor: pendingWe introduce the novel task of multimodal puzzle solving, framed within the context of visual question-answering.
- Samples
- 1,800
- Released
- 2024
- Scores
- 1
IllusionVQA
IllusionVQA is a Visual Question Answering (VQA) dataset with two sub-tasks.
How to run
ltb · illusionvqa
$ npm i -g lettokenburn-cli$ ltb bench info illusionvqa
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
6 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT4-Vision 4-shot | 63.0 | N/A | Independent | medium | Mar 23, 2024 | arxiv.org |
| GPT4-Vision | 58.9 | N/A | Independent | medium | Mar 23, 2024 | arxiv.org |
| Gemini-Pro 4-shot | 52.9 | N/A | Independent | medium | Mar 23, 2024 | arxiv.org |
| Gemini Pro | 51.3 | N/A | Independent | medium | Mar 23, 2024 | arxiv.org |
| LLaVA-1.5-13B | 40.0 | N/A | Independent | medium | Mar 23, 2024 | arxiv.org |
| CogVLM-Chat | 38.2 | N/A | Independent | medium | Mar 23, 2024 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
IllusionVQA is a Visual Question Answering (VQA) dataset with two sub-tasks. The first task tests comprehension on 435 instances in 12 optical illusion categories. Each instance consists of an image with an optical illusion, a question, and 3 to 6 options, one of which is the correct answer. We refer to this task as Logo IllusionVQA-Comprehension. The second task tests how well VLMs can differentiate geometrically impossible objects from ordinary objects when two objects are presented side by side. The task consists of 1000 instances following a similar format to the first task. We refer to this task as Logo IllusionVQA-Soft-Localization.