Winoground
Winoground is a novel task and dataset for evaluating the ability of vision and language models
How to run
$ npm i -g lettokenburn-cli$ ltb bench pull winoground
External harness. pip install crfm-helm && helm-run --run-entries winoground:model=<model> --suite my-suite --max-eval-instances 100 (HELM scenario 'winoground'; some scenarios need extra args) https://github.com/stanford-crfm/helm
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
6 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4V | 54.5 | N/A | Independent | medium | Jan 5, 2024 | arxiv.org |
| LLaVA-1.5-CCoT | 42.0 | N/A | Independent | medium | Nov 27, 2023 | arxiv.org |
| LLaVA-1.5 | 36.0 | N/A | Independent | medium | Nov 27, 2023 | arxiv.org |
| Gemini | 30.8 | N/A | Independent | medium | Jan 5, 2024 | arxiv.org |
| LLaVA-1.5-ZS-CoT | 28.0 | N/A | Independent | medium | Nov 27, 2023 | arxiv.org |
| LLaVA | 24.8 | N/A | Independent | medium | May 10, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Winoground is a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning. Given two images and two captions, the goal is to match them correctly—but crucially, both captions contain a completely identical set of words/morphemes, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set of fine-grained tags to assist in analyzing model performance.
Users must agree to share their contact information before downloading the dataset from Hugging Face. Either agree to the terms and set HUGGING_FACE_ACCESS_TOKEN to an access token of a valid Hugging Face account or have the dataset pre-downloaded at the Hugging Face cache (default path: ~/.cache/huggingface/datasets).
Paper: https://arxiv.org/abs/2204.03162 Website: https://huggingface.co/datasets/facebook/winoground