Flickr30 K
Flickr30k
An image caption corpus consisting of 158,915 crowd-sourced captions describing 31,783 Flickr
How to run
$ npm i -g lettokenburn-cli$ ltb bench pull flickr30-k
External harness. pip install crfm-helm && helm-run --run-entries flickr30k:model=<model> --suite my-suite --max-eval-instances 100 (HELM scenario 'flickr30k'; some scenarios need extra args) https://github.com/stanford-crfm/helm
License is link-only: ltb bench pull fetches rows from the original host for local evaluation only and does not write a solution/ directory. Do not redistribute the generated tasks.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
4 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| ERNIE-ViL 2.0 | 97.2 | N/A | Independent | medium | Sep 30, 2022 | arxiv.org |
| ERNIE-ViL 2.0 | 96.1 | N/A | Independent | medium | Sep 30, 2022 | arxiv.org |
| InternVL-G | 95.7 | N/A | Independent | medium | Dec 21, 2023 | arxiv.org |
| InternVL-C | 94.7 | N/A | Independent | medium | Dec 21, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
An image caption corpus consisting of 158,915 crowdsourced captions describing 31,783 Flickr images.
@article{young2014image, title={From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions}, author={Young, Peter and Lai, Alice and Hodosh, Micah and Hockenmaier, Julia}, journal={Transactions of the Association for Computational Linguistics}, volume={2}, pages={67--78}, year={2014}, publisher={MIT Press} }
Paper: https://shannon.cs.illinois.edu/DenotationGraph/TACLDenotationGraph.pdf Website: https://shannon.cs.illinois.edu/DenotationGraph/