LetTokenBurn
Sign in
Verified86%Harbor: pendingMultimodalKnowledge & Reasoning

AOKVQA

A crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of

PaperRepoDatasetLeaderboardHomepage

How to run

ltb · aokvqa
$ npm i -g lettokenburn-cli
$ ltb bench pull aokvqa

External harness. pip install crfm-helm && helm-run --run-entries a_okvqa:model=<model> --suite my-suite --max-eval-instances 100 (HELM scenario 'a_okvqa'; some scenarios need extra args) https://github.com/stanford-crfm/helm

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

1 sourced entry
HarnessSourceConfidenceLink
Pythia40.1N/AIndependentmediumJul 26, 2018arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

A crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer.

@misc{schwenk2022aokvqa, title={A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge}, author={Dustin Schwenk and Apoorv Khandelwal and Christopher Clark and Kenneth Marino and Roozbeh Mottaghi}, year={2022}, eprint={2206.01718}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Paper: https://arxiv.org/abs/2206.01718 Website: https://huggingface.co/datasets/HuggingFaceM4/A-OKVQA