Model
Gemini 1.5 Pro
Knowledge & Reasoning
Knowledge & Reasoning| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| MMLU | 81.0 | exact_match | N/A | Independent | medium | crfm.stanford.edu |
Math
Math| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| FrontierMath | 2.0 | accuracy | N/A | Independent | medium | arxiv.org |
Agentic / Tool Use
Agentic / Tool Use| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| EHR-RobustGym | 61.6 | success_rate | N/A | Independent | medium | arxiv.org |
Multimodal
Multimodal| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| MM-Vet v2 | 66.9 | gpt-4_score | N/A | Independent | medium | arxiv.org |
| SME | 41.9 | bleu-4 | N/A | Independent | medium | arxiv.org |
| TVBench | 47.6 | average_accuracy | N/A | Independent | medium | arxiv.org |
| Video-MME | 81.3 | accuracy_(%) | N/A | Independent | medium | arxiv.org |
| VideoDB's OCR Benchmark Public Collection | 76.1 | average_accuracy | N/A | Independent | medium | arxiv.org |
| Vinoground | 37.0 | text_score | N/A | Independent | medium | arxiv.org |
| Vinoground | 35.8 | text_score | N/A | Independent | medium | arxiv.org |
| Zero-shot Video Question Answering on LongVideoBench | 66.7 | accuracy_(%_) | N/A | Independent | medium | arxiv.org |
Instruction Following
Instruction Following| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| Intent-Eval | 91.7 | accuracy | N/A | Independent | medium | arxiv.org |
Safety & Alignment
Safety & AlignmentMultilingual
MultilingualReproduced by the community
Loading community runs…