Model
Gemini 1.5 Pro 001 Feb24
Knowledge & Reasoning
Knowledge & Reasoning| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| BBH | 84.0 | average | N/A | Self-reported | medium | storage.googleapis.com |
| MMLU | 82.7 | exact_match | N/A | Independent | medium | crfm.stanford.edu |
| MMLU | 81.9 | exact_match | N/A | Self-reported | medium | storage.googleapis.com |
Multimodal
Multimodal| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| Video-MME | 75.0 | overall_(no_subtitles) | N/A | Independent | medium | video-mme.github.io |
Safety & Alignment
Safety & Alignment| Benchmark | Score | Metric | Harness | Source | Confidence | Link |
|---|---|---|---|---|---|---|
| Cybench | 7.5 | unguided_%_solved | N/A | Independent | medium | cybench.github.io |
Reproduced by the community
Loading community runs…