Balrog
Harbor: pendingTracked by Epoch AI's Benchmarking Hub (results collected from external sources).
Incomplete29%Knowledge & Reasoning
- Samples
- N/A
- Released
- 2024
- Scores
- 39
CreativePreferences
A dataset of 2.8M texts with 317M human preference judgments across 7 creative domains, featuring 42 benchmark tasks for preference prediction.
How to run
ltb · creativepreferences
$ npm i -g lettokenburn-cli$ ltb bench info creativepreferences
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| google/gemma-4-31B-it | 0.7 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Llama-3.1-70B-Instruct | 0.7 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Gemma 2 9B IT | 0.7 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Phi-4 | 0.7 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Qwen2.5 14B Instruct | 0.7 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Llama-3.1-8B-Instruct | 0.7 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Mistral-7B-Instruct-v0.3 | 0.7 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
CreativePreferences contains 2.8 million texts labeled by 317 million human preference judgments across seven domains including math, coding, and creative writing. It comprises 42 benchmark tasks focused on predicting community votes, gatekeeper verdicts, and expert curations.
Scoring involves measuring articulability and verifiability gaps using executable programs, rubrics, and dense models. Key findings reveal robust gaps where tacit knowledge resists articulation, particularly in peer review and creative writing, challenging current AI reward modeling approaches.