LetTokenBurnRH-Detect: A Unified Benchmark for Reward Hacking Detection
A unified benchmark combining 92,761 rows from 11 datasets to evaluate language models' ability to detect reward hacking behaviors across six categories.
$ npm i -g lettokenburn-cli$ ltb bench info rh-detect
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3.5 Flash | 1.0 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Gemini 3.1 FL | 1.0 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Grok 4.1 Fast Reasoning | 1.0 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Llama 4 Maverick | 0.9 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| llama-4 Scout | 0.9 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| GPT-5 nano (2025-08-07) | 0.8 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
RH-Detect harmonizes subsets from eleven public datasets into a common schema with 92,761 rows and six behavior categories. It preserves source provenance and includes open-ended, multiple-choice, and tool-use tracks.
It evaluates detectors using AUROC and accuracy, finding that while top models reach 0.962 AUROC, performance drops significantly on multi-turn tool-use tasks. Single-token SFT improves cross-source robustness.