RoleCapBench
Role-capability leakage benchmark for role-prompted reasoning models
A curriculum-grounded evaluation framework measuring capability leakage in role-prompted models across six educational roles and four assessment levels using standardized assessments.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info rolecapbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3.5 4B | 0.9 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| google/gemma-4-E4B-it | 0.9 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| OLMo-3-7B | 0.9 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: RoleCapBench is constructed from standardized educational assessments to define explicit role-capability boundaries. It evaluates reasoning models on questions corresponding to elementary school through A-level curricula, ensuring the difficulty matches the assigned educational stage.
Scoring & Key Findings: The benchmark measures task accuracy, in-role accuracy (voice), and above-role accuracy (leakage). Results show that despite convincing role voice, open-weight models consistently exhibit capability leakage, achieving high accuracy on advanced questions even when prompted to act as lower-level students. An intervention called 'Injection' was proposed to mitigate this.