Alice
Alice: A Large-Scale German Benchmark for Rubric-Based Multidimensional Automatic Short Answer Scoring
A large-scale German benchmark for rubric-based automatic short answer scoring, measuring learning performance, knowledge elements, and skills via question-specific rubrics.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info alice
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
6 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Llama-3.2 3B | 78.1 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Llama-3.2 1B | 77.2 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| RoBERTa | 75.5 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| XLM-RoBERTa-Long | 73.9 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| BERT-base-cased | 73.8 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| ModernBERT-base | 67.6 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Alice contains student responses annotated for three subtasks: learning performance (LP), knowledge elements (KE), and skills (SK). Each instance includes a question prompt, sample solution, and per-level rubrics.
Scoring is framed as rubric-retrieval, aligning answers to specific rubric items. Experiments show rubric-retrieval outperforms sequence classification on KE and SK subtasks, while zero-shot LLMs trail supervised approaches.