LetTokenBurnOverlap-Unique-Conflict Benchmark
A benchmark of ~22K narrative pairs and 140K instances evaluating LLMs' ability to extract overlapping, conflicting, and unique clauses across factual, argumentative, and political narratives.
$ npm i -g lettokenburn-cli$ ltb bench info ouc-benchmark
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen-3-0.6B | 74.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Phi-4-14B | 71.6 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3-32B | 65.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3 8B | 64.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Gemma-4-4B | 61.7 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| OLMo-3.1-32B | 53.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3 4B | 53.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Nemotron-3-30B | 41.2 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Llama-3.2 3B | 18.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Gemma-4-2B | 11.6 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Phi-4-4B | 11.1 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| OLMo-3-7B | 8.6 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The OUC-Benchmark consists of approximately 22,000 narrative pairs spanning three discourse types: factual, argumentative, and political. It contains roughly 140,000 instances designed to test the extraction of shared (overlap), contradictory (conflict), and exclusive (unique) information between two alternative narratives describing the same event.
The benchmark evaluates open-source LLMs using F1-score as the primary metric across zero-shot, few-shot, and fine-tuned settings. Key findings reveal that while unique information is easily extracted (>75% F1), overlap and conflict extraction remain challenging (<62% for top models). Task-specific supervision significantly improves performance in smaller models (e.g., Qwen-3-8B) on these harder tasks.