LetTokenBurnSyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models
A diagnostic benchmark with six deterministic character-level tasks (counting, containment, palindrome, edit distance, substring extraction) using paired English and random strings to evaluate LLM reasoning.
$ npm i -g lettokenburn-cli$ ltb bench info syntaxbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemma4 31B | 97.0 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Qwen3.6 27B | 96.5 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| qwen3.5-27B | 94.0 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| qwen3.5-9B | 74.5 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| DeepSeek R1 32B | 53.0 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
| Qwen3.5 2B | 48.5 | N/A | Independent | medium | Oct 2, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
SyntaxBench comprises five core character-level tasks and one stress test, instantiated on paired English and length-matched random strings. It uses zero-, one-, and four-shot prompts across open-weight models.
Scoring includes exact-match and relaxed accuracy, Cohen’s kappa, and McNemar tests. Findings show tokenization strongly impacts accuracy, reasoning modes vary in utility, and substring extraction remains largely unsolved.