LetTokenBurnWording Invariance and Sensitivity Evaluation
A matched evaluation suite of 100 tasks measuring wording robustness in verifiable instruction following across exact word count, keyword inclusion, and length range constraints.
$ npm i -g lettokenburn-cli$ ltb bench info wise
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 4.8 | 80.3 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| GPT-5.5 | 62.7 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Claude Sonnet 4.6 | 60.0 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| GPT-4.1-mini | 55.3 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| DeepSeek V4 Pro | 55.0 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Claude Haiku 4.5 | 54.7 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| GPT-4o-mini | 53.7 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| DeepSeek Chat | 36.3 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Kimi K2.6 | 34.3 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Qwen3.6 35B A3B | 26.7 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Mistral Small | 24.7 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Gemini 2.5 Flash | 9.7 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Gemini 3.1 Flash Lite | 6.7 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
WISE comprises 100 matched tasks with varied linguistic realizations of fixed operational constraints (exact count, keyword, range). It holds task, target, and checker constant while varying prompt wording.
Scoring uses deterministic checkers for compliance. Key findings show substantial compliance shifts and ranking reversals based on wording alone, supplementing scores with worst-form performance and stability metrics.