LetTokenBurnVERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence
A paired benchmark of 900 instances (450 grounded-ungrounded pairs) measuring schema-level faithfulness of LLM narrations over structured evidence via four verification gates.
$ npm i -g lettokenburn-cli$ ltb bench info veritygate
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 52.1 | veritygate | Independent | medium | Sep 30, 2026 | arxiv.org |
| Llama-3.3-70B-Instruct | 40.1 | veritygate | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-4o-mini | 19.7 | veritygate | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
VERITYGATE evaluates LLM faithfulness using a four-gate taxonomy: evidence existence, entity allowlisting, number binding, and claim-type rules. It consists of 900 paired instances across credit card optimization scenarios.
Scoring uses claim-survival rate, where frontier models initially fail significantly (e.g., 47.9% for Sonnet). Repair passes improve survival slightly, highlighting persistent structural faithfulness gaps.