LetTokenBurnSimulator Instruction-Following Benchmark
Evaluates full-duplex speech models' ability to follow multi-stage instructions in prescribed order using 200 scenarios and an LLM judge.
$ npm i -g lettokenburn-cli$ ltb bench info simif-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Moshi | 1.0 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| dGSLM | 0.9 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Raon-SpeechChat-9B | 0.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Freeze-Omni | 0.6 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| PersonaPlex | 0.6 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| KAME | 0.6 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| SteerablePlex | 0.4 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Gemini Live | 0.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
SimIF-Bench reuses 200 staged scenarios from Full-Duplex-Bench v2 to test examiner models. It measures topic adherence, naturalness, goal completion, and exact stage ordering via an LLM judge.
Scoring uses majority vote for categorical decisions and median for naturalness (1-5 scale). Results show current open-source models struggle with ordered constraint completion compared to GPT-Realtime.