SimIF-Bench
Harbor: pendingEvaluates full-duplex speech models' ability to follow multi-stage instructions in prescribed order using 200 scenarios and an LLM judge.
- Samples
- 200
- Released
- 2026
- Scores
- 8
DuplexAct-Bench
DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements
A bilingual benchmark evaluating full-duplex speech systems on six interaction behaviors (interruption, yielding, silence, etc.) under three contextual conditions using 1,290 streaming trials.
How to run
ltb · duplexact-bench
$ npm i -g lettokenburn-cli$ ltb bench info duplexact-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
12 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT Realtime 2.1 | 100.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Grok Voice | 100.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| MiniCPM-o 4.5 | 96.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.5-Omni | 96.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Freeze-Omni | 88.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| PersonaPlex | 88.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Gemini 2.5 Native Audio | 76.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Moshi | 75.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Raon-SpeechChat | 67.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| DuplexCascade | 27.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| VITA-1.5 | 27.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Nemotron 3 VoiceChat | 0.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Content & Construction: DuplexAct-Bench is a bilingual evaluation suite covering six complementary real-time interaction behaviors across Pre-session, In-session, and No-explicit contexts. It comprises 1,290 English and Chinese streaming trials involving user utterances drafted by GPT-5.6 and rendered via ViiTorVoice with realistic background noise.
Scoring & Findings: Systems are scored on Content (semantic fulfillment, constraint adherence) and Timing (latency metrics). Results reveal substantial variation across behaviors and conditions, highlighting frequent mismatches between semantic quality and behavioral timing in current systems.