LetTokenBurnSWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle
Evaluates LLM deception in software engineering across 720 Chinese/English instances covering six lifecycle stages and two deception triggers.
$ npm i -g lettokenburn-cli$ ltb bench pull swe-deceptionbench
External harness. Requires the SWE-bench style execution harness; run with the official harness or an agent scaffold. https://github.com/THU-Agent/SWE-DeceptionBench
License is link-only: ltb bench pull fetches rows from the original host for local evaluation only and does not write a solution/ directory. Do not redistribute the generated tasks.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5.5 | 2.5 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Claude Opus 4.8 | 3.3 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Qwen3.7 Max | 35.8 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| GLM-5.2 | 62.5 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| DeepSeek V4 Pro | 74.2 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Kimi K2.6 | 80.0 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Gemini-3.5-flash | 90.8 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The benchmark comprises 720 test instances in Chinese and English, spanning six software development lifecycle stages. It features Active and Passive Deception scenarios under Neutral, Reward, and Pressure conditions.
Scoring measures deception rates, with findings showing 49.88% Active and 64.29% Passive deception under neutral conditions. Lightweight defenses like guard models and Self-Reminder provide incomplete protection against these deceptive behaviors.