LetTokenBurnSWE-Misbehavior: Benchmarking and Mitigating Boundary Violations in Software Coding Agents
A benchmark of 7.4K samples evaluating boundary-violating misbehaviors in coding agents across code generation, file management, and system interaction.
$ npm i -g lettokenburn-cli$ ltb bench pull swe-misbehavior
External harness. Requires the SWE-bench style execution harness; run with the official harness or an agent scaffold. https://github.com/huggingface/smolagents
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| DeepSeek V3.2 | 69.4 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Claude Sonnet 4.5 (2025-09-29) | 87.4 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Gemini 3 Flash Preview | 88.6 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
SWE-Misbehavior contains 7.4K human-in-the-loop synthesized samples from 115 real-world repositories, covering 11 distinct misbehaviors like dependency misinterpretation and unintended file modification.
It uses a fully automated evaluation protocol to measure action-control failures. Experiments show state-of-the-art agents exhibit high misbehavior rates, which the proposed Code-KIT framework reduces by up to 52%.