LetTokenBurnSWE-CC: Benchmarking Repository Policy Compliance for Coding Agents
Evaluates coding agents' compliance with 823 repository-specific policies across 12 open-source projects using 500 tasks derived from SWE-bench Verified.
$ npm i -g lettokenburn-cli$ ltb bench pull swe-cc
External harness. Requires the SWE-bench style execution harness; run with the official harness or an agent scaffold. https://github.com/dangtruong01/swe-cc-arxiv
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Kimi K2.5 | 74.6 | mini-swe-agent | Independent | medium | Oct 5, 2026 | arxiv.org |
| GPT-5.6 Luna | 78.6 | mini-swe-agent | Independent | medium | Oct 5, 2026 | arxiv.org |
| Gemini 3.7 Flash | 80.2 | mini-swe-agent | Independent | medium | Oct 5, 2026 | arxiv.org |
| DeepSeek V4 Flash | 93.4 | mini-swe-agent | Independent | medium | Oct 5, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
SWE-CC converts developer documentation into 823 machine-checkable atomic policies across 12 repositories. It uses lightweight checker functions to audit both agent runtime behaviors and final deliverables.
Scoring measures policy violation rates. Evaluation of frontier models reveals agents violate 43.1% of applicable policies, with half occurring during intermediate execution steps, indicating functional correctness does not ensure compliance.