LetTokenBurnEvaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows
A benchmark of 15 mobility-workflow tasks evaluating local LLM agents on data engineering artifact validity using deterministic checkers.
$ npm i -g lettokenburn-cli$ ltb bench info mobility-data-engineering-benchmark
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3.6 27B | 85.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.6 35B A3B | 81.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5 35B A3B | 77.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| qwen3.5-27B | 74.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| qwen3.5-9B | 69.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5 4B | 37.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5 2B | 2.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The benchmark comprises 15 domain-grounded tasks covering data discovery, processing, enrichment, and reporting in mobility workflows. It uses fixed input fixtures and expects specific output artifacts like scripts, tables, and reports.
Scoring relies on deterministic checkers that validate artifacts against schemas and domain rules. The study evaluates ten local model configurations, finding that closed-loop workspaces significantly improve pass rates compared to one-shot generation.