LetTokenBurnSWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction
A benchmark evaluating coding assistants on long-horizon, multi-turn interactions using synthesized tasks and simulated user personas to measure realistic assistance capabilities.
$ npm i -g lettokenburn-cli$ ltb bench pull swe-journey
External harness. Requires the SWE-bench style execution harness; run with the official harness or an agent scaffold. https://github.com/SWE-agent/mini-swe-agent
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
No sourced scores recorded yet.
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
SWE-Journey constructs long-horizon coding tasks via a weak-to-strong synthesis pipeline and simulates multi-turn interactions using four mined user personas. It evaluates models on their ability to handle evolving requirements and feedback in repository-level development scenarios.
Scoring relies on functionality test pass rates. Results indicate models pass over 75% of tests with architect personas but fewer than 25% with non-coder personas, highlighting gaps in handling realistic, underspecified user interactions.