CritPt
Harbor: pendingCritPt — AI evaluation benchmark listed by opencompass.
Verified71%Knowledge & Reasoning
- Samples
- 70
- Released
- 2025
- Scores
- 138
Closed-loop evaluation benchmark for embedded coding agents
Closed-Loop Evaluation of LLM Agents for Embedded Software Development
A benchmark with 5 embedded-control tasks and 4 feedback scenarios evaluating LLM agents' ability to iteratively develop and verify ESP32 firmware in a simulated closed-loop environment.
How to run
ltb · closed-loop-evaluation-benchmark-for-embedded-coding-agents
$ npm i -g lettokenburn-cli$ ltb bench info closed-loop-evaluation-benchmark-for-embedded-coding-agents
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5.4 | 93.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| qwen3.5-27B | 60.0 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| GPT-5.4-mini | 53.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5 35B A3B | 26.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5 4B | 13.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| qwen3.5-9B | 13.3 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Qwen3.5 2B | 0.0 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The benchmark consists of five deterministic plant simulation tasks for ESP32 firmware, requiring agents to iterate through build, test, and repair cycles based on plain-text specifications. It evaluates performance across four feedback regimes: one-shot, self-verification, CI-style, and oracle-style.
Scoring is based on pass rate across 420 runs. GPT-5.4 achieved the highest pass rate but did not saturate the benchmark, while Qwen3.5-27B was the strongest local model, indicating the suite remains unsaturated and challenging for current agents.