DoGBench
Documentation Generation Benchmark
A benchmark evaluating AI agents' ability to generate and maintain real user-facing software documentation based on code changes or reported gaps.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info dogbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3.8 Max +OpenCode | 47.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6 Sol +Codex | 46.2 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GLM 5.2 +OpenCode | 44.2 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi K2.7 Code +OpenCode | 43.9 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Opus 4.8 +Claude Code | 41.2 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Sonnet 4.6 +Claude Code | 40.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.5 +Codex | 34.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
The DoGBench dataset consists of 292 items from open-source projects like Helm, PostHog, Mautic, and Doc Detective. Each item provides a pre-change repository snapshot and a trigger (code PR or issue), requiring the agent to decide whether documentation needs updating and produce an acceptable patch if so.
Scoring uses task-specific rubrics validated by project maintainers that assess accuracy, completeness, reader guidance, placement, and conventions separately from human-merged patches. The composite score combines patch quality with correct abstention decisions.