LetTokenBurnAn expert-annotated benchmark of 150 clinical trial protocol PDFs for evaluating coding agents on long-document digitization tasks across four categories.
$ npm i -g lettokenburn-cli$ ltb bench info trialpd-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 85.0 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| GPT-6 Astra | 82.5 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| GPT-5.6 Sol | 80.0 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Kimi K3 | 80.0 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Claude Sonnet 5 | 75.0 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| DeepSeek V4 Flash | 75.0 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Claude Opus 4.8 | 72.5 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Qwen3.8 27B | 68.3 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Claude Opus 4.6 | 67.5 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| GPT-4o | 41.7 | agent | Independent | medium | Oct 13, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
TrialPD-Bench comprises 150 publicly available clinical trial protocol PDFs (median 122 pages) annotated by experts. It covers four task categories: disease indication, eligibility criteria, objectives/endpoints, and schedule of activities, requiring structured JSON output based on USDM schemas.
Evaluation measures final-answer performance (F1, recall), evidence recall from agent traces, agent behavior, and cost. Results show even strong agents struggle with the 'last mile' of structuring extracted evidence, achieving at most 67.4% endpoint F1.