LetTokenBurnAPTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry
A benchmark evaluating cross-telemetry robustness in autonomous Advanced Persistent Threat (APT) investigation using 370 cases derived from attack reconstructions and 16.4 million log records.
$ npm i -g lettokenburn-cli$ ltb bench info aptinvestbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-6 Astra | 79.2 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6 Sol | 71.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.5 | 60.7 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V4 Pro | 50.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| MiniMax-M3 | 46.2 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6 Luna | 43.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.6-terra | 43.6 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Content & Construction: The benchmark comprises 370 cases across seven SOC-inspired telemetry conditions, derived from 56 report-informed attack reconstructions with 16.4 million log records. It allows agents to investigate unverified leads under varying evidence availability while keeping reference actions and requirements fixed.
Scoring & Key Findings: Agents submit reports with record-level citations; scoring tracks sufficient evidence across logs, query returns, and formal citations. Results show that while aggregate coverage may appear stable (e.g., 1.6% decline), substantial instability exists where over 35% of previously covered actions lose citation support when telemetry changes from Full to endpoint-only.