NAQD-Env
NAQD Env: A benchmark for selective withdrawal in language agents
A synthetic environment evaluating 350 frozen scenarios of agent reliability by measuring selective action suspension and resumption against a deterministic reference policy.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info naqd-env
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | 0.6 | naqd-env | Independent | medium | Sep 29, 2026 | arxiv.org |
| Qwen2.5-7B-Instruct | 0.5 | naqd-env | Independent | medium | Sep 29, 2026 | arxiv.org |
| Qwen2.5-3B-Instruct | 0.5 | naqd-env | Independent | medium | Sep 29, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: NAQD-Env is a typed-premise environment featuring eleven dependency families covering direct support, propagation, alternatives, delegation, and prohibited combinations. It uses explicit evidence, authorization, and constraint dependencies to simulate scenarios where premises change (e.g., revoked grants), requiring agents to suspend affected actions while preserving unaffected work.
Scoring & Key Findings: The benchmark employs a multidimensional scoring protocol distinguishing attempted violations from executed ones via a simulated gate. Metrics include withdrawal recall, resumption accuracy, and policy agreement. Baseline evaluations of models like Qwen2.5-7B showed low withdrawal recall (at most 0.06) and no valid resumptions observed at eligible opportunities.