LetTokenBurnOR-Bench Hard-1K (subset of OR-Bench: Over-Refusal Benchmark)
OR-Bench Hard-1K measures over-refusal in LLMs: the tendency to reject safe prompts that superficially resemble harmful ones.
$ npm i -g lettokenburn-cli$ ltb bench pull or-bench
External harness. Externally registered Inspect eval. Install from https://github.com/haeliotang/inspect-evals-orbench and run `inspect eval or_bench_hard_1k --model <provider/model>` (see the repo README). https://github.com/haeliotang/inspect-evals-orbench
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3-VL-8B-Instruct | 58.0 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Qwen2.5-VL-7B | 54.9 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| InternVL3.5-8B-Instruct | 50.9 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
OR-Bench Hard-1K measures over-refusal in LLMs: the tendency to reject safe prompts that superficially resemble harmful ones. The 1K subset contains prompts rejected by at least 3 frontier models, making it the most challenging split of the 80K dataset. Prompts were generated by rewriting toxic seeds into benign borderline variants using Mixtral, then filtered via an ensemble LLM moderator. Scoring uses an LLM judge classifying responses as direct_answer, direct_refusal, or indirect_refusal; the metric is rejection rate (lower is better).