CritPt
Harbor: pendingCritPt — AI evaluation benchmark listed by opencompass.
Verified71%Knowledge & Reasoning
- Samples
- 70
- Released
- 2025
- Scores
- 138
KaliBench
KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards
Evaluates LLMs on translating natural language queries into executable Kali Linux CLI commands across 8,504 query-command pairs.
How to run
ltb · kalibench
$ npm i -g lettokenburn-cli$ ltb bench info kalibench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
9 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GLM-5.2 | 81.3 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| DeepSeek V3.2 | 80.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| RedSage-K | 79.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen2.5-Ins | 77.0 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen3 Max (2025-09-23) | 76.3 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen3 Coder Next | 76.3 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-OSS | 75.7 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Llama-3.3-Ins | 74.7 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Mistral-Small-3.2 | 74.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
KaliBench comprises 8,504 verified query-command pairs covering 1,642 tools across 23 capability dimensions and 5 security phases. It is constructed via a pipeline involving LLM-based generation, sandboxed execution, and human refinement.
Scoring uses runtime-free verifiable rewards based on deterministic CLI structure, evaluating tool selection and argument construction. Findings identify argument construction as the primary bottleneck, with SFT and RLVR improving performance significantly.