AgentPersonaBench
Harbor: pendingA benchmark of 2,460 tasks across 867 traits evaluating latent persona adherence via observable actions in survey, chat, web, and app environments.
- Samples
- 2,460
- Released
- 2026
- Scores
- 7
PremiseSQL
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
A benchmark with 1,800 flawed NL2SQL queries and 1,800 originals to evaluate premise critique and abstention capabilities in Text-to-SQL systems.
How to run
ltb · premisesql
$ npm i -g lettokenburn-cli$ ltb bench info premisesql
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
13 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Llama 4 Maverick | 0.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| ChatGLM-5.1 | 0.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Claude Opus 4.6 | 0.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| DeepSeek V4.1 Flash | 0.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-5.4 | 0.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-5.6 Luna | 0.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Doubao Seed 2.0 Pro | 0.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Gemini 3.1 Pro | 0.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-6.1 Sol | 0.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-6 Luna | 0.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Claude Opus 5.5 | 0.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| DeepSeek V3.2 | 0.1 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
PremiseSQL contains 3,600 queries (1,800 flawed, 1,800 original) derived from Spider and BIRD datasets, categorized into four premise failure types. It evaluates whether models can identify flawed premises and abstain from generating misleading SQL while correctly handling valid queries.
The evaluation framework measures intrinsic and prompt-induced behavior across Base, Implicit, and Explicit settings. Results indicate current LLMs are premise-trusting by default, with critique-oriented prompting improving detection but increasing false rejections of valid queries.