rsbench
Harbor: pendingThe advent of powerful neural classifiers has increased interest in problems that require both learning and reasoning.
- Samples
- N/A
- Released
- 2024
- Scores
- 0
auto-nesy-bench
Auto-Formalizing Neuro-Symbolic Predictors Benchmark
Evaluates LLMs' ability to translate natural language constraints into symbolic logic (CNF) for Neuro-Symbolic predictors, measuring formalization quality and downstream accuracy.
How to run
ltb · auto-nesy-bench
$ npm i -g lettokenburn-cli$ ltb bench info auto-nesy-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
5 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemma4 31B | 99.1 | zs-cot | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-oss 120B | 98.4 | zs-cot | Independent | medium | Oct 1, 2026 | arxiv.org |
| gpt-oss-20b | 70.3 | zs | Independent | medium | Oct 1, 2026 | arxiv.org |
| DeepSeek R1 Llama 70B | 67.6 | zs | Independent | medium | Oct 1, 2026 | arxiv.org |
| Gemma3 27B | 13.5 | zs-cot | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Extends rsbench with prompts, data instances, and annotations to test auto-formalization of logical constraints from text into formats like DIMACS, PySAT, or SymPy. It assesses both syntactic validity of generated formulas and their impact on NeSy predictor performance.
Scoring involves evaluating constraint correctness and downstream prediction accuracy using Semantic Probabilistic Layers or Semantic Loss. Findings show LLMs can generate meaningful formulas, with valid syntax leading to high-quality predictions despite some formalization errors.