LetTokenBurn
Sign in
Auto71%Harbor: pendingKnowledge & Reasoning

ScholarCatalyst

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

A retrieval benchmark with 207 CS projects where systems find inspiring prior papers from a 191K corpus given early-stage research questions.

Paper

How to run

ltb · scholarcatalyst
$ npm i -g lettokenburn-cli
$ ltb bench info scholarcatalyst

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

7 sourced entries
HarnessSourceConfidenceLink
Claude Fable 5.10.5tool-calling agentIndependentmediumOct 1, 2026arxiv.org
GPT-5.6 Sol0.5tool-calling agentIndependentmediumOct 1, 2026arxiv.org
Qwen3-Emb-4B0.4N/AIndependentmediumOct 1, 2026arxiv.org
Qwen3-Emb-8B0.4N/AIndependentmediumOct 1, 2026arxiv.org
Gemini 3.7 Flash0.4tool-calling agentIndependentmediumOct 1, 2026arxiv.org
GPT-4.10.4tool-calling agentIndependentmediumOct 1, 2026arxiv.org
Gemini 3.1 Pro0.4tool-calling agentIndependentmediumOct 1, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

ScholarCatalyst contains 894 research questions from 207 recent CS projects, annotated by lead authors to identify prior papers that inspired their work. It uses a 191K-paper corpus restricted to pre-project publications.

Systems are evaluated on retrieving these 'catalyst' papers using Recall@20. The best performing system achieved 0.51 R@20, indicating the task remains unsaturated and challenging for current models.