LetTokenBurn
Sign in
Verified86%Harbor nativeMultimodalAgentic / Tool UseCoding

scienceagentbench

ScienceAgentBench full benchmark adapted to Harbor with Modal GPU support for GPU-backed tasks.

PaperRepoDataset

How to run

ltb · scienceagentbench
$ npm i -g lettokenburn-cli
$ ltb bench run scienceagentbench/scienceagentbench@1 --benchmark scienceagentbench -a claude-code -m anthropic/claude-opus-4-8
$ ltb bench push

Harbor-native: you can also call Harbor directly.

harbor
$ harbor run -d scienceagentbench/scienceagentbench@1 -a claude-code -m anthropic/claude-opus-4-8

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

0 sourced entries

No sourced scores recorded yet.

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

ScienceAgentBench (Harbor)

This dataset provides 102 ScienceAgentBench tasks in Harbor format for evaluating terminal agents on data-driven scientific discovery problems.

ScienceAgentBench tasks ask agents to write Python programs that solve scientific computing problems. Each task includes embedded scientific datasets, a custom evaluator, and an oracle solution from the original benchmark. Tasks span Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology/Cognitive Science.

  • Source: https://github.com/OSU-NLP-Group/ScienceAgentBench
  • Adapter PR: https://github.com/harbor-framework/harbor/pull/630
  • Tasks: 102 total, with 38 deterministic tasks and 64 visual/LLM-judge tasks
  • GPU-backed tasks: sab_12, sab_70, and sab_75

Layout

Each task directory (sab_1 through sab_102) contains:

sab_<N>/
├── task.toml
├── instruction.md
├── environment/
│   ├── Dockerfile
│   └── datasets/
├── solution/
│   └── solve.sh
└── tests/
    ├── test.sh
    ├── eval_runner.py
    ├── eval_program.py
    ├── test_outputs.py
    ├── gold_results/
    ├── benchmark_init/
    └── visual_judge.py

Evaluation

Category A tasks use deterministic checks against reference outputs. Category B tasks compare generated visual outputs against gold figures with a GPT-4o visual judge.

Category B tasks require OPENAI_API_KEY at verification time. VISUAL_JUDGE_MODEL defaults to gpt-4o-2024-05-13 in task metadata.

GPU Support

The generated dataset is Modal-compatible. The three GPU-backed tasks (sab_12, sab_70, sab_75) set gpus = 1 in task.toml and omit environment/docker-compose.yaml, matching Harbor's direct Modal GPU path.

For local Docker GPU execution, regenerate the dataset from the adapter without --modal; that variant emits environment/docker-compose.yaml for GPU passthrough.

Running

# Run oracle on all tasks after registry publication
harbor run -d scienceagentbench -a oracle -e modal

# Run with a specific agent
harbor run -d scienceagentbench -a claude-code -m anthropic/claude-haiku-4-5-20251001 -e modal

# Run from a local checkout
harbor run -p datasets/scienceagentbench -a oracle -e modal

For Category B tasks, ensure OPENAI_API_KEY is available to the verifier environment.

Oracle Notes

CPU-only oracle validation previously reached 95/102. The remaining failures were due to hardware constraints or stochastic thresholds: GPU-backed tasks sab_12, sab_70, and sab_75; high-memory tasks sab_11 and sab_97; and stochastic tasks sab_72 and sab_102.

Modal oracle trials confirmed GPU visibility for sab_12 and sab_70; sab_75 is marked the same way because it uses GPU-backed LDA encoding in the original benchmark.

Data

The embedded scientific task data totals roughly 5.5GB. The dataset-local .gitattributes tracks task input data and gold outputs with Git LFS without adding ScienceAgentBench-specific rules to the repository root.