GSM8K
Harbor adapterGSM8K: 小学应用题（多步推理）
- Samples
- 1,319
- Released
- 2021
- Scores
- 180
SCCGBench
Service-Call Code Generation Benchmark
A benchmark of 3,135 real-world RapidAPI call functions measuring executability and service-interface grounding for LLM-generated code.
How to run
ltb · sccgbench
$ npm i -g lettokenburn-cli$ ltb bench info sccgbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
6 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| DeepSeek V4 Pro | 3.8 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| DeepSeek-Coder-Instruct 33B | 2.9 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Qwen2.5-Coder-32B-Instruct | 2.9 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| CodeGemma-7B | 1.3 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Qwen2.5-Coder-7B | 0.7 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| DeepSeek-Coder-Instruct 1.3B | 0.0 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
SCCGBench contains 3,135 traceable samples from GitHub repositories, using redacted intents to reduce data contamination. It evaluates Web service-call code generation against official documentation.
Scoring uses Mock (virtual-execution hit) and Robust metrics with parameter-level diagnostics. Results show low intent-only success (~3.76%) but high oracle performance (~90%), highlighting API grounding as the main bottleneck.