GSM8K
Harbor adapterGSM8K: 小学应用题（多步推理）
- Samples
- 1,319
- Released
- 2021
- Scores
- 180
SAFIM
Syntax-Aware Fill-In-the-Middle
Syntax-Aware Fill-in-the-Middle (SAFIM) is a benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task.
How to run
ltb · safim
$ npm i -g lettokenburn-cli$ ltb bench info safim
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
14 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| DeepSeek-Coder-Instruct 33B | 69.0 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| DeepSeek-Coder-Instruct 6.7B | 63.4 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| GPT-4 1106 Preview | 53.3 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| CodeLlama-13b-hf | 52.8 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| DeepSeek-Coder-Instruct 1.3B | 52.6 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| CodeLlama-34b-hf | 49.7 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| CodeLlama-7b-hf | 45.0 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| GPT-3.5 Turbo 0301 | 40.9 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| incoder-6B | 33.8 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| codegen-16B-multi | 31.0 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| codegen-2B-multi | 29.6 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| incoder-1B | 29.3 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| codegen-6B-multi | 28.7 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
| codegen-350M-multi | 22.9 | N/A | Independent | medium | Mar 7, 2024 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Syntax-Aware Fill-in-the-Middle (SAFIM) is a benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. SAFIM has three subtasks: Algorithmic Block Completion, Control-Flow Expression Completion, and API Function Call Completion. SAFIM is sourced from code submitted from April 2022 to January 2023 to minimize the impact of data contamination on evaluation results.
- Authors: Linyuan Gong, Sida Wang, Mostafa Elhoushi, Alvin Cheung
- Paper: https://arxiv.org/abs/2403.04814
- Huggingface Dataset: https://huggingface.co/datasets/gonglinyuan/safim
- Leaderboard: https://safimbenchmark.com
- Code & Submission Instructions: [https://github.com/gonglinyuan/safim](https://github.com/gonglinyuan/safim
)
The SAFIM benchmark is partially derived from problem descriptions and code solutions from https://codeforces.com. According to the license of CodeForces, you may publish the texts of Codeforces problems in any open sources, but you must preserve a direct link to the site.