Turbulence
$\textbf{Turbulence}$ is a new benchmark for systematically evaluating the correctness and robustness of instruction-tuned large language models (LLMs) for code generation.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info turbulence
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
4 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4 | 84.8 | N/A | Independent | medium | Dec 22, 2023 | arxiv.org |
| GPT-3.5 Turbo | 61.7 | N/A | Independent | medium | Dec 22, 2023 | arxiv.org |
| CodeLlama:13B-4bit-quantised | 32.7 | N/A | Independent | medium | Dec 22, 2023 | arxiv.org |
| CodeLlama:7B-4bit-quantised | 28.9 | N/A | Independent | medium | Dec 22, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
$\textbf{Turbulence}$ is a new benchmark for systematically evaluating the correctness and robustness of instruction-tuned large language models (LLMs) for code generation. Turbulence consists of a large set of natural language question templates, each of which is a programming problem, parameterised so that it can be asked in many different forms. Each question template has an associated test oracle that judges whether a code solution returned by an LLM is correct. Thus, from a single question template, it is possible to ask an LLM a $\textit{neighbourhood}$ of very similar programming questions, and assess the correctness of the result returned for each question. This new benchmark systematically and automatically identifies cases where LLMs are able to solve some problems in a neighbourhood but do not manage to generalise to solve the whole neighbourhood. Therefore, this method is effective at highlighting robustness issues.