TACO-BAAI
Topics in Algorithmic Code generation dataset
TACO (Topics in Algorithmic Code generation dataset) is a dataset focused on algorithmic code generation, designed to provide a more challenging training dataset and evaluation benchmark for the code generation model field.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info taco-baai
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4 | 31.5 | N/A | Independent | medium | Dec 22, 2023 | arxiv.org |
| StarCoder 15.5B | 11.6 | N/A | Independent | medium | Dec 22, 2023 | arxiv.org |
| CodeLlama-7B-Python | 9.3 | N/A | Independent | medium | Dec 22, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
TACO (Topics in Algorithmic Code generation dataset) is a dataset focused on algorithmic code generation, designed to provide a more challenging training dataset and evaluation benchmark for the code generation model field. The dataset consists of programming competition problems that are more difficult and closer to real programming scenarios. It emphasizes improving or evaluating the model's understanding and reasoning abilities in practical application scenarios, rather than just implementing predefined function functionalities.