OpenAPI completion refined
A human-refined dataset of OpenAPI definitions based on the APIs.guru OpenAPI directory.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info openapi-completion-refined
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
1 sourced entry| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Code Llama - Instruct 7B | 31.1 | N/A | Independent | medium | May 24, 2024 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
A human-refined dataset of OpenAPI definitions based on the APIs.guru OpenAPI directory.
The dataset was collected from the APIs.guru OpenAPI definitions directory. The directory contains more than 4,000 definitions in yaml format. Analysis of the repository revealed that about 75% of the definitions in the directory are produced by a handful of major companies like Amazon, Google, and Microsoft. To avoid the dataset bias towards a specific producer, the maximum number of definitions from a single producer was limited to 20. Multiple versions of the same API were also excluded from the dataset as they are likely to contain very similar definitions.