LetTokenBurnEHRSQL is a benchmark designed to evaluate models on generating structured queries
$ npm i -g lettokenburn-cli$ ltb bench pull ehr-sql
External harness. pip install crfm-helm && helm-run --run-entries ehr_sql:model=<model> --suite my-suite --max-eval-instances 100 (HELM scenario 'ehr_sql'; some scenarios need extra args) https://github.com/stanford-crfm/helm
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
No sourced scores recorded yet.
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Scenario for the EHR SQL dataset.
eicu.sqlite database is available for evaluation.eicu.sql to pass it to the LLM.value field as alternative ground truth result.