LetTokenBurnFinQA is a question answering task over financial reports that requires robust numerical reasoning.
$ npm i -g lettokenburn-cli$ ltb bench pull fin-qa
External harness. pip install crfm-helm && helm-run --run-entries fin_qa:model=<model> --suite my-suite --max-eval-instances 100 (HELM scenario 'fin_qa'; some scenarios need extra args) https://github.com/stanford-crfm/helm
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
FinQA is a question answering task over financial reports that requires robust numerical reasoning.
FinQA: A Dataset of Numerical Reasoning over Financial Data Paper: https://arxiv.org/abs/2109.00122 Code: https://github.com/czyssrs/FinQA
Presented with a financial report consisting of textual contents and a structured table, given a question, the task is togenerate the reasoning program in the domain specific langauge (DSL) that will be executed to get the answer.
We add the sub-headers "Pre-table text", "Table", "Post-table text" to the input. Example:
Pre-table text: printing papers net sales for 2006 decreased 3% ( 3 % ) from both 2005 and 2004 due principally... [more lines] Table: [["in millions", "2006", "2005", "2004"], ["sales", "$ 6930", "$ 7170", "$ 7135"], ["operating profit", "$ 677", "$ 473", "$ 508"]] Post-table text: u.s . uncoated papers net sales in 2006 were $ 3.5 billion , compared with $ 3.2 billion in 2005 and $ 3.3 billion in 2004 . [more lines] Question: brazilian paper sales represented what percentage of printing papers in 2005? Program: