SVAMP
Simple Variations on Arithmetic Math word Problems
SVAMP — AI evaluation benchmark listed by opencompass.
How to run
$ npm i -g lettokenburn-cli$ ltb bench pull svamp
External harness. pip install opencompass && opencompass --models <model_cfg> --datasets svamp_gen (OpenCompass dataset config dir 'SVAMP') https://github.com/open-compass/opencompass
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entriesEvery number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
A challenge set for elementary-level Math Word Problems (MWP). An MWP consists of a short Natural Language narrative that describes a state of the world and poses a question about some unknown quantities.
The examples in SVAMP test a model across different aspects of solving MWPs: 1) Is the model question sensitive? 2) Does the model have robust reasoning ability? 3) Is it invariant to structural alterations?