ARC-Challenge
Harbor: pendingARC-Challenge — AI evaluation benchmark listed by opencompass.
Incomplete14%Knowledge & Reasoning
- Samples
- N/A
- Released
- N/A
- Scores
- 26
siqa
Social Interaction QA
We introduce Social IQa, the first largescale benchmark for commonsense reasoning about social situations.
How to run
ltb · siqa
$ npm i -g lettokenburn-cli$ ltb bench pull siqa --n 50$ ltb bench run ./siqa -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
8 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| phi-1.5-web 1.3B | 53.0 | N/A | Independent | medium | Sep 11, 2023 | arxiv.org |
| phi-1.5 1.3B | 52.6 | N/A | Independent | medium | Sep 11, 2023 | arxiv.org |
| LLaMA 65B | 52.3 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| Chinchilla | 51.3 | N/A | Independent | medium | Mar 29, 2022 | arxiv.org |
| Gopher | 50.6 | N/A | Independent | medium | Dec 8, 2021 | arxiv.org |
| LLaMA 13B | 50.4 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 33B | 50.4 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
| LLaMA 7B | 48.9 | N/A | Independent | medium | Feb 27, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
> We introduce Social IQa, the first largescale benchmark for commonsense reasoning about social situations. Social IQa contains 38,000 multiple choice questions for probing emotional and social intelligence in a variety of everyday situations (e.g., Q: "Jordan wanted to tell Tracy a secret, so Jordan leaned towards Tracy. Why did Jordan do this?" A: "Make sure no one else could hear"). Through crowdsourcing, we collect commonsense questions along with correct and incorrect answers about social interactions, using a new framework that mitigates stylistic artifacts in incorrect answers by asking workers to provide the right answer to a different but related question. Empirical results show that our benchmark is challenging for existing question-answering models based on pretrained language models, compared to human performance (>20% gap). Notably, we further establish Social IQa as a resource for transfer learning of commonsense knowledge, achieving state-of-the-art performance on multiple commonsense reasoning tasks (Winograd Schemas, COPA).
Available in EleutherAI lm-evaluation-harness as task family siqa (1 tasks).