ARC-Challenge
Harbor: pendingARC-Challenge — AI evaluation benchmark listed by opencompass.
Incomplete14%Knowledge & Reasoning
- Samples
- N/A
- Released
- N/A
- Scores
- 26
WiC
Words in Context
WiC is a benchmark for the evaluation of context-sensitive word embeddings.
How to run
ltb · wic
$ npm i -g lettokenburn-cli$ ltb bench info wic
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
10 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| T5-XXL 11B | 76.9 | N/A | Independent | medium | Oct 23, 2019 | arxiv.org |
| PaLM 2-L | 66.8 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| FLAN-T5-Large 783M | 64.7 | N/A | Independent | medium | Apr 27, 2023 | arxiv.org |
| OPT 1.3B | 56.1 | N/A | Independent | medium | May 24, 2024 | arxiv.org |
| OPT-125M | 53.4 | N/A | Independent | medium | May 24, 2024 | arxiv.org |
| PaLM 2-M | 52.0 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| PaLM 2-S | 50.6 | N/A | Independent | medium | May 17, 2023 | arxiv.org |
| GPT-2-XL 1.5B | 49.8 | N/A | Independent | medium | Apr 27, 2023 | arxiv.org |
| UL2 20B | 49.8 | N/A | Independent | medium | May 10, 2022 | arxiv.org |
| GPT-3-175B | 49.4 | N/A | Independent | medium | May 28, 2020 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
WiC is a benchmark for the evaluation of context-sensitive word embeddings. WiC is framed as a binary classification task. Each instance in WiC has a target word w, either a verb or a noun, for which two contexts are provided. Each of these contexts triggers a specific meaning of w. The task is to identify if the occurrences of w in the two contexts correspond to the same meaning or not. In fact, the dataset can also be viewed as an application of Word Sense Disambiguation in practise.
Source: [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](/paper/wic-10000-example-pairs-for-evaluating)