OCW
Only Connect Wall Dataset and creative problem solving tasks
The OCW dataset is for evaluating creative problem solving tasks by curating the problems and human performance results from the popular British quiz show Only Connect.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info ocw
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
10 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-3.5 Turbo | 82.5 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-3.5 Turbo | 82.3 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-3.5 Turbo | 81.2 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-3.5 Turbo | 80.9 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-3.5 Turbo | 80.6 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-4 | 75.8 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-4 | 73.7 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-4 | 73.6 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-4 | 73.4 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
| GPT-4 | 72.9 | N/A | Independent | medium | Mar 15, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
The OCW dataset is for evaluating creative problem solving tasks by curating the problems and human performance results from the popular British quiz show Only Connect.
The OCW dataset contains 618 connecting wall puzzles and solutions in total from 15 seasons of the show. Each show episode has two walls.
The dataset has two tasks: Task 1 (Grouping), and Task 2 (Connections) are identical to the quiz-show’s human participant tasks.
Task 1 (Groupings) is evaluated via six metrics: number of solved walls, number of correct groups (max. four per wall), Adjusted Mutual Information (AMI), Adjusted Rand Index (ARI), Fowlkes Mallows Score (FMS), and Wasserstein Distance (WD), normalized to (0, 1) range, between predicted and ground-truth labels.
Task 2 (Connections) is evaluated with three metrics: exact string matching, ROUGE-1 F1, and BERTScore F1.
Baseline results with pre-trained language models and with few-shot In-context Learning (ICL) with LLMs such as GPT-4 are available here:
"Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset" Saeid Alavi Naeini, Raeid Saqur, Mozhgan Saeidi, John Giorgi, Babak Taati. 2023 https://neurips.cc/virtual/2023/poster/73547