Natural QA

The NaturalQA dataset is from the paper: https://ai.google/research/pubs/pub47761

Original repository can be found at: https://github.com/google-research-datasets/natural-questions

This scenario is adapted from https://huggingface.co/datasets/natural_questions

NaturalQA is a dataset containing 307,373 training examples with one-way annotations, 7,830 development examples with 5-way annotations, and 7,842 5-way annotated test examples. Each example consists of a context (a wikipedia document), a question, and one or five manually annotated long and short answers. The short answer is either a set of entities in the long answer, yes/no or Null.

In this scenario, we restrict our attention to short answers. For efficiency, we use only the dev set---splitting in into train/validation. Additionally, we omit all samples in the dev set for which none of the annotators provided a short answer (and exclude the separate yes/no field). We only provide a single (randomly chosen) answer during training, and the set of all possible answers during validation.

We consider three modes of this scenario:

1. closed book: No context provided
2. open book w/ wiki document: The entire wiki document is used as context
3. open book w/ long answer: Only the long answer marked by the annotators is

provided as the context.

The motivation to consider (3) is that the entire wiki document may not fit into the language model's context window.

Concretely, we prompt models using the following format:

(Optional) Title: <title_1> (Optional) Context: <context text_1> Question: <question_1> Answer: <answer_1> (Optional) Title: <title_2> (Optional) Context: <context text_2> Question: <question_2> Answer: <answer_2> ... Optional) Title: <title_k> (Optional) Context: <context text_k> Question: <question_k> Answer: Target completion: <answer>

Example (mode:closed):

Question: how many customers does edf have in the uk
Answer: '5.7 million'

Question: who is the largest supermarket chain in the uk

Reference

['Tesco', 'Aldi']

Example (mode:open_longans)

Context: A dissenting opinion (or dissent) is an opinion in a legal case in certain legal
systems written by one or more judges expressing disagreement with the majority opinion
of the court which gives rise to its judgment. When not necessarily
referring to a legal decision, this can also be referred to as a minority report.[1][2]

Question: a justice of the supreme court may write a dissenting opinion to
Answer: 'the majority opinion of the court'

Context: Set and filmed in New York City and based on the 1997 book of the same name by
Candace Bushnell, the show follows the lives of a group of four women—three in their
mid-thirties and one in her forties—who, despite their different natures and
ever-changing sex lives, remain inseparable and confide in each other. Starring Sarah
Jessica Parker (as Carrie Bradshaw), Kim Cattrall (as Samantha Jones), Kristin Davis
(as Charlotte York), and Cynthia Nixon (as Miranda Hobbes), the quirky series had multiple
continuing storylines that tackled relevant and modern social issues such as sexuality,
safe sex, promiscuity, and femininity, while exploring the difference between friendships
and romantic relationships. The deliberate omission of the better part of the early
lives of the four women was the writers' way of exploring social life – from sex to
relationships – through each of their four very different, individual perspectives.

Question: where does sex and the city take place

Reference

['New York City']

Example (mode:wiki)

Title: Upstream (petroleum industry)

Context: Upstream ( petroleum industry ) - wikipedia  Upstream ( petroleum industry )  Jump to :
navigation, search For other uses, see Upstream (disambiguation).  The oil and gas industry
is usually divided into three major sectors : upstream
( or exploration and production - E&P),...

Question: what is upstream project in oil and gas
Answer: 'searching for potential underground or underwater crude oil and natural gas fields,
drilling exploratory wells, and subsequently drilling and operating the wells that recover and
bring the crude oil or raw natural gas to the surface'

Title: Collective Soul

Context: Collective Soul - Wikipedia  Collective Soul  Jump to : navigation , search
For other uses , see Collective Soul (disambiguation ) .      This article needs additional
citations for verification .  Please help improve this article by adding citations to
reliable sources . Unsourced material may be challenged and removed .( September 2009 )
( Learn how and when to remove this template message )       Collective Soul     Collective Soul
performing at MMRBQ 2016 , Camden NJ May 21 , 2016 ...

Question: who is the lead singer of collective soul

Reference

['Ed Roland']