Qu AC

The QuAC dataset is from the paper: https://arxiv.org/abs/1808.07036

The original webpage is: http://quac.ai/

QuAC is a QA dataset based on student-teacher dialogue. The student is shown the title and first paragraph of a Wikipedia page and tries to learn information about a section of the page. The training set contains 83,568 questions (11,567 dialogues), while the validation set contains 7,354 questions (1,000 dialogues).

In this Scenario, we show the model all the relevant information (title, background, section title, section text) as well as a prefix of the dialogue and ask for the answer. Each dialogue contains between 4 and 12 questions so we randomly pick a stopping point to query the model (ensuring that at least two question-answer pairs are provided. Answers are at most 30 words long.

For the validation set, there are 4 additional answers collected independently from other annotators (total 5 answers). Following the original paper, we treat all these answers as equally correct and compute the maximum F1 score of the model with respect to any of these answers.

Concretely, we prompt models using the following format:

Title: <title> Background: <first wiki paragraph> Section: <section title> Context: <section text>

Question: <question_1> Answer: <answer_1>

Question: <question_2> Answer: <answer_2>

...

Question: <question_k> Answer:

Target completion: <answer>

Note: Some of the questions do not have an answer in the context so the model needs to answer "CANNOTANSWER". While this behavior might be tricky to learn in the few-shot setting, we still include these examples in the scenario.

Example

Title: Augusto Pinochet

Background: Augusto Jose Ramon Pinochet Ugarte (; Spanish: [au'gusto
pino'(t)Se, -'(t)Set]; 25 November 1915 - 10 December 2006) was a Chilean
general, <...>

Section: Accusations of fascism
Context: Pinochet and his government have been characterised as fascist. For
example, journalist and author Samuel Chavkin, in his book Storm Over Chile:
The Junta Under Siege, <...>

Question: What were the accusations?
Answer: Griffin included Pinochet in a group of pseudo-populist despots
distinct from fascism and including the likes of Saddam Hussein, Suharto,
and Ferdinand Marcos.

Question: What he accused of being a fascist?
Answer: Pinochet attempted to build true fascism, the regime would likely
have been toppled or at least been forced to alter its relationship to the
United States.

Question: Was there conflict because of his views?
Answer: Anna Cento Bull also excluded Pinochet from fascism, although she
has argued that his regime belongs to a strand of Cold War anti-communism

Question: Is there something else interesting to know?
Answer:

Reference

["It is notable that in all the declarations of Pinochet's men, nobody has
mentioned the creators of the new Chilean society and state,"]