Natural Questions (NQ) contains real user questions issued to Google search, and answers found from Wikipedia by annotators. NQ is designed for the training and evaluation of automatic question answering systems.
Please see http://ai.google.com/research/NaturalQuestions to get the data and view the leaderboard. For more details on the design and content of the dataset, please see the paper Natural Questions: a Benchmark for Question Answering Research. To help you get started on this task we have provided some baseline systems that can be branched.
NQ contains 307,372 training examples, 7,830 examples for development, and we withold a further 7,842 examples for testing. In the paper, we demonstrate a human upper bound of 87% F1 on the long answer selection task, and 76% on the short answer selection task.
To run on the hidden test set, you will have to upload a Docker image containing your system to the NQ competition site. Instructions on building the Docker image are given here.
Each example in the original NQ format contains the rendered HTML of an entire Wikipedia page, as well as a tokenized representation of the text on the page.
This section will go on to define the full NQ data format, but we recognize
that most users will only want a version of the data in which the text has
already been extracted. We have supplied a
simplified version of the training set
and we have also supplied a simplify_nq_example function
in data_utils.py which maps from the original format to the
simplified format. Only the original format is provided by our
competition site.
If you use the simplified data, you should call simplify_nq_example on each
example seen during evaluation and you should provide predictions using the
start_token and end_token offsets that correspond to the whitespace
separated tokens in the document text.
As well as recognizing predictions according to token offsets, the evaluation script also recognizes predictions as byte offsets into the original HTML. This allows users to define their own text extraction and tokenization schemes.
To help you explore the data, this repository also contains a simple data browser that you can run on your own machine, and modify as you see fit. We also have provided extra preprocessing utilities and tensorflow dataset code in the repository containing the baseline systems presented in our paper. The rest of this section describes the data format thouroughly in reference to a toy example.
Each example contains a single question, a tokenized representation of the question, a timestamped Wikipedia URL, and the HTML representation of that Wikipedia page.
"question_text": "who founded google",
"question_tokens": ["who", "founded", "google"],
"document_url": "http://www.wikipedia.org/Google",
"document_html": "<html><body><h1>Google</h1><p>Google was founded in 1998 by ..."We release the raw HTML, since this is what was seen by our annotators, and we would like to support approaches that make use of the document structure. However, we expect most initial efforts will prefer to use a tokenized representation of the page.
"document_tokens":[
{ "token": "<h1>", "start_byte": 12, "end_byte": 16, "html_token": true },
{ "token": "Google", "start_byte": 16, "end_byte": 22, "html_token": false },
{ "token": "inc", "start_byte": 23, "end_byte": 26, "html_token": false },
{ "token": ".", "start_byte": 26, "end_byte": 27, "html_token": false },
{ "token": "</h1>", "start_byte": 27, "end_byte": 32, "html_token": true },
{ "token": "<p>", "start_byte": 32, "end_byte": 35, "html_token": true },
{ "token": "Google", "start_byte": 35, "end_byte": 41, "html_token": false },
{ "token": "was", "start_byte": 42, "end_byte": 45, "html_token": false },
{ "token": "founded", "start_byte": 46, "end_byte": 53, "html_token": false },
{ "Token": "in", "start_byte": 54, "end_byte": 56, "html_token": false },
{ "token": "1998", "start_byte": 57, "end_byte": 61, "html_token": false },
{ "token": "by", "start_byte": 62, "end_byte": 64, "html_token": false },Each token is either a word or a HTML tag that defines a heading, paragraph,
table, or list. HTML tags are marked as such using the boolean field html_token.
Each token also has an inclusive start_byte and exclusive end_byte that
identifies the token's position within the example's UTF-8 indexed HTML string.
The first task in Natural Questions is to identify the smallest HTML bounding box that contains all of the information required to infer the answer to a question. These long answers can be paragraphs, lists, list items, tables, or table rows. While the candidates can be inferred directly from the HTML or token sequence, we also include a list of long answer candidates for convenience. Each candidate is defined in terms of offsets into both the HTML and the document tokens. As with all other annotations, start offsets are inclusive and end offsets are exclusive.
"long_answer_candidates": [
{ "start_byte": 32, "end_byte": 106, "start_token": 5, "end_token": 22, "top_level": true },
{ "start_byte": 65, "end_byte": 102, "start_token": 13, "end_token": 21, "top_level": false },In this example, you can see that the second long answer candidate is contained
within the first. We do not disallow nested long answer candidates, we just ask
annotators to find the smallest candidate containing all of the information
required to infer the answer to the question. However, we do observe that 95%
of all long answers (including all paragraph answers) are not nested below any
other candidates.
Since we believe that some users may want to start by only considering
non-overlapping candidates, we include a boolean flag top_level that
identifies whether a candidate is nested below another (top_level = False) or
not (top_level = True). Please be aware that this flag is only included for
convenience and it is not related to the task definition in any way.
For more information about the distribution of long answer types, please
see the data statistics section below.
The NQ training data has a single annotation with each example and the evaluation
data has five. Each annotation defines a "long_answer" span, a list of
short_answers, and a yes_no_answer. If the annotator has marked a long
answer, then the long answer dictionary identifies this long answer using byte
offsets, token offsets, and an index into the list of long answer candidates. If
the annotator has marked that no long answer is available, all of the fields in
the long answer dictionary are set to -1.
"annotations": [{
"long_answer": { "start_byte": 32, "end_byte": 106, "start_token": 5, "end_token": 22, "candidate_index": 0 },
"short_answers": [
{"start_byte": 73, "end_byte": 78, "start_token": 15, "end_token": 16},
{"start_byte": 87, "end_byte": 92, "start_token": 18, "end_token": 19}
],
"yes_no_answer": "NONE"
}]Each of the short answers is also identified using both byte offsets and token indices. There is no limit to the number of short answers. There is also often no short answer, since some questions such as "describe google's founding" do not have a succinct extractive answer. When this is the case, the long answer is given but the "short_answers" list is empty.
Finally, if no short answer is given, it is possible that there is a
yes_no_answer for questions such as "did larry co-found google". The values
for this field YES, or NO if a yes/no answer is given. The default value is
NONE when no yes/no answer is given. For statistics on long answers, short
answers, and yes/no answers, please see the data statistics section below.
The NQ training data contains 307,373 examples. 152,148 have a long answer and 110,724 have a short answer. Short answers can be sets of spans in the document (106,926), or yes or no (3,798). Long answers are HTML bounding boxes, and the distribution of NQ long answer types is as follows:
| HTML tags | Percent of long answers |
|---|---|
<P> |
72.9% |
<Table> |
19.0% |
<Tr> |
1.5% |
<Ul>, <Ol>, <Dl> |
3.2% |
<Li>, <Dd>, <Dt> |
3.4% |
While we allow any paragraph, table, or list element to be a long answer,
we find that 95% of the long answers are not contained by any other
long answer candidate. We mark these top level candidates in the data,
as described above.
Short answers may contain more than one span, if the question is asking for a list of answers (e.g. who made it to stage 3 in american ninja warrior season 9). However, almost all short answers (90%) only contain a single span of text. All short answers are contained by the long answer given in the same annotation.
Please see the evaluation script for a description of the prediction format that your model should output.
If you have a technical question regarding the dataset, code or publication, please create an issue in this repository. This is the fastest way to reach us.
If you would like to share feedback or report concerns, please email us at [email protected].