Convert document context information when importing a table from Watson Discovery

[frreiss]

The output of Watson Discovery's Table Understanding contains information about the area of the document where the table resides. Specifically, the following keys are present in the JSON returned:

• "location" --> Span of the table in the converted HTML version of the document
• "section_title" --> Span of the title of the section containing the table in the converted HTM
• "contexts" --> Spans of key phrases appearing near the table

Currently, the we make limited use of these spans. We use the text "section_title" for identifying a single table when the user passes the select_table argument to io.watson.tables.parse_response(). And parse_response() returns the character offsets from the "location" span as a Python tuple of integers. The contents of "contexts" are not processed.

io.watson.tables.parse_response() should convert all of this context information into Text Extensions for Pandas data structures. I recommend returning a single DataFrame "context" containing two columns: "context_type" (string that can be "location", "section_title" or "contexts"), and "span" (SpanArray of spans into the HTML representation of the document).

Note: We have seen some issues with character offsets not lining up between the spans returned by Table Understanding and the converted HTML returned by Table Understanding. We should fix those issues if possible. If it's not possible to get accurate character offsets, then I recommend replacing the "span" column in the above design with a Pandas Interval plus a string. Make sure to document why the spans are returned this way if you use this workaround.