Scan queries documentation examples are missing the json key rowSignature

[thomas-dufour]

The documentation examples (https://druid.apache.org/docs/latest/querying/scan-query/#example-results) are missing the json key rowSignature.

Side note: this is a an undisclosed breaking change introduced in version 27 in my opinion. The release note https://github.com/apache/druid/releases/tag/druid-27.0.0 never mentioned anything about rowSignature or an updated payload for scan queries. But what matters now is to fix the documentation.

Affected Version

Druid version 31 and Druid version 27 (probably all versions >= 27 but I only tested the most recent one)

Description

Since version 27, a breaking change in the payload format of POST /druid/v2 was introduced and the documentation for scan queries is out of date.
The documentation example can be found here https://druid.apache.org/docs/latest/querying/scan-query/#example-results.
It shows an example

 [{
    "segmentId" : "wikipedia_editstream_2012-12-29T00:00:00.000Z_2013-01-10T08:00:00.000Z_2013-01-10T08:13:47.830Z_v9",
    "columns" : [
      "timestamp", "robot", "namespace", "anonymous", "unpatrolled", "page", "language", "newpage", "user", "count", "added", "delta", "variation", "deleted"
    ],
    "events" : [
     ["2013-01-01T00:00:00.000Z", "1", "article", "0", "0", "11._korpus_(NOVJ)", "sl", "0", "EmausBot", 1.0, 39.0, 39.0, 39.0, 0.0],
     ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "112_U.S._580", "en", "1", "MZMcBride", 1.0, 70.0, 70.0, 70.0, 0.0],
     ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._243", "en", "1", "MZMcBride", 1.0, 77.0, 77.0, 77.0, 0.0]
    ]
} ]

But it is missing a json key rowSignature. The actual payload looks like this

[ {
  "segmentId" : "wikipedia_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2024-12-17T13:08:03.142Z",
  "columns" : [ "__time", "isRobot", "channel", "flags", "isUnpatrolled", "page", "diffUrl", "added", "comment", "commentLength", "isNew", "isMinor", "delta", "isAnonymous", "user", "deltaBucket", "deleted", "namespace", "cityName", "countryName", "regionIsoCode", "metroCode", "countryIsoCode", "regionName" ],
  "events" : [ [ 1466985611080, "true", "#sv.wikipedia", "NB", "false", "Salo Toraut", "https://sv.wikipedia.org/w/index.php?oldid=36099284&rcid=89369918", 31, "Botskapande Indonesien omdirigering", 35, "true", "false", 31, "false", "Lsjbot", 0, 0, "Main", null, null, null, null, null, null ], [ 1466985617457, "false", "#ja.wikipedia", "", "false", "value", "https://ja.wikipedia.org/w/index.php?diff=60239890&oldid=60239620", 125, "value", 4, "false", "false", 125, "false", "value", 100, 0, "value", null, null, null, null, null, null ], [ 1466985634959, "false", "#en.wikipedia", "", "false", "Bailando 2015", "https://en.wikipedia.org/w/index.php?diff=727144213&oldid=727144184", 2, "/* Scores */", 12, "false", "false", 2, "true", "181.230.118.178", 0, 0, "Main", "Buenos Aires", "Argentina", "C", null, "AR", "Buenos Aires F.D." ] ],
  "rowSignature" : [ {"name" : "__time","type" : "LONG" },
  {"name" : "isRobot","type" : "STRING" },
  {"name" : "channel","type" : "STRING" },
  {"name" : "flags","type" : "STRING" },
  {"name" : "isUnpatrolled","type" : "STRING" },
  {"name" : "page","type" : "STRING" },
  {"name" : "diffUrl","type" : "STRING" },
  {"name" : "added","type" : "LONG" },
  {"name" : "comment","type" : "STRING" },
  {"name" : "commentLength","type" : "LONG" },
  {"name" : "isNew","type" : "STRING" },
  {"name" : "isMinor","type" : "STRING" },
  {"name" : "delta","type" : "LONG" },
  {"name" : "isAnonymous","type" : "STRING" },
  {"name" : "user","type" : "STRING" },
  {"name" : "deltaBucket","type" : "LONG" },
  {"name" : "deleted","type" : "LONG" },
  {"name" : "namespace","type" : "STRING" },
  {"name" : "cityName","type" : "STRING" },
  {"name" : "countryName","type" : "STRING" },
  {"name" : "regionIsoCode","type" : "STRING" },
  {"name" : "metroCode","type" : "LONG" },
  {"name" : "countryIsoCode","type" : "STRING" },
  {"name" : "regionName","type" : "STRING" } ]
} ]

There is an additional key rowSignature for each batch json object of the response.
This is true for both list and compactedList resultFormat.

How to reproduce

Install and start Druid 31 locally. Load the Wikipedia example data included in quick-start. Run a curl command

curl "http://localhost:8888/druid/v2?pretty=null" \  
--header 'Content-Type: application/json' \
--data '{
   "queryType": "scan",
   "dataSource": "wikipedia",
   "resultFormat": "compactedList",
   "columns":[],
   "intervals": [
     "2016-01-01/2017-01-02"
   ],
   "batchSize":20480,
   "limit":3   
 }'

[kgyrtkirk]

I understand the above as that the ScanQuery has a new field named rowSignature.

I don't think that should be consider a breaking change as the json format gives the flexibilty to add new fields (afaik).
...or is there something I miss?

[thomas-dufour]

Depending on the implementation , this can be breaking for the user. In our case, we implemented a streaming parser that did not supported the new field. We can argue it is our responsibility to have a parser that handles new fields. The problem is not so much the schema been updated rather than the lack of announcement IMO.