aware_debug sometimes makes invalid JSON

[rkdarst]

Hi,

I have been working to debug a server error and it seems that sometimes the aware_debug table produces invalid JSON. This has been happening for weeks now, so it is something persistent (something in the DB or code, not a one-time thing that happens on upload). The location of the first error (location in uploaded data) is at the same byte offset in the uploaded data, too, also implying it is persistent. The python error message is:

JSONDecodeError at /index.php/aware/v1/.../aware_debug/insert
Expecting ',' delimiter: line 1 column 2193 (char 2192)

Here is an example of where the error occurs:

"battery":86},141567798,"network":"wifi","

When looking at the data and replacing all the valid JSON objects, the first error is this, which exists where an object should otherwise be within the list of rows being uploaded:

141567798,"network":"wifi","battery":86}

Looking in more detail, it seems like the first part of the text of the JSON objects is missing and we consistently have the trailing end of the JSON data. The excerpt above is unique, most of the fragments look like this (this beginning and end text is usually the same, the underscores are a lot of apparently valid JSON with the keys type and timestamp)

","battery_state":1,"label":______________"network":"wifi","battery":81}

This only seems to happen in the aware_debug table, and it appears busty (many rows are fine, then most rows are not fine, etc). Some aware_debug rows were successfully uploaded a long time ago but once this starts, no more succeed (since the server isn't accepting uploads anymore). I have never seen a similar error with another table's upload.

Do you have any thoughts?

Thanks,

• Richard

[tetujin]

Hi Richard,

Thank you for your report! Hmm, it looks like JSON format error on iOS client. I'll check it and share you the reason soon.

Which version of the client are you using?

Thanks,
Yuuki

[rkdarst]

Hi,

Thanks for taking a look! The devices aren't mine so I don't know details. It is almost certainly one of the versions that has appeared in the store in the last 3 weeks. The json data sent in this table says "app_version":"1.9.12(36)". I know they are iPhone 5S (two of them).

Thanks,

• Richard

[tetujin]

Hi Richard,

Thanks! That is the latest version on AppStore.

Also, which sensors are you using in your study?
The aware_debug sensor is accessed from all sensors for saving their events. The information is very helpful for debugging the issue.

Best,
Yuuki

[rkdarst]

Hi,

Right now we aren't in a formal study and the people can turn on and off sensors as they want. I just asked one person and currently these are on:

battery, screen, fused location, ambient noise, activity recognition, device usage, conversation.

Would looking at the JSON itself provide any clues about what sensors are causing it? For example, I see these keys in it some of the broken rows:

sync_data_query_identifier_battery<TIMESTAMP>
sync_data_query_identifier_screen<TIMESTAMP>
sync_data_query_identifier_plugin_ios_activity_recognition<TIMESTAMP>

Does this mean these rows are being generated by these sensors? Most of the other keys look the same.

Thanks,

• Richard

[rkdarst]

Hi,

It's been a while, but I saw this come up again. After some work, I found that the ; character gets included in the POST body as part of build_id and serial. I guess PHP can split this properly, but RFC3986 says that semicolon is a reserved character. In Python urllib.parse and the Django QueryDict parser, it sees this as somehow splitting things and causes a failure in parsing (as seen by unterminated JSON string).

As a workaround in my server, this (python) code makes a POST body that works: body.replace(b';', b' ')).

I'll update again if I find out more.