JSONDecodeError when retrieving german nouns

[shybyte]

Terms

• I have searched all open bug reports
• I agree to follow Scribe-Data's Code of Conduct

Behavior

Using Scribe-Data v4.1.0,
when I try to retrieve German nouns, then I get after some time a json.decoder.JSONDecodeError:

scribe-data g --language German --data-type nouns       
Updating data for language(s): German; data type(s): Nouns
Data updated:   0%|                                                          | 0/1 [00:00<?, ?process/s]Querying and formatting German nouns
Data updated:   0%|                                                          | 0/1 [01:02<?, ?process/s]
Traceback (most recent call last):
  File "/home/shybyte/.local/bin/scribe-data", line 8, in <module>
    sys.exit(main())
  File "/home/shybyte/.local/lib/python3.10/site-packages/scribe_data/cli/main.py", line 304, in main
    get_data(
  File "/home/shybyte/.local/lib/python3.10/site-packages/scribe_data/cli/get.py", line 153, in get_data
    query_data(
  File "/home/shybyte/.local/lib/python3.10/site-packages/scribe_data/wikidata/query_data.py", line 236, in query_data
    results = sparql.query().convert()
  File "/home/shybyte/.local/lib/python3.10/site-packages/SPARQLWrapper/Wrapper.py", line 1196, in convert
    return self._convertJSON()
  File "/home/shybyte/.local/lib/python3.10/site-packages/SPARQLWrapper/Wrapper.py", line 1059, in _convertJSON
    json_str = json.loads(self.response.read().decode("utf-8"))
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 655738 column 9 (char 15065703)

It works fine for other data types like for example verbs.

This might be related to the closed issue #124.

[andrewtavis]

I can confirm that this is on the current version of main as well. Thanks for opening this, @shybyte!

CC @axif0

[andrewtavis]

Just tried to edit the query and removed plural and gender as responses and it does finish. Splitting it into three queries times out, but two does work 🤔 The thing is that the query_data_type_1.sparql, query_data_type_2.sparql, etc method doesn't appear to work as the original nouns JSON response is overwritten.

[andrewtavis]

49df8d8 is the current state of it. @axif0, could you take a look at why the nouns.json file is being overwritten if we try to split noun queries in the same way as we do for verbs? From there, I think that something that we should consider is figuring out the exact error that's being thrown and to return an explicit error message that suggests that people download a Wikidata lexeme dump and run the get process in that way :)

That would be the quick response, and from there we can look into #156 that would be likely where we'd need to go to get the queries to always work.

[andrewtavis]

Feel free to prioritize as you see fit, @axif0 :) No need to jump over to this if you're working on your blogpost still or are working on another issue. We should be able to finalize this in the coming days :)