Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSONStore: read_json_file enhancements #1001

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

rkingsbury
Copy link
Collaborator

@rkingsbury rkingsbury commented Oct 8, 2024

JSONStore uses the read_json_file method during connect to aggregate contents of one or more json files into a MemoryStore. This PR contains several quality of life improvements to make this method more reliable and easier to debug.

Changes

  1. Currently, if any of the json files is improperly formatted or otherwise causes an error, connect fails and it is very difficult to determine the reason. This PR the read_json_file call in a try/except block and adds logging messages that make it easier to tell which file is causing the problem, and what the error is.
  2. Currently, read_json_files uses zopen to access each file in read-write mode. This is unnecessary and can contradict the spirit of the read_only kwarg. This PR change the zopen call to read-only.
  3. If a JSON file contents do not contain a last_updated field, one will now be added (and set to the current time)
  4. If a JSON file DOES contain a last_updated field but was serialized using monty dumpfn, the field will be a complete dict version of a datetime object, e.g.
"last_updated": {
      "@class": "datetime",
      "@module": "datetime",
      "string": "2024-08-07 21:24:31.370707+00:00"
    },

This PR adds logic to read_json_files that will replace last_updated with just the contents of string, because that is what maggma expects.

@rkingsbury rkingsbury changed the title JSONStore: log and and fail gracefully in read_json JSONStore: read_json_file enhancements Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant