Version/aeolus #13
Draft
Version/aeolus #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Old method was not keeping ascii characters as is and instead of updating the old json.dumps method we are just shifting over to orjson like the lcs method is using which has different defaults.
Passing datetime strings to the ingest method is causing it to run about 500-600x slower than if we passed numeric timestamps. This is primarily because of try/catch method we have of determining string vs numeric. I modified the staging tables to accept a string for the datetime and then only convert the numeric timestamps. This makes the string method on par with the numeric method. Since we can control the shape of the data coming to this part I would recommend that we come up with a standard format and use the try/catch correctly, and not as an if/then.
New class is meant to be more flexible and to work for all file formats.
* Cleaned up the ingester to work better for CAC data * Added support for ingesting logging intervals * Support for ingesting instrument and status * Added flag ingest method * Fixed bug in the flag ingest process Flags with null notes where not being matched * Added the start of some testing files The test_flags script is not an automated test yet but I thought the data files and process would still be helpful to have committed * Flagging updates and hourly data rollup fixes * Some cleanup and bug fixes Added some methods that help in the testing/dev environments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Work in progress