Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Parse JSON Notices Section #2669

Closed
wants to merge 0 commits into from

Conversation

Vidushi-GitHub
Copy link
Member

@Vidushi-GitHub Vidushi-GitHub commented Nov 4, 2024

This document includes how to parse, encode, decode the JSON Schema files.

Resolves: #2149

Previous branch (#2506) had 100+ commits behind, rebase was messy and touched many files.

Feedbacks need to be included:
-Provide hyperlinks to these Python methods.
-Please provide sample code for parsing Kafka records in a message consuming loop

Screenshot 2024-11-04 at 10 55 57

Copy link
Member

@tylerbarna tylerbarna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to run prettier

Copy link
Member

@lpsinger lpsinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rerun pre-commit hooks.

Copy link

codecov bot commented Nov 6, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 5.93%. Comparing base (9a85dc8) to head (8e485aa).
Report is 39 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##            main   #2669      +/-   ##
========================================
- Coverage   6.06%   5.93%   -0.14%     
========================================
  Files        171     171              
  Lines       4334    4329       -5     
  Branches     476     474       -2     
========================================
- Hits         263     257       -6     
- Misses      4069    4070       +1     
  Partials       2       2              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jracusin
Copy link
Contributor

This encoding information needs to also be referenced somewhere in the creating JSON notices documentation.

@jracusin
Copy link
Contributor

Add example of pulling message off the network and getting bytes out.
Need sample file available for people to play around with the data.

@Vidushi-GitHub
Copy link
Member Author

@dakota002 please provide your suggestions.

Comment on lines 218 to 229
# Function to validate Base64 strings
def is_base64(s):
try:
base64.b64decode(s, validate=True)
return True
except Exception:
return False

# Output: A Base64 encoded bytestring, e.g., b'a1512dabc1b6adb3cd1b6dcb6d4c6......'
with open("path/to/encoded_file.txt", 'wb') as encoded_file:
encoded_file.write(encoded_string)
# Validate the skymap string
if not is_base64(skymap_string):
print("Invalid Base64 string.")
continue
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove it, it's just a check if file wasn't corrupted.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove.

import base64

# Set Kafka Topic and Producer Configuration
TOPIC = "gcn.circulars"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem like the right topic.

# Connect as a Kafka consumer
consumer = Consumer(client_id='fill me in', # Replace with your client ID
client_secret='fill me in', # Replace with your client secret
config={"message.max.bytes": 204194304},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this for?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


producer = Producer(client_id='fill me in', # Replace with your client ID
client_secret='fill me in', # Replace with your client secret
config={"message.max.bytes": 204194304},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need this?

@lpsinger
Copy link
Member

lpsinger commented Feb 3, 2025

Note that there are merge conflicts. Please fix them.

@Vidushi-GitHub
Copy link
Member Author

Note that there are merge conflicts. Please fix them.

I was rebasing for merge conflicts, looks it got closed :(

@Vidushi-GitHub
Copy link
Member Author

Still working on this branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Parsing JSON Notices
5 participants