Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

abstract pubsub service via gocloud #1664

Merged
merged 18 commits into from
Jan 29, 2024
Merged

abstract pubsub service via gocloud #1664

merged 18 commits into from
Jan 29, 2024

Conversation

pxp928
Copy link
Collaborator

@pxp928 pxp928 commented Jan 25, 2024

Description of the PR

Utilizing gocloud we can abstract out the pubsub used within GUAC. By default, it does still work with NATS Jetstream but now has the ability for other cloud provider/third-party pubsub services to be used (such as AWS, GCP, Azure. NATS, Kafka and RabbitMQ.

I tested it with AWS S3 as the blob store and AWS SQS as the pubsub with authentication handled via AWS CLI:

go run ./cmd/guacingest --blob-addr "s3://test?region=us-east-2" --pubsub-addr "awssqs://sqs.us-east-2.amazonaws.com/123456789012/myqueue?region=us-east-2"

and in another terminal:

go run ./cmd/guaccollect files --blob-addr "s3://test?region=us-east-2" --pubsub-addr "awssqs://sqs.us-east-2.amazonaws.com/123456789012/myqueue?region=us-east-2" ../guac-data/docs

Also fixes #765

Also addressed issue #460 with the addition of sending CDEvents via the pubsub

PR Checklist

  • All commits have a Developer Certificate of Origin (DCO) -- they are generated using -s flag to git commit.
  • All new changes are covered by tests
  • If GraphQL schema is changed, make generate has been run
  • If collectsub protobuf has been changed, make proto has been run
  • All CI checks are passing (tests and formatting)
  • All dependent PRs have already been merged

Copy link
Member

@jeffmendoza jeffmendoza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should figure out the TODO and commented out code before merging probably.

pxp928 added 17 commits January 27, 2024 12:28
@pxp928
Copy link
Collaborator Author

pxp928 commented Jan 27, 2024

Should figure out the TODO and commented out code before merging probably.

Removed the commented out code as we are not using it.

@kodiakhq kodiakhq bot merged commit eee82ba into guacsec:main Jan 29, 2024
8 checks passed
@pxp928 pxp928 deleted the poc-pubsub branch January 29, 2024 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix processor and parser PUBSUB tests to check for logger err msgs
3 participants