Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 2.2.1 #124

Merged
merged 3 commits into from
Mar 26, 2021
Merged

Release 2.2.1 #124

merged 3 commits into from
Mar 26, 2021

Conversation

dilyand
Copy link
Contributor

@dilyand dilyand commented Mar 15, 2021

The logging in sinkBatch can potentially be extracted to make the function a bit less busy. Looking forward to opinions on that.

@dilyand dilyand force-pushed the hotfix/sqs-limits branch from eee0183 to bd5bff8 Compare March 15, 2021 13:26
@dilyand dilyand changed the title Release 0.2.0 Release 2.2.1 Mar 15, 2021
@dilyand dilyand requested review from istreeter and benjben March 15, 2021 13:29
@benjben
Copy link
Contributor

benjben commented Mar 15, 2021

Hey @dilyand , could you link this PR to an issue, so that we can get some context about what we try to achieve please ?

Copy link
Contributor

@istreeter istreeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the most part I think this is a good change. It is clear and easy to understand even though it is implementing quite a complicated retry algorithm. I just made a few comments.

@dilyand dilyand force-pushed the hotfix/sqs-limits branch from bd5bff8 to 0d48512 Compare March 16, 2021 07:44
@dilyand
Copy link
Contributor Author

dilyand commented Mar 16, 2021

@benjben I opened #125.

Copy link
Contributor

@benjben benjben left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks much better than what we had before @dilyand !

It would be great to split KinesisSink into smaller files, e.g.

  • KinesisSink with the initialization and retry logic
  • KinesisEmitter with everything related to writing to Kinesis
  • SqsEmitter with everything related to writing to SQS
  • EventStorage with the buffer

Did you manage to test the switching mechanism ?

Not related to this PR but now that we have the split function it would be great to add it to SQS collector, so that we don't rely on humans always configuring the limits right.

@dilyand
Copy link
Contributor Author

dilyand commented Mar 18, 2021

Anonymous mode not enabled:

1.) No nuid in qs or in cookie:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2 -d '{}'

��d����0:0:0:0:0:0:0:1
�����xEzO;�������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost������$ca6003e0-587a-4c3a-97fb-e0270f8ab91e�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

2.) Nuid in qs:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2?nuid=XYZ -d '{}'

��d����0:0:0:0:0:0:0:1
�����xE}���������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��J����nuid=XYZ��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost�������XYZ�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

3.) Nuid in cookie:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2 --cookie 'sp=ABC' -d '{}'

��d����0:0:0:0:0:0:0:1
�����xE�D��������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*����Cookie: sp=ABC���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost�������ABC�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

4.) Nuid in qs and cookie:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2?nuid=XYZ --cookie 'sp=ABC' -d '{}'

��d����0:0:0:0:0:0:0:1
�����xE��,�������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��J����nuid=XYZ��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*����Cookie: sp=ABC���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost�������XYZ�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

Anonymous mode enabled:

1.) No nuid in qs or in cookie:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2 -H 'SP-Anonymous: *' -d '{}'

��d����unknown
�����xE{���������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*����SP-Anonymous: *���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost������$00000000-0000-0000-0000-000000000000�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

2.) Nuid in qs:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2?nuid=XYZ -H 'SP-Anonymous: *' -d '{}'

��d����unknown
�����xE}�u�������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��J����nuid=XYZ��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*����SP-Anonymous: *���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost������$00000000-0000-0000-0000-000000000000�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

3.) Nuid in cookie:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2 --cookie 'sp=ABC' -H 'SP-Anonymous: *' -d '{}'

��d����unknown
�����xE����������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*����SP-Anonymous: *���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost������$00000000-0000-0000-0000-000000000000�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

4.) Nuid in qs and cookie:

curl -i -X POST localhost:12345/com.snowplowanalytics.snowplow/tp2?nuid=XYZ --cookie 'sp=ABC' -H 'SP-Anonymous: *' -d '{}'

��d����unknown
�����xE��\�������UTF-8�������ssc-2.2.1-rc1-stdout$��,����curl/7.54.0��@���#/com.snowplowanalytics.snowplow/tp2��J����nuid=XYZ��T����{}��^���������Timeout-Access: ����Host: localhost:12345����User-Agent: curl/7.54.0����Accept: */*����SP-Anonymous: *���!application/x-www-form-urlencoded��h���!application/x-www-form-urlencoded������ localhost������$00000000-0000-0000-0000-000000000000�zi���Aiglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0�

@dilyand dilyand force-pushed the hotfix/sqs-limits branch from 2a7f767 to 4e58556 Compare March 24, 2021 10:46
Copy link
Contributor

@istreeter istreeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

@dilyand dilyand force-pushed the hotfix/sqs-limits branch from 4e58556 to b253b68 Compare March 26, 2021 09:57
@dilyand dilyand merged commit b253b68 into master Mar 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants