Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Graphql implementation of the repo StargazersStream #123

Merged
merged 25 commits into from
May 19, 2022
Merged
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7d90097
initialization graphql stargazers
ericboucher May 17, 2022
0440ec0
Fix stargazers query
ericboucher May 18, 2022
89bb766
simplify query
ericboucher May 18, 2022
348f303
Update repository_streams.py
ericboucher May 18, 2022
42310c8
Add early exit with fake "since"
ericboucher May 18, 2022
ed77f27
Add new stream and order alphabetically
ericboucher May 18, 2022
3977dee
Add missing user params
ericboucher May 18, 2022
eec8d3d
Test different path for api_calls_tests_cache
ericboucher May 18, 2022
3b8e9b3
Retry pytest once on error
ericboucher May 18, 2022
f46c2e1
Fix push target in github action
ericboucher May 18, 2022
6ee2765
Update test_tap.yml
ericboucher May 18, 2022
7d43ea6
Update test_tap.yml
ericboucher May 18, 2022
2b433c6
Update test_tap.yml
ericboucher May 18, 2022
8ad4f69
Update test_tap.yml
ericboucher May 18, 2022
5b5b64c
Use alternative_sync_chidren in tap_core
ericboucher May 18, 2022
9e0dd6d
Update test_tap.yml
ericboucher May 18, 2022
6c58936
Update test_tap.yml
ericboucher May 18, 2022
0bb3531
Update test_tap.yml
ericboucher May 18, 2022
b58691a
Update tap_github/tests/fixtures.py
ericboucher May 18, 2022
288dd81
Override REST stream
ericboucher May 19, 2022
dc52830
Update repository_streams.py
ericboucher May 19, 2022
79d1970
Update test_tap.yml
ericboucher May 19, 2022
443825b
Revert "Override REST stream"
ericboucher May 19, 2022
1c79c06
Keep both streams but add warning
ericboucher May 19, 2022
862e7ca
Update repository_streams.py
ericboucher May 19, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion tap_github/repository_streams.py
Original file line number Diff line number Diff line change
Expand Up @@ -1479,7 +1479,7 @@ def parse_response(self, response: requests.Response) -> Iterable[dict]:
class StargazersStream(GitHubRestStream):
"""Defines 'Stargazers' stream. Warning: this stream does NOT track star deletions."""

name = "stargazers"
name = "stargazers_rest"
path = "/repos/{org}/{repo}/stargazers"
primary_keys = ["user_id", "repo", "org"]
parent_stream_type = RepositoryStream
Expand All @@ -1488,6 +1488,13 @@ class StargazersStream(GitHubRestStream):
# GitHub is missing the "since" parameter on this endpoint.
missing_since_parameter = True

def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# TODO - remove warning with next release.
self.logger.warning(
"This stream is deprecated. Please use the Graphql version instead 'stargazers'."
)
ericboucher marked this conversation as resolved.
Show resolved Hide resolved

@property
def http_headers(self) -> dict:
"""Return the http headers needed.
Expand Down Expand Up @@ -1532,6 +1539,14 @@ class StargazersGraphqlStream(GitHubGraphqlStream):
# The parent repository object changes if the number of stargazers changes.
ignore_parent_replication_key = False

def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# TODO - remove warning with next release.
self.logger.warning(
"This stream might conflict with previous implementation of 'stargazer'. "
"Looking for the older version, use 'stargazers_rest'."
)

def post_process(self, row: dict, context: Optional[Dict] = None) -> dict:
"""
Add a user_id top-level field to be used as state replication key.
Expand Down