Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pr_commits table is not ingested properly because of bookmarks #185

Open
NicolasGuary opened this issue Jan 3, 2023 · 0 comments
Open

Comments

@NicolasGuary
Copy link

Hello,
I just noticed missing data (commits) on the pr_commits table using this integration on Stitch.

The root cause I suspect for this is the incremental mode for this table and the use of bookmarks. Indeed, we fixed the lack of data by triggering a full re-sync of the integration data, but missing commits on this table continued to appear in the future.

Let's take this sample use case :

  • Integration bookmark for this table and a given repo is set to : "pr_commits": {"since": "T"}
  • Commit A is created at T-2
  • Commit B is created a T-1
  • Pull Request 1 is opened at T, and includes commits A and B
  • Commit C is created at T+1 and associated to PR 1

When ingestion will be triggered again, it will fetch PR 1 and associate it Commit C, as it has been done after T, the current bookmark for the table, but Commits A & B won't be ingested by the integration.

A quick workaround, that won't be cost efficient but will allow to fix the issue is to set this table ingestion mode to full (or allow users to customize this behavior).
It would also be possible to set an offset in days, to allow the retrieval of olders commits in the pr_commits table.

This is a a blocker as it will happen a lot when you have a git workflow using feature branches merged into a develop branch and then picking commits to be merged into main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant