Skip to content

Fixing currently_syncing with no bookmark sync issue.

Compare
Choose a tag to compare
@s7clarke10 s7clarke10 released this 05 Oct 04:54
· 14 commits to master since this release
d91d73d
Feature/fix currently syncing rerun bug (#10)

* Fix to stream not running currently_syncing, no bookmark

* Bumping version

* Ensuring all currently_syncing streams are loaded

We have encountered a scenario where a stream starts but it failed and there was no bookmark written.

When the job is re-run the state file had currently_syncing for the given stream but with no bookmark written it didn't appear in the streams_with_state list. This small change will ensure that the currently_syncing_list will be populated whether there is a bookmark or not by filtering against the ordered_streams list which includes both streams_without_state and streams_with_state.

Example:

state.json looks like the example below. In this scenario the "YELLOW_BANANAS" stream would not sync with the previous code because it is Currently Syncing with no bookmark / state.

{"currently_syncing": "YELLOW_BANANAS"}