Split out streams for 'lifetime' time periods

[acarter24]

Fixing issue with certain streams failing looking for an end_time when it was not present.

Also adds end_time to the PK list for Period stream to allow for collection of historical periods, otherwise the replication is on id only and overwrites last periods values.

Resolves #14 #13

[prratek]

Thanks @acarter24! This approach broadly makes sense to me and we can get around to reviewing and merging in the next couple of weeks. If you have steps to reproduce the SQLite error in particular that would make it easier for us to test that it resolves that issue (and doesn't break our usage with target-bigquery in any way)

[acarter24]

Cheers. In terms of reproducing the error, I pretty much just downloaded the tap and ran it and got the error.

meltano, version 2.16.1
python 3.10

Would the json payload from the API be useful?

[prratek]

Nope, shouldn't need the payload! Other info that would be helpful:

• What target (and variant) did you use?
• Any stream selection rules

[acarter24]

To recreate:

Loaders: https://github.com/MeltanoLabs/target-sqlite and https://github.com/jwills/target-duckdb both show the issue

Stream selection to narrow it down to failing stream only

    select:
    - user_insights_audience.*

NB if I invert the filter, clear state etc, it completes without issue :

    select:
    - !user_insights_audience.*
    - *.*

EDIT:Just FYI I was also able to solve one part of the issue by overriding key-properties in meltano.yml for the relevant tables

  - name: tap-instagram
    variant: voxmedia
    pip_url: tap-instagram
    config:
      ig_user_ids: ${TAP_INSTAGRAM_IG_USER_IDS}
      access_token: ${TAP_INSTAGRAM_ACCESS_TOKEN}
      start_date: 2023-03-01
    metadata:
      user_insights_*:
        key-properties: ["end_time", "id"]
    select:
    - '*.*'
    - '!user_insights_audience.*'