user_insights_daily only capturing the last day when using target-sqlite

[acarter24]

I am trying to get the history of insights by day over time. With the user_insights_* feeds, I am only seeing the latest days data persisted to sqlite, despite setting the appropriate date boundaries in the config.

I can see the relevant queries in the Graph API query history and meltano is paginating through previous dates, but only the latest gets inserted into the table.

Is it somehow related to the primary_keys = ["id"] set on these streams? Does the PK need to be [id, end_time] to retain all the rows? When I run the extractor using target-jsonl I can see all the previous days in the output file which makes me think it is PK/upsert related.

tap-instagram/tap_instagram/streams.py

Line 725 in a617855

primary_keys = ["id"]

EDIT: Just starting to realise I think this this is related to #13 :)