Implement State Capability

[pnadolny13]

Related to discussions in #11, we should implement state here.

[pnadolny13]

An example implementation is in tap-spreadsheets-anywhere

[aaronsteers]

@pnadolny13 - The above mentioned example is a file-based bookmark, where the update timestamp of the file is the bookmarked attribute.

The CSV use case, however, seems better suited for a hash-key-based bookmark, where each record is hashed and the bookmark is essentially an array of "seen" records.

Wdyt of file-based vs record-based implementation here?

In the file-based model, all records in the file will come through if the file is modified. In the record-based model, any records already seen would be suppressed from an incremental stream.

The reason I ask is because we'd benefit from a reference implementation of the spec here:

• Feature: duplicate-proof replication using record hashes in STATE meltano/sdk#161

[TyShkan]

@aaronsteers @pnadolny13 I'd appreciate both approaches being working together or independently:

1. Skip files that were already processed and weren't modified since the last sync (e.g., bookmark for a file name and it's modification time)
2. Skip lines that were already processed (e.g., using the last synced line number for a file)
3. Update objects that were changed using known keys (e.g., when receive a new state for the same object with a known id or columns that represent an unique key)

Use cases:

• new files could be added to a sync folder/bucket for the same stream
• new lines could be appended to exist files