diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index 2df36fd6..a7e1775a 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -279,3 +279,57 @@ await spawn({ ``` This lets you simulate a soft timeout and validate that your worker shuts down. + +## Time-scoped syncs + +Time-scoped syncs allow using a custom timestamp to control the scope of data extraction. This capability enables more granular control over which data gets synchronized between external systems and DevRev. + +Enable time-scoped syncs by first adding the capability to your manifest: + +```yaml +imports: + - # slug and other import information ... + capabilities: + - TIME_SCOPED_SYNCS +``` + +Your data extraction implementation must handle two optional parameters from the event's `EventContext`: + +- **`extract_from`**: Timestamp in RFC3339 format indicating the starting point of extraction. This applies to both initial and incremental syncs. +- **`reset_extract_from`**: A boolean flag for incremental syncs that determines whether data should be re-extracted. + +The extraction logic depends on the sync type and parameter combination: + +**Initial syncs**: Use `extract_from` as the starting timestamp for data extraction. + +**Incremental syncs**: +- If `reset_extract_from` is `true`: Start from `extract_from` if it is provided; otherwise, extract all data. +- If `reset_extract_from` is `false` or not provided: Use the `adapter.state.lastSuccessfulSyncStarted` timestamp. + +```typescript +const { reset_extract_from, extract_from } = adapter.event.payload.event_context; + +// The start of a new sync. +if (adapter.event.payload.event_type === EventType.ExtractionDataStart) { + + // Handle extract_from parameter for any sync type + if (extract_from) { + console.log(`Starting extraction from given timestamp: ${extract_from}.`); + // ... + } + + // Handle incremental sync logic + if (adapter.event.payload.event_context.mode === SyncMode.INCREMENTAL) { + + // If `reset_extract_from` is true, the extraction should start from extract_from (if provided) + // or from the beginning (if extract_from is not provided). + if (reset_extract_from) { + console.log(`reset_extract_from is true. Starting extraction from provided timestamp (${extract_from}) or from the beginning.`); + // If reset_extract_from is false or not provided, it should use the lastSuccessfulSyncStarted timestamp to get only the new or updated data. + } else { + console.log(`Starting extraction from lastSuccessfulSyncStarted: (${adapter.state.lastSuccessfulSyncStarted}).`); + // ... + } + } +} +``` \ No newline at end of file diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 9db5da85..7e43bc52 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -100,4 +100,10 @@ The loader receives information about changes in DevRev since the last successfu A 2-way sync consists of the following phases: 1. Data loading -2. Attachments loading \ No newline at end of file +2. Attachments loading + +### Time-scoped sync + +Normally, an initial sync extracts all data from an organization, and an incremental sync only extracts changes since the last successful sync. A _time-scoped sync_, on the other hand, allows for the extraction of data beginning from a custom timestamp. + +This approach is useful for initially testing the integration with a smaller data set—such as data from the last month—before conducting a full import of all available data.