Skip to content

Add time-scoped sync description for Airdrop #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 14, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions fern/docs/pages/airdrop/data-extraction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -279,3 +279,57 @@ await spawn({
```

This lets you simulate a soft timeout and validate that your worker shuts down.

## Time-scoped syncs

Time-scoped syncs allow using a custom timestamp to control the scope of data extraction. This capability enables more granular control over which data gets synchronized between external systems and DevRev.

Enable time-scoped syncs by first adding the capability to your manifest:

```yaml
imports:
- # slug and other import information ...
capabilities:
- TIME_SCOPED_SYNCS
```

Your data extraction implementation must handle two optional parameters from the event's `EventContext`:

- **`extract_from`**: Timestamp in RFC3339 format indicating the starting point of extraction. This applies to both initial and incremental syncs.
- **`reset_extract_from`**: A boolean flag for incremental syncs that determines whether data should be re-extracted.

The extraction logic depends on the sync type and parameter combination:

**Initial syncs**: Use `extract_from` as the starting timestamp for data extraction.

**Incremental syncs**:
- If `reset_extract_from` is `true`: Start from `extract_from` if it is provided; otherwise, extract all data.
- If `reset_extract_from` is `false` or not provided: Use the `adapter.state.lastSuccessfulSyncStarted` timestamp.

```typescript
const { reset_extract_from, extract_from } = adapter.event.payload.event_context;

// The start of a new sync.
if (adapter.event.payload.event_type === EventType.ExtractionDataStart) {

// Handle extract_from parameter for any sync type
if (extract_from) {
console.log(`Starting extraction from given timestamp: ${extract_from}.`);
// ...
}

// Handle incremental sync logic
if (adapter.event.payload.event_context.mode === SyncMode.INCREMENTAL) {

// If `reset_extract_from` is true, the extraction should start from extract_from (if provided)
// or from the beginning (if extract_from is not provided).
if (reset_extract_from) {
console.log(`reset_extract_from is true. Starting extraction from provided timestamp (${extract_from}) or from the beginning.`);
// If reset_extract_from is false or not provided, it should use the lastSuccessfulSyncStarted timestamp to get only the new or updated data.
} else {
console.log(`Starting extraction from lastSuccessfulSyncStarted: (${adapter.state.lastSuccessfulSyncStarted}).`);
// ...
}
}
}
```
8 changes: 7 additions & 1 deletion fern/docs/pages/airdrop/getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -100,4 +100,10 @@ The loader receives information about changes in DevRev since the last successfu
A 2-way sync consists of the following phases:

1. Data loading
2. Attachments loading
2. Attachments loading

### Time-scoped sync

Normally, an initial sync extracts all data from an organization, and an incremental sync only extracts changes since the last successful sync. A _time-scoped sync_, on the other hand, allows for the extraction of data beginning from a custom timestamp.

This approach is useful for initially testing the integration with a smaller data set—such as data from the last month—before conducting a full import of all available data.
Loading