Adds Resumable Sync for higher stability of syncing large datasets #3266

alishakawaguchi · 2025-02-15T16:41:06Z

No description provided.

vercel · 2025-02-15T16:41:10Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
neosync-docs	⬜️ Ignored (Inspect)	Visit Preview		Feb 16, 2025 10:05pm

github-actions · 2025-02-15T16:41:32Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Feb 16, 2025, 10:05 PM

codecov · 2025-02-15T16:47:26Z

Codecov Report

Attention: Patch coverage is 7.00935% with 796 lines in your changes missing coverage. Please review.

Project coverage is 22.73%. Comparing base (dd8480b) to head (506c151).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...kg/workflows/tablesync/activities/sync/activity.go	0.00%	269 Missing ⚠️
worker/pkg/benthos/sql/input_sql_raw.go	15.05%	76 Missing and 3 partials ⚠️
backend/services/mgmt/v1alpha1/job-service/runs.go	0.00%	65 Missing ⚠️
...orker/pkg/workflows/tablesync/workflow/workflow.go	0.00%	55 Missing ⚠️
worker/pkg/query-builder2/querybuilder.go	0.00%	48 Missing ⚠️
worker/pkg/workflows/datasync/workflow/workflow.go	0.00%	35 Missing ⚠️
.../activities/gen-benthos-configs/benthos-builder.go	0.00%	34 Missing ⚠️
backend/pkg/sqlmanager/mssql/mssql-manager.go	0.00%	28 Missing ⚠️
backend/pkg/mssql-querier/system.sql.go	0.00%	27 Missing ⚠️
internal/benthos-stream/client.go	0.00%	27 Missing ⚠️
... and 22 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3266      +/-   ##
==========================================
- Coverage   23.39%   22.73%   -0.66%     
==========================================
  Files         387      390       +3     
  Lines       44448    44893     +445     
==========================================
- Hits        10399    10208     -191     
- Misses      32994    33665     +671     
+ Partials     1055     1020      -35

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2025-02-15T16:52:16Z

Benchstat Geomean Results

0.18% sec/op, -0.05% B/op, 0.00% allocs/op

Benchstat results

                                   │ main-benchmark.txt │        branch-benchmark.txt        │
                                   │       sec/op       │    sec/op     vs base              │
CleanPostgresType-4                         42.63n ± 4%   42.16n ±  3%  -1.11% (p=0.039 n=6)
Runner_Single-4                             7.713µ ± 5%   7.739µ ±  4%       ~ (p=0.485 n=6)
GenerateBool-4                              11.45n ± 1%   11.43n ±  1%       ~ (p=1.000 n=6)
GenerateBusinessName-4                      285.4n ± 0%   297.9n ± 13%  +4.36% (p=0.002 n=6)
GenerateCardNumber-4                        28.41n ± 1%   28.45n ±  1%       ~ (p=0.329 n=6)
GenerateCategorical-4                       104.1n ± 2%   101.7n ±  0%  -2.26% (p=0.002 n=6)
GenerateCity-4                              173.7n ± 0%   173.4n ±  1%       ~ (p=0.297 n=6)
GenerateCountry-4                           105.2n ± 1%   105.0n ±  2%       ~ (p=1.000 n=6)
GenerateEmail-4                             1.346µ ± 0%   1.333µ ±  0%  -0.97% (p=0.002 n=6)
GenerateFirstName-4                         203.9n ± 0%   199.5n ±  1%  -2.16% (p=0.002 n=6)
GenerateFloat64-4                           28.73n ± 1%   28.82n ±  2%       ~ (p=0.084 n=6)
GenerateFullAddress-4                       1.429µ ± 3%   1.456µ ±  0%       ~ (p=0.058 n=6)
GenerateFullName-4                          1.222µ ± 1%   1.234µ ±  1%  +0.98% (p=0.019 n=6)
GenerateGender-4                            41.42n ± 0%   41.54n ±  1%  +0.28% (p=0.041 n=6)
GenerateInt64-4                             28.41n ± 2%   28.19n ±  0%       ~ (p=0.589 n=6)
GenerateInt64PhoneNumber-4                  51.46n ± 2%   53.71n ±  1%  +4.37% (p=0.002 n=6)
GenerateInternationalPhoneNumber-4          175.8n ± 1%   174.2n ±  1%  -0.91% (p=0.006 n=6)
GenerateIpAddress-4                         569.3n ± 0%   541.6n ±  2%  -4.86% (p=0.002 n=6)
GenerateLastName-4                          187.4n ± 1%   193.7n ±  1%  +3.36% (p=0.002 n=6)
GenerateRandomString-4                      985.6n ± 1%   980.5n ±  0%  -0.52% (p=0.026 n=6)
GenerateSHA256Hash-4                        1.016µ ± 0%   1.014µ ±  1%       ~ (p=0.186 n=6)
GenerateSSN-4                               305.8n ± 1%   303.3n ±  0%  -0.82% (p=0.002 n=6)
GenerateState-4                             105.2n ± 2%   105.1n ±  0%       ~ (p=0.844 n=6)
GenerateStreetAddress-4                     204.0n ± 1%   206.9n ±  1%  +1.47% (p=0.002 n=6)
GenerateStringPhoneNumber-4                 179.3n ± 0%   173.0n ±  0%  -3.51% (p=0.002 n=6)
GenerateUnixTimestamp-4                     77.13n ± 1%   77.72n ±  3%  +0.77% (p=0.009 n=6)
GenerateUsername-4                          351.5n ± 0%   359.0n ±  6%  +2.15% (p=0.002 n=6)
GenerateUTCTimestamp-4                      104.2n ± 1%   104.6n ±  2%       ~ (p=0.162 n=6)
GenerateUUID-4                              734.0n ± 0%   735.4n ±  0%       ~ (p=0.732 n=6)
GenerateZipcode-4                           105.2n ± 2%   105.7n ±  0%       ~ (p=0.212 n=6)
TransformCharacterScramble-4                171.8n ± 1%   175.2n ±  0%  +1.98% (p=0.002 n=6)
TransformE164PhoneNumber-4                  180.2n ± 3%   177.1n ±  0%  -1.72% (p=0.002 n=6)
TransformEmail-4                            1.798µ ± 2%   1.805µ ±  1%       ~ (p=0.177 n=6)
TransformFirstName-4                        233.9n ± 0%   235.4n ±  0%  +0.66% (p=0.002 n=6)
TransformFloat64-4                          110.8n ± 4%   110.6n ±  1%       ~ (p=1.000 n=6)
TransformFullName-4                         1.235µ ± 2%   1.244µ ±  0%       ~ (p=0.061 n=6)
TransformInt64-4                            33.74n ± 1%   33.62n ±  1%       ~ (p=0.699 n=6)
TransformInt64PhoneNumber-4                 57.84n ± 2%   60.22n ±  1%  +4.11% (p=0.002 n=6)
TransformLastName-4                         191.6n ± 0%   193.0n ±  2%  +0.70% (p=0.004 n=6)
TransformString-4                           1.022µ ± 0%   1.028µ ±  1%  +0.54% (p=0.004 n=6)
TransformStringPhoneNumber-4                211.0n ± 2%   207.1n ±  0%  -1.83% (p=0.002 n=6)
TransformUuid-4                             36.54n ± 1%   36.70n ±  1%       ~ (p=0.058 n=6)
geomean                                     194.0n        194.3n        +0.18%

                                   │ main-benchmark.txt │         branch-benchmark.txt         │
                                   │        B/op        │     B/op      vs base                │
CleanPostgresType-4                        0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                          2.312Ki ± 1%     2.294Ki ± 3%       ~ (p=0.143 n=6)
GenerateBool-4                             0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                     304.0 ± 0%       304.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                       8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                      64.00 ± 0%       64.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                             160.0 ± 0%       160.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                          24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                            385.0 ± 0%       385.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                        208.0 ± 0%       208.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                          8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                      512.0 ± 0%       512.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                         418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                           16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                            7.000 ± 0%       7.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                 8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4         40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                        31.00 ± 0%       31.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                         176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                     186.0 ± 0%       186.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                       288.0 ± 0%       288.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                              46.00 ± 0%       46.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateState-4                            24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                    240.0 ± 0%       240.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                    8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                         205.0 ± 0%       205.0 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                     24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                             80.00 ± 0%       80.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                          24.00 ± 0%       24.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4               40.00 ± 3%       39.50 ± 1%       ~ (p=1.000 n=6)
TransformE164PhoneNumber-4                 40.00 ± 0%       40.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                           489.0 ± 0%       489.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                       224.0 ± 0%       224.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                         88.00 ± 0%       88.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                        418.0 ± 0%       418.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                           8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                8.000 ± 0%       8.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                        176.0 ± 0%       176.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformString-4                          202.0 ± 0%       202.0 ± 0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4               56.00 ± 0%       56.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                            16.00 ± 0%       16.00 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               ²                 -0.05%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                   │ main-benchmark.txt │        branch-benchmark.txt        │
                                   │     allocs/op      │ allocs/op   vs base                │
CleanPostgresType-4                        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
Runner_Single-4                            24.00 ± 0%     24.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBool-4                             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                     2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                       1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCategorical-4                      2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCity-4                             2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateCountry-4                          2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateEmail-4                            8.000 ± 0%     8.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFirstName-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFloat64-4                          1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                      10.00 ± 0%     10.00 ± 0%       ~ (p=1.000 n=6) ¹
GenerateFullName-4                         6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateGender-4                           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64-4                            0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                 1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateLastName-4                         2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateRandomString-4                     3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSHA256Hash-4                       7.000 ± 0%     7.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateSSN-4                              3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateState-4                            2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                    2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                    1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUsername-4                         6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                     1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateUUID-4                             3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
GenerateZipcode-4                          2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4               2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformE164PhoneNumber-4                 3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformEmail-4                           14.00 ± 0%     14.00 ± 0%       ~ (p=1.000 n=6) ¹
TransformFirstName-4                       3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformFloat64-4                         3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformFullName-4                        6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64-4                           1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformLastName-4                        2.000 ± 0%     2.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformString-4                          4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformStringPhoneNumber-4               4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=6) ¹
TransformUuid-4                            1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

backend/pkg/dbschemas/sql/postgresql/queries/system.sql

…ue to paranoia

alishakawaguchi and others added 14 commits February 13, 2025 15:36

postgres query to get unique indexes

86ce01e

postgres manager unique indexes map

2af34c1

add orderby field to runconfigs

5b811b2

mysql

633cf61

unique indexes mssql

a074284

page query and limit

13a0682

Adds Resumable Sync (#3263)

1ed2448

add table alias to order by and where

69765da

fix lint and cli

07ca0aa

Merge branch 'main' into resumable-sync

eb0ae81

Updates Runs Page to work with Child Workflows (#3264)

10f00cd

Fixes Pagination of SQL Args and Goqu dialect (#3265)

5524181

fixes continue as new

b09d138

fix jobRunId in benthos builder

7e7d4f9

alishakawaguchi added the Improvement Created by Linear-GitHub Sync label Feb 15, 2025

alishakawaguchi self-assigned this Feb 15, 2025

Fixes Lexico Ordering with paged queries (#3267)

ad7b6c2

nickzelei self-assigned this Feb 15, 2025

bumps page size to 100k

bc80198

nickzelei reviewed Feb 16, 2025

View reviewed changes

backend/pkg/dbschemas/sql/postgresql/queries/system.sql Outdated Show resolved Hide resolved

updates postgres unique indexes sql

d48a95a

vercel bot deployed to Preview February 16, 2025 21:52 View deployment

nickzelei changed the title ~~WIP - Resumable sync~~ Adds Resumable Sync for higher stability of syncing large datasets Feb 16, 2025

adds backwards compat activity deprecated for temporary switch over d…

506c151

…ue to paranoia

nickzelei approved these changes Feb 16, 2025

View reviewed changes

nickzelei merged commit b612c8b into main Feb 16, 2025
22 of 24 checks passed

nickzelei deleted the resumable-sync branch February 16, 2025 22:27

BrewTestBot mentioned this pull request Feb 16, 2025

neosync 0.5.21 Homebrew/homebrew-core#207932

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Resumable Sync for higher stability of syncing large datasets #3266

Adds Resumable Sync for higher stability of syncing large datasets #3266

alishakawaguchi commented Feb 15, 2025

vercel bot commented Feb 15, 2025 •

edited

Loading

github-actions bot commented Feb 15, 2025 •

edited

Loading

codecov bot commented Feb 15, 2025 •

edited

Loading

github-actions bot commented Feb 15, 2025 •

edited

Loading

Adds Resumable Sync for higher stability of syncing large datasets #3266

Adds Resumable Sync for higher stability of syncing large datasets #3266

Conversation

alishakawaguchi commented Feb 15, 2025

vercel bot commented Feb 15, 2025 • edited Loading

github-actions bot commented Feb 15, 2025 • edited Loading

codecov bot commented Feb 15, 2025 • edited Loading

Codecov Report

github-actions bot commented Feb 15, 2025 • edited Loading

Benchstat Geomean Results

vercel bot commented Feb 15, 2025 •

edited

Loading

github-actions bot commented Feb 15, 2025 •

edited

Loading

codecov bot commented Feb 15, 2025 •

edited

Loading

github-actions bot commented Feb 15, 2025 •

edited

Loading