Compression for sync streams

[rkistner]

Background on compression options we're investigating in PowerSync: #330

This enables gzip and zstd as transport compression options for sync streams. Since data sync is often performed on metered mobile connections or slow WiFi connections, this could reduce data costs and increase performance for those cases.

This uses the standard Accept-Encoding request header, and Content-Encoding response header. As an example, Chrome would send accept-encoding: gzip, deflate, br, zstd, then we'd response with content-encoding: zstd.

zstd is the best option, and is supported by default in Chrome 123+ and Firefox 126+. gzip is provided as a fallback option, and is supported in Safari and many native clients.

Since browsers send the accept-encoding header by default, this will have an immediate effect for the Web SDK. For other SDKs we'd have to test individually, and see whether we need more client-side changes to enable compression.

Performance

These are not scientific benchmarks, but just gives an initial idea of expected performance. Since decompression is performed by platform-provided native code in each SDK, this adds minimal overhead and minimal client-side changes (we're not manually decompressing using JavaScript, for example).

Synthetic test dataset: 1.6 million rows of data, 384MiB uncompressed download size, 117MiB data size (the rest is metadata).

zstd compression gives 72.4MiB download size (19% of the original size); gzip gives 76.2MiB (20% of the original size).

In all cases (uncompressed, zstd, gzip), it took around 13s to download the data with curl, and 52s with the diagnostics app. So on a desktop machine, the primary difference is in bandwidth usage, rather than performance.

When downloading the data with curl, the nodejs process averaged around 90% CPU usage for uncompressed data, and 120% when using zstd or gzip. Note that it uses more than 1 core since compression uses a separate thread.

Testing clients

OkHttp supports gzip by default - likely affects both React-Native and Kotlin.

Chrome:

• Works with zstd and gzip out-of-the-box: accept-encoding: gzip, deflate, br, zstd

NodeJS SDK:

• Uses gzip out of the box: accept-encoding: gzip, deflate
• We can make minor SDK modifications to use zstd on Node versions that support it

Dart/Flutter:

• Uses gzip out of the box: accept-encoding: gzip

React-Native on Android:

• Uses gzip out of the box: accept-encoding: gzip

Kotlin on Android:

• Uses gzip out of the box: accept-encoding: gzip

TODO

• Test for issues and performance on each SDK
• Proper benchmarks According to my initial tests, performance is on par to uncompressed speeds when bandwidth is not a limitation. Will test constrained bandwidth environments later.
• Test permessage-deflate for websockets permessage-deflate compression for websockets #337

[changeset-bot]

🦋 Changeset detected

Latest commit: 2c3f94c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@powersync/service-core	Minor
@powersync/service-types	Minor
@powersync/service-image	Minor
@powersync/service-core-tests	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
test-client	Patch
@powersync/service-schema	Minor
@powersync/lib-service-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[simolus3]

Looks good to me 👍

I also wonder if we want to consider supporting brotli to give Safari something better than gzip, but that may be to small to matter.

[simolus3]

minor nit, feel free to ignore: Since this is the only part that is meaningfully different between zstd and gzip, can we move it into a separate function? Then maybeCompressResponseStream could take the stream.Transform instance for the compressor and the common tracker.setCompressed(), transform and encodingHeaders lines can be shared.