Skip to content

A mix of proxy and an HTTP database server that replicates/mimics the Turso server endpoint

Notifications You must be signed in to change notification settings

digitalmio/turso-edge-pop

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

23 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Turso Edge POP

Turso Edge Point of Presence – a mix of proxy and an HTTP database server that replicates/mimics the Turso server endpoint.

This project started as a Bun & Hono rewrite of the Turso CDN project and grew "a bit" from there.

Please note that this project is in its early stages. If you have found any bugs or have any suggestions, please open an issue or contact me directly on X or Bluesky.

Motivation

Turso disabled Edge Replicas functionality and for new databases, you will not be able to replicate your data globally: https://turso.tech/blog/upcoming-changes-to-the-turso-platform-and-roadmap#simplifying-our-replication-model I built this project to be able to simulate the original functionality whilst having more control over the data and sync process.

Why are you using Bun not XYZ?

As a JS/TypeScript developer for this project, I was choosing between Bun and Deno as they can easily compile to a single executable file and are super fast. Both runtimes are great, but I chose Bun as I have more experience with it.

How does it work?

Turso Edge POP is an HTTP server that replicates Turso database endpoint behaviour. It runs by default on port 3000.

On start, it creates a local database and syncs it with the origin Turso database. It stays up to date through interval sync (by default every 60 seconds) and can optionally listen for changes via Redis pub/sub to update the database as needed on top of the interval sync.

POP provides a handful of endpoints, same as Turso, but the most important ones are POST /v2/pipeline, POST /v3/pipeline and POST /.

Based on the request type, it will either proxy the request to the origin Turso database or query the local database using the official Turso LibSQL client. The SDK internally then decides if the query will be from the local or origin database.

The / endpoint statements are all executed against the local database via the LibSQL client.

The /v2 and /v3 endpoints are more complex. Turso Edge POP will run all execute and close statements locally. All other (mainly batch) statements are proxied to the origin database and returned without any changes back to the client.

The app also exposes a /health and /version informational endpoints.

Turso Edge POP supports HTTP requests only (no websocket support etc), hence you need to use http/https url when connecting to POP.

When you point your client to the POP, it will either respond with local data or seamlessly proxy your request to the origin server. This process is automatic and does not require any special setup from the client. On successful origin response, POP will also process a sync update to fetch a fresh copy if any writes were made.

How to deploy? Where to host?

You will need to deploy POPs on your own infrastructure. The app requires some form of persistent storage to store the local database. The simplest option will be Fly.io - they offer an Anycast network so you do not need to worry about routing.

From my experience - any form of shared storage is not reliable enough for this, or any other database, use case. You will need a Block Storage solution or you can simply host it on the server disk space.

If you would like to host POPs on VPS, then you will need to manage GEO routing (so that the client can connect to the closest POP) yourself. This can be achieved by using DNS providers that offer GEO routing. I can recommend GCore, ClouDNS or Bunny.

Full documentation is coming soon.

Keeping the database in sync

If you are interested in this project, you probably would like to have multiple copies of your database in different regions of the world in sync.

The simplest way that works out of the box is interval sync. You can define via the env variable TURSO_SYNC_INTERVAL how often the local database will be synced with the origin database. If you do not set it up, it will be syncing every minute (60 seconds).

If this works for you, you can stop reading here.

If you would like to get updates on the database as they happen, you can use Redis pub/sub. This requires you to have a Redis server running and configured. The simplest way would most likely be to use Upstash free tier.

You can set up Redis connection details via the env variable REDIS_CONNECTION_STRING. The app will subscribe to the channel defined by REDIS_SYNC_CHANNEL. By default, it is set to sync. As long as the query that makes changes to the database is executed via the app, and client or proxy response indicates the change via rowsAffected, the app will publish a sync message to the channel. All servers that are subscribed to the channel will receive the message and will sync the database.

Running the app from binary file (simplest, easiest way)

Download the binary file from the releases page for your platform. For MacOS (Darwin) and Linux you have option for ARM64 and x64, for Windows you have x64 only.

To run the app, you will need to provide the same environment variables as when running from source code (check below in section "Development and running from source code"). Bun reads .env file, but this solution is advisable for development only. To run the binary file run the following command (example for MacOS ARM64).

./tursoEdgePop-darwin-arm64

It is possible that you'll need to change the permissions of the file to make it executable (example for MacOS ARM64).

chmod +x tursoEdgePop-darwin-arm64

Development and running from source code

You will need to have Bun installed on your machine. This will not be required in the future. The binary file will be self-contained and will be able to run without any additional dependencies.

The app does require connection and configuration details to run. You will need to provide them via environment variables. A full list of options as Zod schema is available in the env.ts file.

The main required options are:

  • TURSO_DATABASE_URL - URL of the origin Turso database.
  • TURSO_AUTH_TOKEN - Auth token for the origin Turso database.
  • PROXY_AUTH_TOKEN - Auth token for the Turso Edge POP server.

Most likely, you will also need to provide a local database file path. By default, it is set to /app/data/local.db to fit Fly.io deployment. To amend this, please use the DB_FILEPATH environment variable.

If you prefer, you can store environment variables in a .env file in the root of the project / same folder as binary, but this is NOT recommended for production use.

To install dependencies:

bun install

To run app:

bun dev

This will start the app on port 3000.

Building own binary file

Please follow the instructions from https://bun.sh/docs/bundler/executables to build a binary file.

Based on my use of MacOS and wanting to get a Linux binary, I would need to run the following commands from the root folder of the project:

bun build --compile --minify --sourcemap --target=bun-linux-x64-modern ./src/index.ts --outfile turso-edge-pop

This will create a binary file in the root folder of the project called turso-edge-pop. You will need to change the chmod of the file to make it executable.

chmod +x turso-edge-pop

and then you can run it:

./turso-edge-pop

Note: This is NOT an official Turso project and is not associated with Turso in any way. Turso Edge POP is an independent project that requires a Turso/LibSQL server to proxy and sync database requests.

About

A mix of proxy and an HTTP database server that replicates/mimics the Turso server endpoint

Resources

Stars

Watchers

Forks