Getting dagster-dev online quickly & speeding up development cycles

[chollinger93]

Note: This was originally a PR I put together to hook directly into dagster dev using the ngrok SDK, but this will quickly show how to achieve a similar result with 2 terminal sessions.

I figured this was worth sharing in some form, since it has become such a core path of my workflow. Suggestion to share it here specifically is courtesy of @erinkcochran87!

Transparency disclaimer: I do work for ngrok, but I'm writing this as a regular software engineer who happens to be a big Dagster fan and finds that the 2 tools work wonderfully together. The fact that I run our data platform does, however, help a bit to marry them even more, since I made Dagster central to our own data stack. :-)

Use Case: Exposing dagster dev

If you're working on Dagster, you might find yourself wanting to expose dagster dev to others (or yourself on a different machine), preferably quickly and painlessly, since nobody likes slow iteration cycles.

I frequently have this use case, since we develop on headless Linux VMs and use our Macs mostly as terminals.

I know other folks might have the pleasure to deal with more complex corporate networks, ephemeral VMs, dev containers etc., where just sharing a web-based application isn't trivial.

This is also very useful to share things like an asset graph with folks who might not have the toolset to run a version of your dagster codebase themselves, but would still like to peruse. This is super useful when paired with dbt, since it visualizes the SQL dependency graph via the Dagster UI and makes it easier to grok for folks. Also works well for small dev servers!

Setup

This is where ngrok comes into play. ngrok allows you to expose any application to the internet, both for production and dev use cases. You can create an account and use ngrok for free.

Once you have an account and are logged in, you can grab the requisite client application from your dashboard:

You'll need an auth token, which you can find here.

For instance, for an x86 Linux:

sudo tar -xvzf ~/Downloads/https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz -C /usr/local/bin
ngrok config add-authtoken $YOUR_AUTH_TOKEN

The same also works with Windows:

choco install ngrok
ngrok config add-authtoken $YOUR_AUTH_TOKEN

Or nix:

export NIXPKGS_ALLOW_UNFREE=1
nix-shell -p ngrok

Or MacOS:

brew install ngrok/ngrok/ngrok
ngrok config add-authtoken $YOUR_AUTH_TOKEN

You get the idea. :-)

Exposing the service

This works best with two terminal sessions. VS Code has this neat feature for it:

In one terminal, run dagster-dev as you normally would. We use poetry for dependency and venv management.

In the other terminal, simply run ngrok http 3000:

In the resulting ngrok session, simply click the link and you should be able to access your local dagster dev instance from anywhere with an internet connection:

The neat thing, this doesn't interfere with live reloads or any other local development you might do. Speaking from experience, having to fully deploy a service, even to dev, to test a simple thing can be frustrating to say the least.

Also, it really does allow access from anywhere:

More Options

Other options that are very useful for at least my workflow:

You can create a custom domain and link that to your deployment:

ngrok http --domain=my-dagster-domain 3000

You can limit access via OAuth:

ngrok http 3000 --oauth-allow-email [email protected] --oauth google

And, recently, there is a neat traffic inspector (think "Wireshark but not terrifying") technical preview to debug more complex network magic:

There's a bunch more features, but this is what I personally use most frequently, specifically for Dagster.

Production

If you self host dagster, ngrok also offers a Kubernetes ingress controller, which allows you to use ngrok for authentication in a productionized K8s deployment: You can find docs here.

Shameless plug: That is exactly what we do to serve our dagster production instances.