Companies / Projects using Dagster

[yuhan]

If you are using Dagster at your company/organization, you can share it here in this discussion!

All forms are welcomed (e.g. blog posts, video demos, or talks), and also feel free to leave a brief introduction to your use cases or tell your story.

Some companies/organizations:

• GoodEggs: Good Data at Good Eggs (blog series)
• Drizly: Building shared spaces for data teams at Drizly (blog post)
• Mapbox: Incrementally Adopting Dagster at Mapbox (blog post)
• LINE (Thailand): Build Data Application with Dagster (talk)
• Doordash: Increasing Operational Efficiency with Scalable Forecasting (blog post)
• Prezi: Migrating Data Pipelines to Dagster (talk)
• Geomagical Labs (part of the IKEA family): Building Customer-Facing Applications with Dagster (talk)
• MIT Open Learning: Running Dagster in Production (talk)
• Bestplace: Building a Modern Data Analytics Platform with Dagster (talk)
• VMware: Monitoring Driven Development with Dagster (talk)

Explore more awesome projects we featured in our community meetings in this Community Member Showcase.

---

If you are interested in sharing more details about how your organization uses Dagster, please don’t hesitate to reach out to us at [email protected]. We’d love to help you host an event, write a blog post, or support you in any way we can.

If you are looking for people with Dagster experience, check out #7214.

[alex-service-ml]

Containerized Dagster Development

Hey folks! I recently open-sourced my Dagster project boilerplate with support, available here! As a companion piece, I also wrote this technical blog post explaining how it works.

Notable Features That Make Your Life Easier

• Unified Dockerfile for development & deployment; easily integrates with CI/CD processes
• Picks up code changes immediately (just hit Reload in dagit; don’t have to restart the container!)
• Tractable package management using poetry. No more hideous pip freeze > requirements.txt
• Packages the source code according to PEP517 & PEP518

Feedback

I would love any feedback, especially around ways to make it more useful in the real world. I can be reached via issues on my repo or in the Dagster Slack

Credits

I want to thank my employer, Mile Two, for supporting the open source movement, and of course Elementl for such a wonderful tool

[bakerwho]

The simplest Deployable Dagster pipeline (in 120 lines of Python)

While learning Dagster, I found myself hunting for a start-to-finish working example - a Minimum Viable Pipeline. I couldn’t find one, so I built it myself.

I wrote it up in a technical blog post here, or you can directly clone and run the Github repo.

What is a Minimal Viable Pipeline

This simple example just processes a string (select the input, change case, strip punctuation). I hope this can be an abstraction for all kinds of complex Data/MLOps.

The repo is just 120 lines of code, and uses only the Dagster concepts you need (there are loads more). The post explains what each of those constructs does.

It should take <10 minutes to get your first job running.

Questions?

HMU on Twitter @aabir_ak with any questions or requests!

Credits

I appreciate my company and team who support the strategy of spending time hacking to learn valuable tools. And thanks Dagster team and community for the encouragement and good vibes!

[cyy7645]

I think all repositories under dagster/examples are work as one of Minimal Viable Pipeline.
Just post it here in case some one wanna see more examples.

[KevinHaynesLyn]

Deploying Dagster on ECS using GitHub Actions

In the course of setting up my first ECS deployment of Dagster, I found that the Docker-Compose-based examples didn't work well with the way my company had set up our infrastructure. I was able to get it working through a lot of trial and error. In order to help those who may be attempting a similar deployment, I put together an article with simple code examples.

The full article can be found here

Feedback

Any and all feedback and suggestions are welcome. I would especially love to hear...

• Is there anything we did that could be done better?
• What topics would be interesting to read about in a future article?

You can comment on the article, reach me in the Dagster Slack workspace, or tweet @data_pinguin

Credits

A big, big thank you to Mollie and the rest of the team at Elementl for the feedback on the press release and article. I'd also like to thank Rebecca Hadi, Amy Shetler, Abby Seliga, and Jon Linczak at Lyn Health who helped bring this article to life.

[sspaeti]

Building a Data Engineering Project in 20 Minutes

On my blog called sspaeti.com, I implemented a side-project Building a Data Engineering Project in 20 Minutes where I scraped a real estate website to get insight for buying the next flat here in Switzerland.

Abstract

It's a full-blown data engineering project that focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

The goal is to touch on the shared data engineering challenges and use promising new technologies, tools or frameworks, most of which I wrote about in Business Intelligence meets Data Engineering with Emerging Technologies. As well, everything runs on Kubernetes in a scalable fashion and locally with Kubernetes on Docker Desktop.

💡 What Will You Learn?

Below I noted the key learnings which are integrated into a full-fledged data engineering project to illustrate the how part in the most hands-on way. Hopefully, you'll find something interesting for you!

• Scraping with Beautiful Soup: How to get value out from a website with basic Python skills.
• Change Data Capture (CDC) with Scraping: Using a fingerprint to verify against the data lake if a property needs to be downloaded or not.
• How to use an S3-Gateway / Object Storage: Placing an S3 API in front of your object storage in the so-called “gateway-mode” to stay cloud-agnostic. This allows you to change the object-store from Amazon S3 to Azure blob storage or Google cloud storage with ease.
• UPSERTs and ACID Transactions: Besides schema evolution mentioned above, Delta Lake also provides merge, update and delete directly on your distributed files.
• Automatic Schema Evolution: With the growing popularity of data lakes and ELT, data engineers are left with lots of data but no schemas. To integrate schema and especially schema changes, automatic schema evolution is important.
• Integrating Jupyter Notebooks - the right way: Notebooks hold important data transformations, calculations or machine learning models yet it's always hard to copy the living code in your data pipelines. We will integrate notebooks as a step of our pipeline with Dagster.
• Learning about Apache Druid: Druid is one of the fastest Data Warehouse / OLAP Solutions. It's optimized for fast real-time ingestion and immutable data. On the downside, it's hard to set up, luckily, below you see how to do exactly that.
• Open-Source dashboarding with Apache Superset: How to use Superset with its many out-of-box connections. On top, it's free of charge compared to Tableau, Looker and others.
• DevOps with Kubernetes: How to run Kubernetes locally and install all of the tools here. If you haven't used Kubernetes, don't worry, examples and local set-up with Kubernetes for Docker are included.
• Introduction to features of Dagster: Showing how all of the data engineering parts can be tided together with one open-sourced tool called Dagster (alternative to Airflow).
• And many more which I won't mention but you'll hopefully see along the way.

Source Code

The source code you can find on practical-data-engineering for the data pipeline or in data-engineering-devops with all it's details to set things up.

Others

I hope that helps others who want to implement their side-project with dagster.

[stkbailey]

We have started to use Dagster at Whatnot, and it's been a great experience. We wrote a little about it in this eng blog..

[pietercusters]

The Why and How of Dagster User Code Deployment Automation

Read here how we at Vandebron fully automated the deployment of new user code repositories.

TL;DR

If you want to deploy new Dagster user code respositories, you need to modify and redeploy the whole Dagster system (while they are presented as separate in the docs). This is undesirable for many reasons, most notably because it slows down a migration or the regular development process. This post presents a way to avoid this and build a fully automated CI/CD-pipeline for (new) user code.

[bendnorman]

We're liberating public energy data to support climate advocates and researchers with Dagster in the Public Utility Data Liberation project (PUDL)! You can read more here.

[tacastillo]

Hi folks! We'll be closing this discussion in favor of having y'all share in more permanent and sticky places.

If you use Dagster and want to share how cool your work is, feel free to reach out to me on Slack and we could make a case study about your team's data platform.

[cmpadden]

We've now introduced the dagster-io/awesome-dagster repository, where community members can share their work and other helpful resources. Feel free to make contributions over there if you feel you have something to include. Thanks!