How to get code into deployment using a Dockerfile in a Dagster-dbt project deployed using helm/kubernetes

[rob-teeuwen]

I am experimenting with Dagster to schedule my dbt pipeline. I already had a dbt project. I followed the tutorial to get a scaffolded Dagster project into my dbt folder. This all worked like a charm, I can materialize my models using Dagster locally without any issues.

Now I want to deploy this to an existing Azure managed kubernetes cluster (AKS). I followed this toturial to get my initial setup, which also works fine as long as I use the example project. I can reach the webserver and see the example code.

Then I get stuck trying to get my own code into the deployment. I could use some guidance how to format the Dockerfile and the values.yaml.

Dockerfile

There doesn't seem to be much documentation on how to format the Dockerfile besides this example. I updated it by adding the dependencies that I need for my code. Next I have to change the copy statement. I don't know what build_cache/ directory is, I don't have it, and I can't find any information on it when I search, either on google, github, or slack 🤷. So I simply changed it to copy the entire Dagster folder. It builds without issues. My resulting Dockerfile is the following (my root folder is my dbt project folder and also contains the Dockerfile. The Dagster project is in ./dbt_test_dagster):

FROM python:3.11-slim

# All packages are hard-pinned to `dagster`, so setting the version on just `DAGSTER` will ensure
# compatible versions.
RUN pip install \
    dagster==1.5.10 \
    dagster-postgres \
    dagster-aws \
    dagster-k8s \
    dagster-celery[flower,redis,kubernetes] \
    dagster-celery-k8s \
    dbt-duckdb \
    duckdb \
    dagster-dbt 

# Get example pipelines
COPY ./dbt_test_dagster/ /

Values.yml

The example in the tutorial points to a "python-file" called "repo.py", but I don't have a repo.py. I have the dbt scaffolded project, which works fine as it is, so I don't really want to make a repo if I don't need to, I'd rather keep it simple. So I tried referring to my dagster directory as a package or as a module, neither of which work. My override-values.yml file looks like this:

dagster-user-deployments:
  enabled: true
  deployments:
    - name: "<some-random-name>"
      image:
        repository: "<link to container registry>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--package-name"
        - "/dbt_test_dagster" # I also tried "/dbt_test_dagster/dbt_test_dagster" here, referring to the folder of python files inside the Dagster project
      port: 3030

Obviously, this doesn't work, but I don't know why. Upgrading helm with these overrides creates a pod dagster-dagster-user-deployments-blablabla which enters a CrashLoopBackOff cycle. When inspecting with kubectl describe pod it shows "successfully pulled image " after which it says "Back-off restarting failed container". I feel like both my Dockerfile and my values.yml are incorrect.

[RushilDaya]

Hi Rob, I faced the same issue as you, after some experimenting i was able to get it to run.
I feel that the build_cache line in the dockerfile is an error

i did the same thing you did for the docker file. an error i had was that i was building on a macbook which wasn't compatible with my kubernetes cluster (GKE in my case) so i needed to build like this
docker build -t TAG . --platform=linux/amd64

I needed to change my values.yml from what you have though

deployments:
    - name: "my-example"
      image:
        repository: "my-container-registry-link"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "definitions.py"
      port: 3030

I used the --python-file argument. and pointed it to a file in the root of my docker working directory. inside that file i included a Definitions object

from dagster import Definitions

defs = Definitions(
    assets=[rushil_asset],
)

hope something here helps you

[un1xman]

@RushilDaya you are my new hero. Thanks mate. It was very valueable comment