Dagster Cloud Serverless and dbt deps

[slopp]

If you are deploying to Dagster Cloud Serverless and are using dbt core with dbt dependencies, you will likely see this message:

DagsterDbtCliFatalRuntimeError ...Compilation Error dbt found package(s) specified in packages.yml, but only ... Run "dbt deps" to install packages

To install dbt dependencies into a Dagster Cloud Serverless application follow these steps:

1. If you are using Dagster's GitHub action workflow, start by modifying the .github/actions/deploy.yaml file to disable fast deploys: ENABLE_FAST_DEPLOYS: 'false'. If you are using the CLI to deploy, use dagster-cloud deploy-docker instead of dagster-cloud deploy-python-executable

2. Create a file called dagster_cloud_post_install.sh in the root of your Dagster project. This file should include the dpt deps command, e.g.:

dbt deps --project-dir ./dbt_project --profiles-dir ./dbt_project/config 

3. Make this file executable, by running chmod +x dagster_cloud_post_install.sh.

4. Add the file to your Git repository

FAQ

• I followed these steps but am still seeing the same error?

Errors in the dbt deps command will show up in the GitHub action logs, not in Dagster Cloud. While the deploy action may have succeeded, Dagster could not load the code. Review the logs in the GitHub action.

• I am seeing the error: exit code 126

This means the file dagster_cloud_post_install.sh is not executable. Re-attempt steps 3 and 4. You must re-commit the file after running chmod +x. Unfortunately some Git clients do not recognize this change, but you can use the git commands manually -- or add a comment to the file to force the client to recognize the diff.

• I am still seeing dbt-related errors and my project won't load in Dagster Cloud

Often if you have env_var references within profiles.yaml those references have to be resolved when the code is loaded. Try adding environment variables in Dagster Cloud (Deployment > Environment Variables) and then Redeploying (Deployment > Redeploy).

[shalabhc]

Using dbt deps with fast deploys

If you are using fast deploys (ENABLE_FAST_DEPLOYS: 'true), add the following steps to install dbt-core and run dbt deps to your deploy.yml file, right after the fast deploys checkout step:

      - name: Checkout
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
          path: project-repo
          
      # Add these two lines:
+     - run: "pip install dbt-core"
+     - run: "dbt deps --project-dir $GITHUB_WORKSPACE/project-repo/dbt_project"

This will install the deps in your dbt_project. The entire project directory is packaged and deployed as part of fast deploys, making these available during runtime.

[patrikdevlin]

For anyone who is using dbt local packages the symlink won't carry over into the pex image. As a workaround you configure setting up the deps like this where you define the DbtCliResource:

# If DAGSTER_DBT_PARSE_PROJECT_ON_LOAD is set, a manifest will be created at run time.
# Otherwise, we expect a manifest to be present in the project's target directory.
if os.getenv("DAGSTER_DBT_PARSE_PROJECT_ON_LOAD"):
    ...
else:
    dbt_resource.cli(["deps"]).wait()
    dbt_manifest_path = dbt_project_dir.joinpath("target", "manifest.json")

[paloderosa]

I think the current solution should be using recent pipeline definitions for gitlab and github.

These are not however the definitions assigned when setting up a new code location in Dagster Cloud. It would be great if there existed a code location setup flow in which we could setup a project with dbt and dbt packages.

[mjkanji]

+1. For the Docker deploy option, changing my deploy file to also check out a separate DBT repo does not work, as the DBT repo is not copied over to the Docker container. Please see the code below.

  # Skipped the PEX bits as they're not relevant here

  dagster_cloud_docker_deploy:
    name: Docker Deploy
    runs-on: ubuntu-20.04
    if: needs.dagster_cloud_default_deploy.outputs.build_info
    needs: dagster_cloud_default_deploy
    strategy:
      fail-fast: false
      matrix:
        location: ${{ fromJSON(needs.dagster_cloud_default_deploy.outputs.build_info) }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Checkout DBT repo
        uses: actions/checkout@v3
        with:
          repository: owner/dbt-repo
          ref: main
          path: dbt
          token: ${{ secrets.DBT_REPO_PAT }}

      - name: Print contents
        run: "ls -lah ./"

      - name: Build and deploy to Dagster Cloud serverless
        uses: dagster-io/dagster-cloud-action/actions/[email protected]
        with:
          dagster_cloud_api_token: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
          location: ${{ toJson(matrix.location) }}
          base_image: "python:${{ env.PYTHON_VERSION }}-slim"
          # Uncomment to pass through Github Action secrets as a JSON string of key-value pairs
          # env_vars: ${{ toJson(secrets) }}
          organization_id: ${{ secrets.ORGANIZATION_ID }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The GH runner does have the DBT repo (as shown by the Print Contents step). See below:

However, the dbt deps step in the post install script doesn't work, because the DBT repo is not found in the container. I've double checked this by setting the post intall script to show the current contents vi ls -lah and you can see the DBT repo is now missing:

[slopp]

@mjkanji your issue should be discussed / resolved here: #13767 (reply in thread)

[slopp]

@paloderosa the links you reference are specific to setting up a dbt project through the Dagster Cloud web interface:

For cases where you need to customize things beyond the UI-driven setup we still recommend starting from the GitHub actions described in this discussion.

[MinchoG]

Sorry but this thread is not really helping for serverless gitlab as per recommendations here:
https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/dbt/serverless-ci-dbt.yml

can someone tell all serverless gitlab users what actions they need to take to have dbt deps installed properly
thanks

latest edit: I have found this https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/dbt/serverless-ci-dbt.yml
however gitlab runner doesn't have apt-get

[garethbrickman]

@MinchoG The serverless-ci-dbt.yml template for GitLab is outdated and it is on our radar to update