How to add private Python packages to a Dagster cloud deployment?

[shalabhc]

If I have a private GitHub repository, eg. my-org/private-repo and I want to include a Python package from that repository in our dagster cloud project, how can I do that?

[shalabhc]

If you use fast deploys in your workflow, the following steps can install a private package as a dependency to your dagster cloud project. If you have ENABLE_FAST_DEPLOYS: "true" in your .github/workflows/deploy.yml then you use fast deploys.

1. In your deploy.yml file, add the following to the top of steps: section in the dagster-cloud-default-deploy job.

     - name: Checkout internal repository
       uses: actions/checkout@v3
       with:
         token: ${{ secrets.GH_PAT }}
         repository: my-org/private-repo
         path: deps/private-repo
         ref: some-branch   # optional to check out a specific branch
     
     - name: Build a wheel
       # adjust the `cd` command to cd into the directory with setup.py
       run: >
         cd deps/private-repo &&
         python setup.py bdist_wheel &&
         mkdir -p $GITHUB_WORKSPACE/deps &&
         cp dist/*whl $GITHUB_WORKSPACE/deps
                   
   
     # If you have multiple private packages, the above two steps should be repeated for each but the following step is only
     # needed once
     - name: Configure dependency resolution to use the wheel built above
       run: >
         echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
         echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
         echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
         echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV

2. Create a github personal access token and set it as the GH_PAT secret for your actions.
3. In your dagster project's setup.py add your package name to the install_requires section:

       install_requires=[
         "dagster",
         "dagster-cloud",
         "private-package",   # add this line - must match your private Python package name

Once the deploy.yml is updated, any code deploy should checkout your private repository, build the package and install it as a dependency in your dagster cloud project. Repeat the above steps for branch_deployments.yml.

[CharlesFr]

@johannkm thanks for this guide, I've implemented it and it works.

But FYI, this line mkdir $GITHUB_WORKSPACE/deps && initially caused the action to fail because the file/folder already existed.

I need to install two dependencies, should I add all three steps for each dependency - or just the first two and the last third is shared?

PS; when you have some time I would greatly appreciate your answer here: #17823

[shalabhc]

But FYI, this line mkdir $GITHUB_WORKSPACE/deps && initially caused the action to fail because the file/folder already existed.

Thanks for the report. We have changed it to mkdir -p which should not fail if the folder already exists. However I am surprised if the deps folder exists on a fresh Github action runner.

I need to install two dependencies, should I add all three steps for each dependency - or just the first two and the last third is shared?

It would be just the first two steps repeated for each wheel and the last step is only needed once.

[pcadmanbosse]

Is there an equivalent for "non fast" deploys?

[CharlesFr]

@shalabhc are the python wheels built by Github Action cached or saved anywhere?

I have an issue where I can see from Github Actions logs that the latest branch of private module is correctly pulled. But the "Python Executable Deploy" step is failing due to DagsterImportError which I believe is because an old wheel is used?

Is is possible to check the logs where packages of the pipeline are installed?

[tyler-hanson]

I'm getting the same issue @CharlesFr , were you ever able to fix it?

[neerajprad]

Thanks for the instructions here, these were super helpful. I am not sure if this is related to the issue above. My build was successful but I later got an ImportError in Waiting for agent to sync changes to {project}.... Any suggestions on how to debug this further?
Note that the ImportError is on the library that was built and added as a dependency.

Deploying to a full deployment: prod
Running on Ubuntu 20.04
Not in a pull request, will not post PR comment
Running ['generated/gha/dagster-cloud.pex', '-m', 'dagster_cloud_cli.entrypoint', 'serverless', 'deploy-python-executable', '/home/runner/work/sailfin-dag/sailfin-dag/project-repo', '--python-version=3.10', '--deps-cache-to=sailfin-ai/sailfin-dag/main', '--deps-cache-from=sailfin-ai/sailfin-dag/main', '--build-method=local', '--location-name=*', '--location-file=/home/runner/work/sailfin-dag/sailfin-dag/project-repo/dagster_cloud.yaml', '--git-url=https://github.com/sailfin-ai/sailfin-dag/tree/1f59bff2ef2bdd397f5099ef991c3843fedb0f14', '--commit-hash=1f59bff2ef2bdd397f5099ef991c3843fedb0f14', '--url=http://sailfin.dagster.cloud/prod', '--location-load-timeout=3600', '--agent-heartbeat-timeout=90']
Going to deploy location sailfin
Building Python executable for sailfin from directory /home/runner/work/sailfin-dag/sailfin-dag/project-repo/. and Python 3.10.
Built source pex: /tmp/tmpj5msossr/source-tmp-34feed14-fe10-4f9f-bd4b-cfe3fe142e2e.pex
Will deploy sailfin on image: 657821118200.dkr.ecr.us-west-2.amazonaws.com/dagster-cloud-serverless-base-py3.10:1.7.5
Built /tmp/tmpj5msossr/source-d74197da1d7ea12cbd14e134b90d377922267a83.pex (827 KB) from:
 - /home/runner/work/sailfin-dag/sailfin-dag/project-repo/.
Reusing cached dependencies: deps-7bb0e82ec5c54d6bd85d64333f82460651eacbbc.pex
Uploading Python executable for sailfin.
View the status of your locations at http://sailfin.dagster.cloud/prod/workspace.

Waiting for agent to sync changes to sailfin...
Still waiting for agent to sync changes to sailfin. This can take a few minutes.
Error: Some locations failed to load after being synced by the agent:
Error loading sailfin:
dagster._core.errors.DagsterImportError: Encountered ImportError: `No module named 'fingraph'` while importing module sailfin_dag.definitions. Local modules were resolved using the working directory `/venvs/a0708530506f/lib/python3.10/site-packages/working_directory/root`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets. 

[jboreiko]

I ran into this issue as well. If you examine the log lines above you can see Resuing cached dependencies: deps-<hash>.pex. This lead me to force_rebuild_deps in the pex deploy action (https://github.com/dagster-io/dagster-cloud-action/blob/main/actions/build_deploy_python_executable/action.yml#L23-L26) that allows you to override this caching. If you enable the force build your newly created wheel will once again be installed vs. the old version reused in the cache.

      - name: Python Executable Deploy
        if: steps.prerun.outputs.result == 'pex-deploy'
        uses: dagster-io/dagster-cloud-action/actions/[email protected]
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/$DAGSTER_CLOUD_FILE"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "${{ env.PYTHON_VERSION }}"
          force_rebuild_deps: 'true'
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}