How do I run steps that run arbitrary containers

[dagsir[bot]]

Hi team,

We're trying to setup a job that runs 2 ops, where each op is defined via k8s_job_op and runs on its own image in Kubernetes. The images are arbitrary and know nothing about Dagster.

I'm getting an error in the step-pod that is spawned, that makes it seem like its required for the op images to contain all of the Dagster definitions (Sorry for some typos, I had to type this up manually):

_dagster.core.errors.DagsterImportError: Encountered ImportError: No module named dummy_dagster_module while importing module dummy_dagster_module. Local modules were resolved using the working directory '/app'. If another working directory should be used, please explicitly specify the appropriate path using the -d or --working-directory for CLI based targets or the working_directory configuration option for workspace targets

My questions are:

1. Is it possible to run a command in an arbitrary image that doesn't hold any of the Dagster definitions/ops/jobs/assets code? Option A in <https://github.com/dagster-io/dagster/discussions/14387|this post> suggests it is possible but doesn't have any examples
2. Could me defining module_name="dummy_dagster_module" in the pyproject.toml be causing the error? Is there something else I should be using?

---

The question was originally asked in Dagster Slack.

[dmsfabiano]

I guess the broader question here may be: how can I run a Kuberneras job, whose code is defined by dagster on an specialized image that doesn't know about dagster?

Is there a way to do this through code locations / grpc servers? Or would the issue be that the dagster code gets tangled with the specialized image?

[prha]

It should be possible to launch k8s jobs as ops using images that don't have any knowledge of Dagster.

Can you share a snippet of what your ops look like?

Here's a unittest showing the use of k8s_job_op to launch k8s jobs using images with no concept of dagster:

dagster/integration_tests/test_suites/k8s-test-suite/tests/test_k8s_job_op.py

Line 30 in cf185d5

def test_k8s_job_op(namespace, cluster_provider):

[dmsfabiano]

@prha Thank you for going through this!

k8s_job_op does work to run arbitrary commands in an image that knows nothing about dagster. But then you lose some of the Dagster capabilities and automatic input/output handling between ops, right?

What the error in my original question is actually from and what we are really trying to figure out is if we can do something like the below snippet and have each op run in its own image that knows nothing Dagster. The problem is it seems that when the dagster-step pod spins up for the 1st op, it tries to search for the module_name directory that we defined in the pyproject.toml (where the dagster code is).

@op(
   tags={
       "dagster-k8s/config": {
           "container_config": {
               "resources": {
                   "requests": {"cpu": "100m", "memory": "500Mi"},
                   "limits": {"cpu": "200m", "memory": "1Gi"},
               },
               "image": os.environ.get("DAGSTER_OP_1_IMAGE"),
           },
       }
   }
)
def op_1(context):
   """placeholder"""
   return 1

@op(
   tags={
       "dagster-k8s/config": {
           "container_config": {
               "resources": {
                   "requests": {"cpu": "200m", "memory": "500Mi"},
                   "limits": {"cpu": "300m", "memory": "1Gi"},
               },
               "image": os.environ.get("DAGSTER_OP_2_IMAGE"),
           },
       }
   }
)
def op_2(context, number: int):
   """placeholder"""
   return number + 1

@graph
def graph_1():
   """placeholder"""
   op_2(op_1())

job_1 = graph_1.to_job(name="job_1", executor_def=k8s_job_executor)

Essentially, we are trying to emulate an ecosystem where we have highly specialized ML images, that do not know anything about dagster. However, want to potentially define the job behavior within dagster as shown above. Is there a way to pass that on to the op?

One thing we've considered is to include the Dagster module code in a mounted volume so that the images that know nothing about Dagster will still have the Dagster module code. We were hoping however, there was a better alternative, perhaps through grpc server/code loc?

[prha]

Ah I see. In this case, you might want to wrap normal ops around invocations of execute_k8s_job.

That way, you can use the dagster machinery (e.g. io managers, etc), but still invoke the meat of the execution in your specialized images that have no concept of dagster.

Similar discussion: #18050