Unable to run docker op in Dagster Cloud

[NRaf]

As part of my workflow I need to run a Docker container that executes some NodeJS code. As a first step, I'm trying to get Docker up and running in a job.

The following code works fine locally:

docker_hello = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["/bin/echo", "HELLO"],
    },
    name="first_op",
)
docker_goodbye = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["/bin/echo", "GOODBYE"],
    },
    name="second_op",
)


@job
def docker_test():
    docker_goodbye(docker_hello())

However, when I try to run this on Docker Cloud, I get the following error:

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "first_op":
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_plan.py", line 245, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_step.py", line 499, in core_dagster_event_sequence_for_step
    for user_event in _step_output_error_checked_user_event_sequence(
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_step.py", line 183, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_step.py", line 87, in _process_asset_results_to_events
    for user_event in user_event_sequence:
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_core/execution/plan/compute.py", line 197, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn, compute_context):
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_core/execution/plan/compute.py", line 166, in _yield_compute_results
    for event in iterate_with_context(
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_utils/__init__.py", line 480, in iterate_with_context
    with context_fn():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/venvs/29ca235b6116/lib/python3.11/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
    raise error_cls(
The above exception was caused by the following exception:
dagster._check.functions.CheckError: Failure condition: Couldn't import module dagster_cloud_serverless_agent.serverless.user_code_launcher when attempting to load the configurable class dagster_cloud_serverless_agent.serverless.user_code_launcher.ServerlessUserCodeLauncher

Does Dagster Cloud support running Docker images?

[garethbrickman]

If this is in a Serverless deployment of Dagster Cloud then I don't think it's not possible to operate a Docker container independently from your deployment's container. That would require using Dagster+ Hybrid with a Docker agent.

An alternative that comes to mind is to host that Docker container in another external execution environment and then use Dagster Pipes to orchestrate it with Dagster Cloud.