[K8s] Setting image tag from UI

[HynekBlaha]

Discussed in #19970

I would like to contribute by implementing the feature mentioned in step 3).
I believe this quality-of-life improvement is something that might be useful for other users as well.
Does it make sense to you, maintainers (@sryza)?

^{Originally posted by HynekBlaha February 22, 2024}
Why do I want to do it?
I believe it would help my team process backfills in parallel on rapidly changing staging environment.

Details:

• Staging Dagster instance running in Kubernetes
• Team of 10 members deploying to a single code location
• Each team member wants to run backfill for a job he works on (image build from his branch)
• Each member would like to run backfill (queued) which would take long time (eg. 1 day) to process
• They would like to run multiple backfill jobs from various images in parallel

Possible solutions:

1. By default, job is executed with the image of current code location.

• Job runs with image that is current when it gets dequeued.
• ❌ On staging dagster, code location images change rapidly. This leads to executing job with different image than expected.

2. Hardcode image to job_tags

• The image tag is dynamically resolved in our CI.
• ❌ Hardcoding staging images is code is prone to errors (and not a good practise in general).

@job(executor_def=k8s_job_executor, tags={
    "dagster-k8s/config": {
        "container_config": {
            "image": "myregistry/myimage:mytag"
        }
    }
})
def my_k8s_job():
    my_op()

3. Set image from UI

• Each team member can explicitly set job image tag.
• Once submitted, another member can deploy different image to code location and submit another backfill with different image tag.
• In my opinion the best option.
• ❌ However, when the job gets dequeued the image gets unexpectedly overridden with current code deployment image tag.

Notes:

We tried other approaches, such as:

• Creating multiple code locations from the same codebase - it complains about same asset being defined multiple times.
• Splitting the codebase to multiple code locations - doesn't solve the problem, just reduces the probability of clash and increases the project complexity.

In my opinion, the solution 3) is a bug. If an image is provided as run tag from UI, it should be used.
Is there better approach that I overlooked?

Thank you for your hard work on Dagster. We love it.

[gibsondan]

Hi @HynekBlaha - i just wanted to check what exactly doesn't work about the solution that was proposed here of using {"container_config": "image"} : #19970 (comment)

You mentioned that that didn't work because it wouldn't get propagated to the executor - but the executor can also be configured in the UI right? You could do something like this:

execution:
  k8s:
    job_image: your_image

Appreciate that setting both a tag and custom executor config is kind of complex though.

One reason that we are hesitant to support this in a first-class way is that Dagster currently uses information from the code server to generate snapshots about the code that are then used during the run. So if the code server and the run are using different images that are out of sync, we can't actually guarantee that the run will always work. That may not be an issue in your specific case, but just explaining why we would be hesitant to make, say, a prominent link the UI that allows you to vary the image at runtime.

[HynekBlaha]

Hi @gibsondan,

You mentioned that that didn't work because it wouldn't get propagated to the executor - but the executor can also be configured in the UI right?

I have to admit I didn't get that idea. While the executor can be configured in the UI, once you enter "Open Launchpad", only single partition can be selected. In our use case, we need to submit backfills (with higher hundreds of partitions).

If I explicitly set tag:
dagster-k8s/config:{"container_config": {"image": "my_image:latest", "resources": {"requests": {"cpu": "4", "memory": "10Gi"}}}},
once the job starts, additional tag dagster/image with current code location image is added.
While the image is not used, it makes investigations using GraphQL chaotic. (scenario: My image had a bug. I need to find all affected runs to re-run them to fix corrupted data).

As far as I know, run tags provided from should UI have precedence. But dagster/image gets overwritten every time.

One reason that we are hesitant to support this in a first-class way is that Dagster currently uses information from the code server to generate snapshots about the code that are then used during the run. So if the code server and the run are using different images that are out of sync, we can't actually guarantee that the run will always work.

I agree, in rare cases it has happened to us as well. I still think the same incoherency issues would happen if I set the image via dagster-k8s/config. It is just more complicated setup - the json can be quite verbose and it's easier to make a mistake.
I believe everyone who decides to change job image is aware of possible shortcomings, that could happen if current and explicit image diverge a lot.

We wish there was a more user-friendly way to explicitly set image in UI for job/backfill (via existing tag: dagster/image).
Just to put into perspective how bothersome is to explicitly set the image: we eventually decided to split the original single code deployment to several, so we can test better in parallel (at cost of 7x dagster dev start-up time on localhost).

Please let me know if my points make sense to you.
Thank you, Hynek

[HynekBlaha]

I forgot the most important point 🙄,
We don't plan to run any exotic images. We just want to submit backfill with the same image as the code deployment currently has.
We have global job policy = retry two times.
Even if I would wait until the backfill is completely submitted, if my teammate updates the code deployment with his image, the retried job starts with his image, not mine. While this makes sense in production, where you usually want to run jobs with the most up-to-date image, the code deployment images on staging cluster can change frequently (testing a new assets in parallel).

[gibsondan]

Got it - I know it has been open for a long time, but I believe we are planning to work on that issue for allowing you to set configuration for backfills fairly soon (there is a project underway to unify runs and backfills more in various ways - that is one of them).

This is a lot less than what you are asking for, but I would also entirely support a change that made the K8sRunLauncher set the dagster/image tag to the image that it's actually going to use if it is being overridden via dagster-k8s/config tag - that seems uncontroversial and I agree that it's very confusing for the tag to be something different than the image that the run is actually going to be using. Would you be interested in starting with that?

We're also not in love at all with the dagster-k8s/config solution (and generally the process of configuring runs via complex JSON tags) - it would be very nice if there was a way to more explicitly configure runs in the UI in the same way that you can configure the executor.

A problem with making it always prefer dagster/job_image if it's set is that when you retry the run, the tags from the run being retried are copied over - and the run launcher has no way of knowing if they were set in the UI or if they are from the previous run. So if we made the run launcher by default accept that tag, it would be a potentially breaking change in the default run retry behavior for everybody.

[HynekBlaha]

Got it - I know it has been open for a long time, but I believe we are planning to work on that issue for allowing you to set configuration for backfills fairly soon (there is a project underway to unify runs and backfills more in various ways - that is one of them).

Glad to hear that!
Just being curious, is there a discussion or some other place where I could read about it (or other planned features)?

This is a lot less than what you are asking for, but I would also entirely support a change that made the K8sRunLauncher set the dagster/image tag to the image that it's actually going to use if it is being overridden via dagster-k8s/config tag - that seems uncontroversial and I agree that it's very confusing for the tag to be something different than the image that the run is actually going to be using. Would you be interested in starting with that?

Sure, I will be happy to fix it. :)

A problem with making it always prefer dagster/job_image if it's set is that when you retry the run, the tags from the run being retried are copied over - and the run launcher has no way of knowing if they were set in the UI or if they are from the previous run. So if we made the run launcher by default accept that tag, it would be a potentially breaking change in the default run retry behavior for everybody.

Thank you for pointing this out, it makes sense. If retried jobs would always always prefer original dagster/job_image, it could lead to unexpected fails after update due to inconsistencies we talked about (just in much more common case).

We're also not in love at all with the dagster-k8s/config solution (and generally the process of configuring runs via complex JSON tags) - it would be very nice if there was a way to more explicitly configure runs in the UI in the same way that you can configure the executor.

My first idea was to suggest another way of setting an image in a non-breaking way (e.g. with tag dagster/force_job_image).
But the more I think about it, it doesn't bring that much value and would replicate what can be already achieved by more verbose dagster-k8s/config. On the other hand, this tag might also be propagated to step jobs, but as you mentioned there will be a solution once the unification of runs and backfill is in place in the future. So I won't push this idea as it doesn't feel that tight and conceptual.

Thank you for finding time to discuss this idea, @gibsondan .
I understand the reasoning behind it. Plus I learned how to set job image for k8s_job_executor in UI. :)

[HynekBlaha]

Label dagster/image fixed in: #24076