Artifact Garbage Collect UID in artgc pod should not be defined by default

[antoinetran]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

In OpenShift environment, the artgc pod is in error because by default the UID 8737 is set at https://github.com/argoproj/argo-workflows/blob/v3.6.4/workflow/common/security_context.go#L13

I expect the UID to not be set by default, so that:

• in OpenShift environment, the UID is in the authorized range
• in other environment, the UID is the one define the container image

Workaround:
It is possible to override that.
According to doc #11485 it is possible to override the pod definition of artgc, to be set in values.yaml of helm chart:

  workflowDefaults:
    spec:
      artifactGC:
        podSpecPatch: |+
          containers:
          - name: main
            securityContext:
              # OpenShift: must be in UID range.
              runAsUser: 123456789

I also tried many other things like runAsUser: null, runAsUser: "null", runAsUser: "null", runAsUser: "" but none of these deletes the 8737 UID.
Also tried strategic merge patch:

  workflowDefaults:
    spec:
      artifactGC:
        podSpecPatch: |+
          containers:
          - name: main
            securityContext:
              $patch: delete
              runAsUser: 123456789

This will result in UID=8737.

Version(s)

v3.6.4

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

Any helloworld that has artifact S3 as output.

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[antoinetran]

When we deploy OpenShift, the UID range change at each new project. I don't want to hard-code the UID as written in the workaround above. Instead it should be empty and then we get no issue at all!

[antoinetran]

Related to #12316