Flux 2.4.0 - PSA label - Failed to create temporary working directory

[HamzaZo]

In a newly created cluster with no existing policies(kyverno or OPA), I updated the Flux Operator to the latest version. It appears that the PSA label added to the flux-system namespace is blocking the reconciliation. Upon checking the logs of the source-controller pod, I found the following entries:

{"level":"error","msg":"Reconciler error","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"bitbucket","namespace":"flux-system"},"namespace":"flux-system":"name":"bitbucket","error":"failed to create temporary working directory: mkdir /tmp/gitrepository-flux-system-bitbucket-xxxxx: permission denied"

$ flux get ks 
NAME       REVISION                 SUSPEND            READY     MESSAGE       
app        refs/heads/xxx             false                     false         tmp dir error: mkdir /tmp/kustomization  permission denied 

Removing the PSA label

pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest

and restart the source-controller pods works fine

Any idea why the PSA labels were added?

[stefanprodan]

Those labels are in upstream Flux for some years now: https://github.com/fluxcd/flux2/blob/main/manifests/install/namespace.yaml

I don't see how these would block the tmp operations, PSA should not impact write operation.

[HamzaZo]

I agree, it's strange. I'm using version 1.32, by the way. I also noticed that even if I remove those labels, they get added again when the operator restarts(or other components in flux-system added them). Is there a way to prevent these labels from being added without relying on a custom mutating rule? @stefanprodan

[stefanprodan]

You can remove them in the FluxInstance:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
spec:
  kustomize:
    patches:
      - patch: |
          - op: remove
            path: /metadata/labels/pod-security.kubernetes.io~1warn
          - op: remove
            path: /metadata/labels/pod-security.kubernetes.io~1warn-version 
        target:
          kind: Namespace

Another option is to set the cluster type to openshift

[HamzaZo]

Thanks for your help

[HamzaZo]

I’ve reopened the issue because it persists. I’m unsure why the source-controller/kustomization-controller encounters the error: error: failed to create temporary working directory: mkdir /tmp/gitrepository-xxxxx-bitbucket: permission denied. I am running Kubernetes version 1.32.1 along with the latest version of the operator, I haven’t experienced this issue previously.

Is there something in the FluxInstance configuration that I may be missing or need to add? @stefanprodan

[stefanprodan]

This error can't be related to the operator, the way it configures source-controller is identical to upstream. If this happens occasionally, I suspect it's related to your node storage. Moving the storage in-memory would be an option to avoid these errors. Did you enabled persistance storage for source-controller?

[HamzaZo]

I have the same issue on the source-controller and kustomize-controller both of them, when I changed from emptyDir to use
persistentVolume instead I no longer see the error on the source-controller

I don't think its related to the node storage because I create a dumy pod with emptyDir and I can create dir and do everything on it.

[stefanprodan]

The empty dir permissions are set by kubelet on the node storage, so it can't be anything else.

[stefanprodan]

In the FluxInstance you can move tmp to in-memory like so:

patches:
  - patch: |
      - op: replace
        path: /spec/template/spec/volumes/0
        value:
          name: tmp
          emptyDir:
            medium: Memory      
    target:
      kind: Deployment
      name: kustomize-controller
  - patch: |
      - op: replace
        path: /spec/template/spec/volumes/1
        value:
          name: tmp
          emptyDir:
            medium: Memory      
    target:
      kind: Deployment
      name: source-controller

[HamzaZo]

do you think it might be related to the securityContext: fsGroup

with persistentVolume

kubectl exec -it source-controller-xxxxx -- sh 
/tmp $ id 
uid=65534(nobody) gid=65534(nobody) groups=1337,65534(nobody)
/tmp $ mkdir test 
/tmp $ ls -la 
total 20 
drwxrwxrwx   3 root  root   4096  18 fév 09:43 .
drwxrwxrwx  1 root  root   4096  18 fév 09:43 ..
drwxrwxrwx  81 root  root  8192  18 fév 09:48 .snapshot
drwxr-xr-x  2 nobody  nobody  4096  18 fév 09:48 test 


with emptyDir 

kubectl exec -it source-controller-xxxxx -- sh 
/tmp $ id 
uid=65534(nobody) gid=65534(nobody) groups=1337,65534(nobody)
/tmp $ mkdir test 
/tmp $ ls -la 
total 20 
drwxrwxrwx   3 root  1337   4096  18 fév 09:43 .
drwxrwxrwx  1 root  root   4096  18 fév 09:43 ..
drwxr-xr-x  2 nobody  1337  4096  18 fév 09:48 test 

[HamzaZo]

ok thanks I'll update the fluxInstance and test it