Running device plugin with mixed mode MIG without SYS_ADMIN

[vishnukarthikl]

Hello all, I am evaluating whether the device plugin can be run without SYS_ADMIN capabilities for mixed mode MIG. Currently the capability is needed to query the MIG slice's memory info. But this also increases the security surface area of the Pod and I am considering if we can reduce it.

Based on @klueska comment, it seems possible to pass the capabilities directly into the container without having to explicitly add SYS_ADMIN. I tried to bind mount the host's /proc/driver/nvidia/capabilities/mig/monitor into container but running into pod error. Using a build from release-0.13

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-gpu-dp-daemonset
  namespace: $namespace
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      gpu-device-plugin: nvidia
  template:
    metadata:
      labels:
        gpu-device-plugin: nvidia
    spec:
      containers:
      - args:
        - --fail-on-init-error=false
        - --mig-strategy=mixed
        - --pass-device-specs=true
        env:
        - name: CUDA_DEVICE_ORDER
          value: PCI_BUS_ID
        - name: NVIDIA_MIG_MONITOR_DEVICES
          value: all
        image: gcr.io/$project/nvidia/k8s-device-plugin:v0.13.0
        imagePullPolicy: IfNotPresent
        name: nvidia-device-plugin-ctr
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kubelet/device-plugins
          name: device-plugin
        - name: nvidia-mig-monitor
          mountPath: /proc/driver/nvidia/capabilities/mig/monitor
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/lib/kubelet/device-plugins
          type: ""
        name: device-plugin
      - name: nvidia-mig-monitor
        hostPath:
          path: /proc/driver/nvidia/capabilities/mig/monitor

  Warning  Failed     13s (x5 over 110s)   kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/proc/driver/nvidia/capabilities/mig/monitor" to rootfs at "/proc/driver/nvidia/capabilities/mig/monitor": "/run/containerd/io.containerd.runtime.v2.task/k8s.io/nvidia-device-plugin-ctr/rootfs/proc/driver/nvidia/capabilities/mig/monitor" cannot be mounted because it is inside /proc: unknown

Has anyone made this working? Any examples would definitely help.

Thanks

[klueska]

You would have to inject both /proc/driver/nvidia/capabilities/mig/monitor and /dev/nvidia-caps/nvidia-cap1

[klueska]

Though you may not be able to mount the /proc stuff directly (which shouldn't strictly be necessary). Try it with just the device node.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed.