AWS CSI wrapper doesn't seem to work

[ldoktor]

Describe the bug

The persistency of AWS CSI wrapper doesn't seem to work with kata-remote.

How to reproduce

1. Deploy CAA

pushd $CAA/src/cloud-api-adaptor
oc get nodes
kubectl label node XXX node.kubernetes.io/worker=
make deploy
watch oc get runtimeclasses.node.k8s.io

2. Deploy CSI

kubectl create secret generic aws-secret \
    --namespace kube-system \
    --from-literal "key_id=${AWS_ACCESS_KEY_ID}" \
    --from-literal "access_key=${AWS_SECRET_ACCESS_KEY}"
kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.36"
watch kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver

3. Deploy PeerPod CSI

pushd $CAA/src/csi-wrapper/examples/aws
kubectl apply -f ../../crd/peerpodvolume.yaml
kubectl apply -f rbac-ebs-csi-wrapper-runner.yaml
kubectl apply -f rbac-ebs-csi-wrapper-podvm.yaml
kubectl patch deploy ebs-csi-controller -n kube-system --patch-file patch-controller.yaml
kubectl -n kube-system delete replicaset -l app=ebs-csi-controller
kubectl patch ds ebs-csi-node -n kube-system --patch-file patch-node.yaml
watch kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver

4. Test dynamic provisioning

oc apply -f dynamic-provisioning
watch kubectl exec app -- cat /data/out.txt
kubectl exec app -- touch /data/test1
kubectl exec app -- ls -al /data/

oc delete -f dynamic-provisioning/pod.yaml
oc apply -f dynamic-provisioning/pod.yaml
watch kubectl exec app -- cat /data/out.txt
kubectl exec app -- touch /data/test2
kubectl exec app -- ls -al /data/

The outcome is that the ls shows only the files of this instance only while with persistent storage I'd expect to get the previous files as well, shouldn't it?

---

I also tried using the class directly:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Mi
  storageClassName: ebs-sc
---
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  serviceAccountName: csi-ebs-podvm-sa
  containers:
    - name: test
      image: registry.fedoraproject.org/fedora
      command: ["sleep", "infinity"]
      securityContext:
        runAsNonRoot: false
        runAsUser: 0
        privileged: true
      volumeMounts:
      - mountPath: /workspace/workspace
        name: persistent-storage
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim
  runtimeClassName: kata-remote

The same test by recreating the pod only, keeping the pvc. The result was the same, only files from the current instance are present. Note the pvc is kept, has the same name and in AWS only one volume is created and kept.

CoCo version information

confidential-containers/cloud-api-adaptor e14ad0f

What TEE are you seeing the problem on

None

Failing command and relevant log output

[medic@fedora dynamic-provisioning ]$ watch kubectl exec app -- cat /data/out.txt
[medic@fedora dynamic-provisioning ]$ kubectl exec app -- touch /data/test1
kubectl exec app -- ls -al /data/
Defaulted container "app" out of: app, csi-podvm-node-driver, csi-podvm-wrapper
Defaulted container "app" out of: app, csi-podvm-node-driver, csi-podvm-wrapper
total 8
drwxr-xr-x 2 root root   80 Nov  5 08:48 .
drwxr-xr-x 1 root root 4096 Nov  5 08:42 ..
-rw-r--r-- 1 root root 1904 Nov  5 08:48 out.txt
-rw-r--r-- 1 root root    0 Nov  5 08:48 test1
[medic@fedora dynamic-provisioning ]$ oc delete -f pod.yaml 
pod "app" deleted
[medic@fedora dynamic-provisioning ]$ oc apply -f pod.yaml 
pod/app created
[medic@fedora dynamic-provisioning ]$ watch kubectl exec app -- cat /data/out.txt
[medic@fedora dynamic-provisioning ]$ kubectl exec app -- touch /data/test2
Defaulted container "app" out of: app, csi-podvm-node-driver, csi-podvm-wrapper
[medic@fedora dynamic-provisioning ]$ kubectl exec app -- ls -al /data/
Defaulted container "app" out of: app, csi-podvm-node-driver, csi-podvm-wrapper
total 8
drwxr-xr-x 2 root root   80 Nov  5 08:53 .
drwxr-xr-x 1 root root 4096 Nov  5 08:52 ..
-rw-r--r-- 1 root root  420 Nov  5 08:53 out.txt
-rw-r--r-- 1 root root    0 Nov  5 08:53 test2
[medic@fedora dynamic-provisioning ]$ oc delete -f pod.yaml 
pod "app" deleted
[medic@fedora dynamic-provisioning ]$ oc apply -f pod.yaml 
pod/app created
[medic@fedora dynamic-provisioning ]$ watch kubectl exec app -- cat /data/out.txt
[medic@fedora dynamic-provisioning ]$ oc delete -f pod.yaml 
pod "app" deleted
[medic@fedora dynamic-provisioning ]$ oc apply -f pod.yaml 
pod/app created
[medic@fedora dynamic-provisioning ]$ kubectl exec app -- touch /data/test3
Defaulted container "app" out of: app, csi-podvm-node-driver, csi-podvm-wrapper
[medic@fedora dynamic-provisioning ]$ kubectl exec app -- ls -al /data/
Defaulted container "app" out of: app, csi-podvm-node-driver, csi-podvm-wrapper
total 8
drwxr-xr-x 2 root root   80 Nov  5 09:04 .
drwxr-xr-x 1 root root 4096 Nov  5 09:04 ..
-rw-r--r-- 1 root root  280 Nov  5 09:05 out.txt
-rw-r--r-- 1 root root    0 Nov  5 09:04 test3

[bpradipt]

@ldoktor an obvious question, did CSI worked without kata-remote?

[ldoktor]

The original did but with the CSI wrapper it complained about non-existent mounts (IIRC).