Migrate CRI-O jobs away from kubernetes_e2e.py

[saschagrunert]

The kubernetes_e2e.py script is deprecated and we should use kubetest2 instead.

All affected tests are listed in https://testgrid.k8s.io/sig-node-cri-o

cc @kubernetes/sig-node-cri-o-test-maintainers

Ref: https://github.com/kubernetes/test-infra/tree/master/scenarios, #20760

[haircommander]

/sig node

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[saschagrunert]

/remove-lifecycle stale

[kannon92]

/triage accepted
/priority important-longterm

[elieser1101]

Does this still need help? can i start looking at it?

[saschagrunert]

@elieser1101 I'd appreciate your eyes on that. 🙏

[elieser1101]

/assign

[elieser1101]

I opened many PRs to replicate the presubmit ones. After merging I would like to create a noop PR to test all the changes I made and fix anything broken.

After that I can start working on the periodics.
Reviews needed

[elieser1101]

Any feedback or suggestions would be appreciated.

/cc saschagrunert kannon92 krzyzacy

[saschagrunert]

The kubetest2 dra jobs seems to have a syntax error:

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/127985/pull-kubernetes-node-e2e-crio-cgrpv1-dra-kubetest2/1844649030915723264
https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/127985/pull-kubernetes-node-e2e-crio-cgrpv2-dra-kubetest2/1844649032593444864

Error: unknown flag: --label-filter

Should we fix that up here or is it another issue?

[elieser1101]

those 2 are part of the batch I migrated to kubetest2, I can look at it

[kannon92]

ah sorry, I missed this. #33647

@elieser1101 There are quite a few ones failing.

[pacoxu]

With #33658, pull-kubernetes-node-e2e-crio-cgrpv1-dra-kubetest2 is now passed. pull-kubernetes-node-e2e-crio-cgrpv2-dra-kubetest2 is similar and should be fixed as well.

• example: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/127511/pull-kubernetes-node-e2e-crio-cgrpv1-dra-kubetest2/1846087616173182976

[elieser1101]

For the test that don't pass I can se the following (on kubernetes/kubernetes#128092)
pull-kubernetes-node-crio-cgrpv1-evented-pleg-e2e-kubetest2 fails and the non kubetest2 have been failing for some time now
https://testgrid.k8s.io/sig-node-presubmits#pr-crio-cgrpv1-evented-pleg-gce-e2e-kubetest2
https://testgrid.k8s.io/sig-node-presubmits#pr-crio-cgrpv1-evented-pleg-gce-e2e

And for the
pull-kubernetes-node-crio-cgrpv2-imagefs-e2e-kubetest2 https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/pr-logs/directory/pull-kubernetes-node-crio-cgrpv2-imagefs-e2e-kubetest2
pull-kubernetes-node-crio-cgrpv2-splitfs-e2e-kubetest2 https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/pr-logs/directory/pull-kubernetes-node-crio-cgrpv2-splitfs-e2e-kubetest2

both if i review the job history worked at some point but is not consistent so i'm not sure if there is something on my side to complete for this, any pointers on how to proceed with these jobs will be helpful
@kannon92

[kannon92]

The non kubetest jobs for imagefs seem pretty green. It sounds like there is a kubetest migration issue.

[kannon92]

@elieser1101

Where are we with this for presubmits?

[elieser1101]

Still got no luck with the jobs mentioned here

I think could be related whit a kubetest2 issue to which I opened a PR, at the moment is no possible to set container-runtime-endpoint which is always defaulting to containerd.

we can see the command the jobs are running includes the flag duplicated.

--container-runtime-endpoint=unix:///run/containerd/containerd.sock --container-runtime-endpoint=unix:///var/run/crio/crio.sock

[kannon92]

I see an issue with the DRA tests:

• https://testgrid.k8s.io/sig-node-cri-o#pr-node-kubelet-crio-cgrpv1-dra-kubetest2
• https://testgrid.k8s.io/sig-node-cri-o#pr-node-kubelet-crio-cgrpv2-dra-kubetest2

I think there is an issue with the label-filter and it is not finding the right jobs for DRA.