Knative operator is active while knative-serving installation failed

[Deezzir]

Bug Description

The knative-serving charm is in active state while the logs indicate there is an unsupported version of K8s in the cluster. It might be a bad UX design, because nothing is deployed and the charm does not complain about it.

To Reproduce

Deploy Kubeflow charm 1.9 in EKS cluster with K8s version 1.25

Environment

Deploy Kubeflow charm 1.9 in EKS cluster with K8s version 1.25

Relevant Log Output

2025-01-15T11:10:37.013Z [knative-operator] {"severity":"EMERGENCY","timestamp":"2025-01-15T11:10:37.013411678Z","logger":"knative-operator","caller":"sharedmain/main.go:386","message":"Version c
heck failed","commit":"fbc6e15","knative.dev/pod":"knative-operator","error":"kubernetes version \"1.25.16-eks-2d5f260\" is not compatible, need at least \"1.26.0-0\" (this can be overridden with
 the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/share
dmain/main.go:386\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:256\nknative.dev/pkg/injection/sharedmain.
MainWithContext\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:210\nknative.dev/pkg/injection/sharedmain.Main\n\t/root/parts/operator/build/vendor/knative.dev/p
kg/injection/sharedmain/main.go:140\nmain.main\n\t/root/parts/operator/build/cmd/operator/main.go:26\nruntime.main\n\t/snap/go/10335/src/runtime/proc.go:250"}

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6794.

This message was autogenerated

[orfeas-k]

Discussed with the team, we should evaluate what is the expected behaviour of the upstream controller when it logs a SEVERITY message? In this case, we 'll be able to know better if this is something that we need pebble or charm to handle.

[kimwnasptd]

I tried to look into the behaviour of the upstream images.

The following are the case in the upstream image:

1. We see the same logs on the upstream knative-operator Deployment, when we use an incompatible version of K8s
2. The container crashes and the Pod goes to Error state

kubectl get pods -n knative-operator knative-operator-7b5d6b545f-4hk5c
NAME                                READY   STATUS             RESTARTS      AGE
knative-operator-7b5d6b545f-4hk5c   0/1     CrashLoopBackOff   6 (94s ago)   7m28s

And the logs are:

kubectl logs -n knative-operator knative-operator-7b5d6b545f-4hk5c
2025/01/30 10:45:48 Registering 2 clients
2025/01/30 10:45:48 Registering 3 informer factories
2025/01/30 10:45:48 Registering 3 informers
2025/01/30 10:45:48 Registering 2 controllers
{"severity":"INFO","timestamp":"2025-01-30T10:45:48.884580007Z","logger":"knative-operator","caller":"profiling/server.go:65","message":"Profiling enabled: false","commit":"4959713-dirty","knative.dev/pod":"knative-operator-7b5d6b545f-4hk5c"}
{"severity":"EMERGENCY","timestamp":"2025-01-30T10:45:48.885154491Z","logger":"knative-operator","caller":"sharedmain/main.go:390","message":"Version check failed","commit":"4959713-dirty","knative.dev/pod":"knative-operator-7b5d6b545f-4hk5c","error":"kubernetes version \"1.25.16\" is not compatible, need at least \"1.30.0-0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\tknative.dev/[email protected]/injection/sharedmain/main.go:390\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\tknative.dev/[email protected]/injection/sharedmain/main.go:255\nknative.dev/pkg/injection/sharedmain.MainWithContext\n\tknative.dev/[email protected]/injection/sharedmain/main.go:209\nknative.dev/pkg/injection/sharedmain.Main\n\tknative.dev/[email protected]/injection/sharedmain/main.go:140\nmain.main\n\tknative.dev/operator/cmd/operator/main.go:26\nruntime.main\n\truntime/proc.go:272"}

[kimwnasptd]

We can reproduce the issue with the following:

sudo snap install microk8s --classic --channel=1.25/stable
sudo microk8s kubectl config view --raw > $HOME/.kube/config
kubectl apply -f https://github.com/knative/operator/releases/download/knative-v1.17.0/operator.yaml

[kimwnasptd]

In the charm, I see that the process fails and Pebble never starts it again. Most probably because it failed in less that 1 second

2025-01-30T11:10:38.278Z [pebble] HTTP API server listening on ":38813".
2025-01-30T11:10:38.278Z [pebble] Started daemon.
2025-01-30T11:10:50.854Z [pebble] GET /v1/plan?format=yaml 372.961µs 200
2025-01-30T11:10:50.855Z [pebble] POST /v1/layers 174.951µs 200
2025-01-30T11:10:50.865Z [pebble] POST /v1/services 5.71997ms 202
2025-01-30T11:10:50.872Z [pebble] Service "knative-operator" starting: /ko-app/operator
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 2 clients
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 3 informer factories
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 3 informers
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 2 controllers
2025-01-30T11:10:50.895Z [knative-operator] {"severity":"INFO","timestamp":"2025-01-30T11:10:50.895659082Z","logger":"knative-operator","caller":"profiling/server.go:65","message":"Profiling enabled: false","commit":"fbc6e15","knative.dev/pod":"knative-operator"}
2025-01-30T11:10:50.896Z [knative-operator] {"severity":"EMERGENCY","timestamp":"2025-01-30T11:10:50.896234204Z","logger":"knative-operator","caller":"sharedmain/main.go:386","message":"Version check failed","commit":"fbc6e15","knative.dev/pod":"knative-operator","error":"kubernetes version \"1.25.16\" is not compatible, need at least \"1.26.0-0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:386\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:256\nknative.dev/pkg/injection/sharedmain.MainWithContext\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:210\nknative.dev/pkg/injection/sharedmain.Main\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:140\nmain.main\n\t/root/parts/operator/build/cmd/operator/main.go:26\nruntime.main\n\t/snap/go/10335/src/runtime/proc.go:250"}
2025-01-30T11:10:50.916Z [pebble] GET /v1/changes/1/wait?timeout=4.000s 50.048053ms 200
2025-01-30T11:10:59.298Z [pebble] GET /v1/plan?format=yaml 128.13µs 200
2025-01-30T11:11:08.315Z [pebble] GET /v1/notices?timeout=30s 30.008120623s 200
2025-01-30T11:11:12.713Z [pebble] GET /v1/plan?format=yaml 114.713µs 200
2025-01-30T11:11:14.508Z [pebble] GET /v1/plan?format=yaml 115.518µs 200
2025-01-30T11:11:18.235Z [pebble] GET /v1/plan?format=yaml 106.775µs 200
2025-01-30T11:11:21.270Z [pebble] GET /v1/plan?format=yaml 117.282µs 200
2025-01-30T11:11:24.295Z [pebble] GET /v1/plan?format=yaml 135.279µs 200