When using ServiceLB in CIS profile, only traffic coming on node external ip is reaching the ServiceLB service.

[xyzzyz]

Environmental Info:
RKE2 Version:
rke2 version v1.28.12+rke2r1 (27989db)
go version go1.22.5 X:boringcrypto

Node(s) CPU architecture, OS, and Version:
RHEL 8.9
Linux lattice.local 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Mar 14 14:20:09 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

Describe the bug:
When you enable CIS profile, and create a ServiceLB-backed LoadBalancer (which should effectively just be a Host Port), only traffic that connects to the external node ip is able to reach the service. No other node's addresses work, including 127.0.0.1. This works just fine on non-CIS RKE2, I can hit my service on any interface.

Steps To Reproduce:

# curl -sfL https://get.rke2.io | sh -
...
# cat /etc/rancher/rke2/config.yaml 
profile: cis
enable-servicelb: true
# systemctl start rke2-server.service
...
# cat echo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: echo
        image: n0r1skcom/echo
        ports:
        - containerPort: 3333
        securityContext:
          runAsUser: 1000
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
---
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  selector:
    app: echo
  ports:
  - protocol: TCP
    port: 4444
    targetPort: 3333
  type: LoadBalancer
# kubectl apply -f echo.yaml
deployment.apps/echo created
service/echo created
$ kubectl get svc
NAME         TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
echo         LoadBalancer   10.43.94.2   10.0.2.15     4444:32316/TCP   47s
kubernetes   ClusterIP      10.43.0.1    <none>        443/TCP          8m13s

Expected behavior:
I can reach my echo LoadBalancer on 10.0.2.15:4444 and on 127.0.0.1:4444

Actual behavior:
I can only reach it on the external IP:

$ curl 10.0.2.15:4444
Container information:
Hostname:	echo-597494bdb8-tcd9x

Interface	NetMask		IP
lo		255.0.0.0	127.0.0.1	
eth0		255.255.255.255	10.42.0.8	

TCP Remote information:
IP:	10.0.2.15

Data received:
GET / HTTP/1.1
Host: 10.0.2.15:4444r
User-Agent: curl/7.61.1
Accept: */*
$ curl 127.0.0.1:4444 --connect-timeout 1
curl: (28) Connection timed out after 1001 milliseconds

Additional context / logs:
The cause of this behavior is NetworkPolicies set up by CIS profile. When I connect to the service on the external IP, it appears to be coming from the external IP, but when I connect to it using 127.0.0.1, the traffic appears to be coming from the corresponding ServiceLB pod running in a different kube-system namespace, and as such gets rejected by CIS profile NetworkPolicy.

I can work around this behavior by adding extra NetworkPolicies, e.g.:

$ curl 127.0.0.1:4444 --connect-timeout 1
curl: (28) Connection timed out after 1002 milliseconds
$ cat echo-policy.yaml 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress-on-port-3333
spec:
  podSelector:
    matchLabels:
      app: echo
  policyTypes:
  - Ingress
  ingress:
  - ports:
    - protocol: TCP
      port: 3333
$ kubectl apply -f echo-policy.yaml 
networkpolicy.networking.k8s.io/allow-all-ingress-on-port-3333 created
$ curl 127.0.0.1:4444 --connect-timeout 1
Container information:
Hostname:	echo-597494bdb8-tcd9x

Interface	NetMask		IP
lo		255.0.0.0	127.0.0.1	
eth0		255.255.255.255	10.42.0.8	

TCP Remote information:
IP:	10.42.0.10

Data received:
GET / HTTP/1.1
Host: 127.0.0.1:4444
User-Agent: curl/7.61.1
Accept: */*
$ kubectl -n kube-system get pods -o wide | grep svc
svclb-echo-a178aca6-nhp44                               1/1     Running     0             12m   10.42.0.10   lattice.local   <none>           <none>

but I feel that it's ServiceLB that should be adding these policies in addition to setting up iptables rules. I don't want to add these NetworkPolicies, because if I add them when it's not necessary (in non-CIS mode), it will break other things (as adding a first network policy to the cluster has an effect of blocking everything that's not in that network policy). This is additionally made extra confusing by the fact that you have to add a rule to target container port, not the Load Balancer service port.

[brandond]

ServiceLB does not manage network policies, nor do we intend to make it do so. Since ServiceLB is disabled by default, rke2 doesn't ship with any policies that cover it. If you're going to enable it on a hardened cluster, you will need to add netpols for it as well.

We could perhaps cover this in the docs with some example policies.

[xyzzyz]

That sounds pretty reasonable (even if not exactly the most convenient from the user's perspective). I'd appreciate something in the docs that would mention this, and give guidance on what kind of NetworkPolicies would be most appropriate to set up with ServiceLB + CIS profile.