-
Notifications
You must be signed in to change notification settings - Fork 621
[pod-gateway] helm chart doesn't work with cilium #1633
Comments
the ping to I would suggest to debug 2 things:
|
so there are things I've tried/tested:
Here are some additional logs from test pod
|
Could you please also post the logs from the gateway pod init container? This is the one setting up the ip, routes and iptables of the gateway. What is your policy enforcement? https://docs.cilium.io/en/v1.9/policy/intro/ I expect the problem being the UDP traffic not going through... |
I use default policy enforcement mode Here are all logs from gateway-init container from my test pod
pod-gateway routes container
and also pod-gateway pod-gateway container
|
I've managed to fix connection issue with cilium, it turns out it encapsulates all traffic between nodes using vxlan by default. In order to use pod-gateway you need to disable encapsulation (switch to native-routing). Here are instructions how to do that: https://docs.cilium.io/en/v1.11/concepts/networking/routing/#native-routing Thanks for your support! btw. please update helm chart with newest pod-gateway and webhook image |
Glad you managed to fix the issue and for sharing the solution!
You can always override the image tags in your |
@legalgig - glad you got it working - is there any disadvantages on using native routing? Or why is not vxlan on vxlan working? Ideally we will add the instructions with the trade-offs to https://docs.k8s-at-home.com/guides/pod-gateway/ |
sure I'll try to create PR this week ;)
As far as I can tell there are no real disadvantages on using native routing it is just much needier in terms of configuration than vxlan encapsulating. If you want to use native routing you need to fulfill one of these points:
in most cases homelabbers tend to have all k8s hosts in one L2 network so it suffices to set following helm values:
Good question, when you use vxlan encapsulation it creates routes for pod cidr (in this case 10.0.0.0/16) and everything that is not within that cidr (i.e. traffic from pods to pod-gateway on 172.16.0.0/24 network) is routed into default gateway (your home router) where it goes into nirvana because your router isn't aware how to route packets to your k8s pods. I'm just speculating so don't quote me on that :) |
I'm having the same issue "No DHCPOFFERS received" on flannel Stuck on this for few weeks My pod-gateway config:
OpenVPN connects to server and I am able to curl blocked websites pod-gateway logs
pod-gateway routes logs
transmission service (routed pod) logs
Any guidance or help appreciated :) |
What are your network policies? You should have:
Was this config working for you before? If yes, what has changed? Beyond the network policy I do not see any other reason for the problem which is that the vxlan packages are not being transmitted. Flannel by default does not use vxlanx so I do not think this is the same problem as reported by @legalgig so opening a new issue would be better. |
Just another thing to consider - there are some VPNs that route ALL traffic through them so the DHCP server gets a query but the reply does not arrive. When this is the case the DHCP server should still log that the query was received:
@bjw-s found this with being the case with gluetun. |
I can confirm this solved my problem. Although the dhcp problems are still there
I have this image but everything else is working as expected. |
Helm chart name
pod-gateway
Helm chart version
5.4.2
Container name
ghcr.io/k8s-at-home/pod-gateway
Container tag
v1.2.6
Description
Hi,
I've been using pod-gateway for quite some time and it was working perfectly fine with calico but recently I've decided to migrate whole cluster from calico to cilium. After migration pod-gateway stopped working. It seems that there is no connection over vxlan interface. I've already tried different vxlan id, and removing NOT_ROUTED_TO_GATEWAY_CIDRS and/or VPN_LOCAL_CIDRS
Kuberentes:
Here are some logs
pod-gateway container
app gateway init logs
cilium values
Expected result
fix integration with cilium cni
Helm values to reproduce
Additional Information
No response
Repo link
No response
The text was updated successfully, but these errors were encountered: