Ingress routes to incorrect podIP #342
Description
Is this a BUG or FEATURE REQUEST?:
Bug
Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?:
Yes
Bug:
Y
What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details
istioctl version: 0.7.1
kubectl version: v1.9.6-gke.1
Is Istio Auth enabled or not ?
Not using auth
Used standard Istio 0.7.1 scripts with automatic sidecar injection
The ingress is used with HTTPS
What happened:
We have a cluster with a lot of services. This doesn't always happen, but is frequent enough. Ingress returns 503 error, browser displays: upstream connect error or disconnect/reset before headers.
Output of curl http://istio-pilot:15003/v1/registration contains these lines:
{
"service-key": "testing-client.app.svc.cluster.local|http-testing-c",
"hosts": [
{
"ip_address": "10.4.3.28",
"port": 80
}
]
},
Pod YAML:
podIP: 10.4.3.28
hostIP: 10.132.0.11
Service YAML:
clusterIP: 10.7.255.54
However, the ingress somehow doesn't route to the correct IP addresses for host and pod and tries routing the requests to some other IP:
"GET / HTTP/2" 503 UF 0 57 1004 - "10.132.0.9" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36" "eb88831b-5569-9d52-98af-10d574882fd7" "testing-client.website.com" "10.4.4.247:80"
So basically the pilot has the right IP address for the service but ingress is routing to incorrect IP.
Restarting pilot pods fix the issue temporarily. Restarting any other components or the service has no effect.
What you expected to happen:
Ingress should always route to correct IP.
How to reproduce it:
No idea, seems to happen randomly. Some deployments go alright, others don't. The manifests for deploying the app are pretty standard and are deployed through helm.
Feature Request:
N