Bug: Bookinfo tasks not working with Istio 0.8.0 on GKE

[jbrook]

BUG

Istio 0.8.0 LTS release from github releases.

Installed with Istio auth

What happened:

Installed Istio LTS release on GKE and tried to follow Bookinfo tasks. It worked up until creating the default v1 routes for the services. Error message when trying to access http://<gateway_url>/productpage:

503 - "upstream connect error or disconnect/reset before headers"

What you expected to happen:

I expected to see the bookinfo page backed by v1 of each of the services.

This works correctly with a slightly older daily release: istio-release-0.8-20180520-18-17

It also fails with a recent daily release: release-0.8-20180605-09-15

How to reproduce it:

Start a GKE 1.9 cluster from Google Cloud Shell:

gcloud container clusters create hello-istio \
    --cluster-version=1.9 \
    --machine-type=n1-standard-2

cluster role bindings:

kubectl create clusterrolebinding cluster-admin-binding \
    --clusterrole=cluster-admin \
    --user=$(gcloud config get-value core/account)

Download and install Istio 0.8.0:

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=0.8.0 sh -
cd ./istio-0.8.0
export PATH=$PWD/bin:$PATH
kubectl apply -f install/kubernetes/istio-demo-auth.yaml

Enable automatic sidecar injection for the default namespace:

kubectl label namespace default istio-injection=enabled

Deploy Bookinfo:

kubectl apply -f samples/bookinfo/kube/bookinfo.yaml

Create gateway and corresponding virtual service:

istioctl create -f samples/bookinfo/routing/bookinfo-gateway.yaml

Find the external IP of the load balancer:

kubectl get svc istio-ingressgateway -n istio-system

Use the external IP to access the productpage in a browser:

http://<external ip>/productpage

This works.

Create default v1 routing rule according to instructions here:

istioctl create -f samples/bookinfo/routing/route-rule-all-v1.yaml

Try to access the product page in a browser and get a 503 error with the message:

upstream connect error or disconnect/reset before headers

Extra info:

Don't see any errors or requests arriving (after the initial 200s) in istio-proxy sidecar for the productpage pod. It seems to be listening:

[2018-06-05 21:49:10.621][14][info][upstream] external/envoy/source/server/lds_api.cc:62] lds: add/upda
te listener '0.0.0.0_9080'
[2018-06-05 21:49:10.622][14][info][upstream] external/envoy/source/server/lds_api.cc:62] lds: add/upda
te listener '0.0.0.0_80'

istio-ingressgateway pod shows the following logs for a single failed request - note 404s:

[2018-06-06T10:59:18.862Z] "GET / HTTP/1.1" 404 NR 0 0 3 - "10.24.0.1" "Mozilla/5.0 (Windows NT 10.0; W
OW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "f359ecc8-82d1-9cf1-a4
a7-585fdc2feddc" "35.204.229.59:80" "-"
[2018-06-06T10:59:22.233Z] "GET / HTTP/1.1" 404 NR 0 0 2 - "10.164.0.6" "Mozilla/5.0 (Macintosh; IntelM
ac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" "210577e4-a
3a3-9cbe-98b2-59e94e90306e" "35.204.229.59" "-"
[2018-06-06T10:59:27.667Z] "GET /productpage HTTP/1.1" 503 UC 0 57 1 - "10.164.0.6" "Mozilla/5.0 (Macin
tosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
" "cce0f0ae-36a5-93d6-a0e1-9939a4f78515" "35.204.229.59" "10.24.0.14:9080"

[vadimeisenbergibm]

The same problem on IBM Cloud.

[vadimeisenbergibm]

The problem seems to be related to Istio auth. I do not see it with install/kubernetes/istio-demo.yaml.

To diagnose the problem, no need to create a gateway, it can be diagnosed by running curl from the sleep container (samples/sleep/sleep.yaml).

The steps to reproduce:

1. kubectl apply -f install/kubernetes/istio-demo-auth.yaml
2. kubectl label namespace default istio-injection=enabled
3. kubectl apply -f samples/bookinfo/kube/bookinfo.yaml
4. kubectl apply -f samples/sleep/sleep.yaml

kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}') -c sleep -- curl -v reviews:9080/reviews/1
* Hostname was NOT found in DNS cache
*   Trying 172.21.52.247...
* Connected to reviews (172.21.52.247) port 9080 (#0)
> GET /reviews/1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: reviews:9080
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-powered-by: Servlet/3.1
< content-type: application/json
< date: Wed, 06 Jun 2018 16:43:23 GMT
< content-language: en-US
< content-length: 295
< x-envoy-upstream-service-time: 1068
* Server envoy is not blacklisted
< server: envoy
< 
* Connection #0 to host reviews left intact
{"id": "1","reviews": [{  "reviewer": "Reviewer1",  "text": "An extremely entertaining play by Shakespeare. The slapstick humour is refreshing!"},{  "reviewer": "Reviewer2",  "text": "Absolutely fun and entertaining. The play lacks thematic depth when compared to other plays by Shakespeare."}]}

6. istioctl create -f samples/bookinfo/routing/route-rule-all-v1.yaml

kubectl exec -it $(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}') -c sleep -- curl -v reviews:9080/reviews/1
* Hostname was NOT found in DNS cache
*   Trying 172.21.52.247...
* Connected to reviews (172.21.52.247) port 9080 (#0)
> GET /reviews/1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: reviews:9080
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 57
< content-type: text/plain
< date: Wed, 06 Jun 2018 16:50:21 GMT
* Server envoy is not blacklisted
< server: envoy
< 
* Connection #0 to host reviews left intact
upstream connect error or disconnect/reset before headers

After deleting the rules, reviews becomes available again.

@jbrook Could you please check that Istio without Auth works in your environment?
@wattli Could you please check this issue?

[holger-hoffmann]

Try using
istioctl create -f samples/bookinfo/routing/route-rule-all-v1-mtls.yaml
instead of
istioctl create -f samples/bookinfo/routing/route-rule-all-v1.yaml.

Credits go to Kim Christensen, he pointed that out on the istio-users google group: BookInfo request routing with 0.8.0 does not work?.

[sisiras]

@holger-hoffmann Thank you for the reply. But it doesn't work following mTLS auth enabled. Could you please help ?