Trying to run on kind NodePort. Cannot find GW_IP

[agardnerIT]

I'm trying to test this out on kind on WSL2 windows localhost. Everything (seems to) work until I need the GW_IP and I can't find it.

TLDR: I got it working with port-forward AND gateway - but (and this is probably my lack of k8s knowledge) but I don't understand why I need port forward AND gateway. I don't need port forward with an nginx ingress and I thought gateway replaced ingress?

Here's what I'm doing:

kind create cluster --config=config.yaml

Where config.yaml:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
name: demo-cluster
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP

Note: I don't think that kubeadmConfigPatches block is necessary - it is just left over from a different cluster - but I don't think it will affect anything in this demo.

NAME                         STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP
demo-cluster-control-plane   Ready    control-plane   47m   v1.27.3   172.18.0.2    <none>

Then I follow the instructions exposing as a NodePort.

As I say, everything appears to go fine.

Then I jump into the cafe example.

First thing it asks me to do is save GW_IP into a variable - but doesn't explain how I retrieve that value.

Then it asks to save GW_PORT - again with no indication of where I retrieve that value - but I assume it's 80?

Any pointers or assistance you can provide would be great!

Happy to raise PRs if we identify docs improvements.

Debugging

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  labels:
    domain: k8s-gateway.nginx.org
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    hostname: "127.0.0.1.nip.io"

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: coffee
spec:
  parentRefs:
  - name: gateway
    sectionName: http
  hostnames:
  - "127.0.0.1.nip.io"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee
    backendRefs:
    - name: coffee
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: tea
spec:
  parentRefs:
  - name: gateway
    sectionName: http
  hostnames:
  - "127.0.0.1.nip.io"
  rules:
  - matches:
    - path:
        type: Exact
        value: /tea
    backendRefs:
    - name: tea
      port: 80

docker exec -it demo-cluster-control-plane curl 127.0.0.1.nip.io:80     
curl: (7) Failed to connect to 127.0.0.1.nip.io port 80: Connection refused

Got it working inside cluster

This command shows an IP address of 10.244.0.8 so changing gateway: hostname: "10.244.0.8.nip.io" and gateway hosts to the same value, I can at least get to the endpoints from within the cluster:

docker exec -it demo-cluster-control-plane curl http://10.244.0.8.nip.io:80/coffee
Server address: 10.244.0.9:8080
Server name: coffee-7dd75bc79b-69q2s
Date: 13/Jul/2023:00:29:48 +0000
URI: /coffee
Request ID: 90de65a36ea3f0d463cf4373a2339a90

>kubectl get gateway/gateway -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1beta1","kind":"Gateway","metadata":{"annotations":{},"labels":{"domain":"k8s-gateway.nginx.org"},"name":"gateway","namespace":"default"},"spec":{"gatewayClassName":"nginx","listeners":[{"hostname":"127.0.0.1.nip.io","name":"http","port":80,"protocol":"HTTP"}]}}
  creationTimestamp: "2023-07-12T23:17:02Z"
  generation: 6
  labels:
    domain: k8s-gateway.nginx.org
  name: gateway
  namespace: default
  resourceVersion: "7146"
  uid: 1070a91b-851a-4a6e-8e33-d248e13de689
spec:
  gatewayClassName: nginx
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    hostname: 127.0.0.1.nip.io
    name: http
    port: 80
    protocol: HTTP
status:
  addresses:
  - type: IPAddress
    value: 10.244.0.8
  conditions:
  - lastTransitionTime: "2023-07-13T00:04:12Z"
    message: Gateway is accepted
    observedGeneration: 6
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2023-07-13T00:04:12Z"
    message: Gateway is programmed
    observedGeneration: 6
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2023-07-13T00:04:12Z"
      message: Listener is accepted
      observedGeneration: 6
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2023-07-13T00:04:12Z"
      message: Listener is programmed
      observedGeneration: 6
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2023-07-13T00:04:12Z"
      message: All references are resolved
      observedGeneration: 6
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    - lastTransitionTime: "2023-07-13T00:04:12Z"
      message: No conflicts
      observedGeneration: 6
      reason: NoConflicts
      status: "False"
      type: Conflicted
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

Working Config

1. Create the NodePort service
2. Port forward: kubectl -n nginx-gateway port-forward 80:80 443:443 svc/nginx-gateway
3. Apply all example YAMLs (changing hostnames to 127.0.0.1.nip.io
4. curl http://127.0.0.1.nip.io/tea and curl http://127.0.0.1.nip.io/coffee works

But I have NO idea why I need port-forward AND a gateway. It seems to partially defeat the purpose. I don't need to port forward with an nginx ingress - and I thought gateway was a replacement?

[pleshakov]

hi @agardnerIT

Thanks for the detailed report. I see a number of things we can improve. Let me know your thoughts

(1)

But I have NO idea why I need port-forward AND a gateway. It seems to partially defeat the purpose. I don't need to port forward with an nginx ingress - and I thought gateway was a replacement?

The mentioned demo-cluster config assumes that NKG (NGINX Kubernetes Gateway) container maps its ports 80 and 443 to the same ports on the node. This is not the case of for its pod spec https://github.com/nginxinc/nginx-kubernetes-gateway/blob/4923dd98468a6881ecabc3be94af946ca43727a6/deploy/manifests/deployment.yaml#L62-L66.

I don't think it makes sense to do enable port mapping by default, because this will can lead to conflicts with other apps that can map those ports or NKG replicas.

However, we're bringing the helm chart support https://github.com/nginxinc/nginx-kubernetes-gateway/pulls and we can add enabling port-mapping as an option. Once it is there, there will be no need to port-forward in your case.

Additionally, it is possible to update https://github.com/nginxinc/nginx-kubernetes-gateway/blob/4923dd98468a6881ecabc3be94af946ca43727a6/deploy/manifests/deployment.yaml#L62-L66 before deploying NKG, to enable port mapping now:

       ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443

(2)

First thing it asks me to do is save GW_IP into a variable - but doesn't explain how I retrieve that value.

Then it asks to save GW_PORT - again with no indication of where I retrieve that value - but I assume it's 80?

Any pointers or assistance you can provide would be great!

Would it help if we?
(a) Link from the examples back to https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/docs/installation.md#expose-nginx-kubernetes-gateway
(b) extend both Create a NodePort Service and Create a LoadBalancer Service sections with commentary about getting the IP and the ports.
(c) Add a section about port-forwarding, for non-production clusters (that will cover local kind cluster) with a link here https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/docs/running-on-kind.md#running-on-kind

(3)
IP address of the Gateway status.

By default, NKG sets the IP of the pod to the Status of the Gateway resource, as a way to signal what IP(s) to use to send traffic:

status:
  addresses:
  - type: IPAddress
    value: 10.244.0.8

When NKG exposed via Service, that pod IP is not that helpful, because external clients will not use it to send traffic to NKG. We have an issue for that #604

P.S.
Note that in the Gateway API, you can configure NKG to listen to any port, not just 80 and 443. This is a big difference with the Ingress Controller, where ports 80 and 443 are assumed and enabled by default. So if you use some other ports in the Gateway resource, it will be necessary to update the Service resource or port-mapping.

spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    port: 8080
    protocol: HTTP
    hostname: "*.example.com"

[pleshakov]

note: Converted the issue to a discussion. If enhancements come out of it, we can add them as issues. This follows https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/CONTRIBUTING.md#issues-and-discussions

[agardnerIT]

Thanks for your fast response! Allow me a few days to digest and experiment with this and I'll get back to you