Let's Encrypt How-To documentation

[Curtingham]

Question

I have been working through the How-To guides for NGINX Gateway Fabric on the official documentation site. In particular, I've set the following up in my environment, and they've all worked perfectly (the verification cURL commands all produce the expected results):

• Installing NGINX Gateway Fabric (via manifests)
• Expose NGINX Gateway Fabric
• Routing traffic to your application
• Routing to Applications Using HTTP Matching Conditions

My question relates to the next How-To guide, Securing Traffic using Let's Encrypt and Cert-Manager, which I have not had such luck with. I am interested to find out what I may be able to do to get this part working.

Recreating the problem

I applied the manifests in the Securing Traffic How-To exactly as shown with the indicated necessary changes, and one minor change:

• I replaced the email address with my own valid email as indicated. It is an email I have registered with LetsEncrypt.
• I replaced the occurrences of cafe.example.com in the Gateway and HTTPRoute manifests with a valid domain that I control.
  I have a DNS record in place, and a load balancer on my firewall directing http/https to the two exposed NGINX Gateway Fabric service ports. I believe the external routing component of this is as it needs to be - For the environments set up in the previous guides, I was able to externally access the HTTP services running behind NGINX Gateway Fabric via the same DNS record.
• I made one change that is not discussed in the How-to guide: I replaced the acme server field with the ACME staging URL to avoid hitting their production server any more than necessary. I get the same results with the staging URL as I do with the production URL.

The problem I'm trying to resolve appears halfway through the guide, immediately after the Gateway object has been created. The guide indicates that kubectl get secret cafe-secret should give a result, but in my case, I get a NotFound error.

Observations

I located some errors and diagnostic output that may be relevant, mainly in the Gateway object:

• status.listeners.conditions has multiple occurrences of Message: tls.certificateRefs[0]: Invalid value: default/cafe-secret: secret does not exist
  • Following up on that, I ran kubectl get secret:

NAME                TYPE     DATA   AGE
cafe-secret-xs9x2   Opaque   1      54m

• I tried putting that Secret name into the Gateway declaration, but got the same result (the invalid value error was still present but with the new name)
• There is also a warning under events that may be worth noting. It relates to the HTTP listener:

Events:
  Type     Reason             Age                From                       Message
  ----     ------             ----               ----                       -------
  Normal   CreateCertificate  13m                cert-manager-gateway-shim  Successfully created Certificate "cafe-secret"
  Warning  BadConfig          13m (x9 over 13m)  cert-manager-gateway-shim  Skipped a listener block: [spec.listeners[0].hostname: Required value: the hostname cannot be empty, spec.listeners[0].tls: Required value: the TLS block cannot be empty]

I am able to get rid of the hostname warning by adding the name declaration to that part of the manifest, but I have not been able to resolve the error regarding the TLS block.

I've included the full output of kubectl describe cluster issuer letsencrypt-prod and 'kubectl describe gateway gateway' as .txt attachments for reference.
describe.clusterissuer.txt
describe.gateway.txt

Thank you in advance for any help!

[ciarams87]

Hi @Curtingham, thanks for your interest in this project!!

1. The warning message is coming from cert-manager and can be ignored - I believe it will be resolved in the next version of cert-manager. See Securing Gateway resources with non HTTPS listeners generate BadConfig events cert-manager/cert-manager#6197.

2. It looks here that the ACME challenge is likely failing, meaning the Secret is never created - so therefore is showing as missing here in the status (worth noting that this error will appear at some point as the Gateway is created before the Secret - this is normal).

My guess is that traffic from LetsEncrypt server cannot reach the Challenge service created by cert-manager.

Have you tried going through the ACME challenge troubleshooting guide? In addition to the cert-manager troubleshooting guide, you can check the cert-manager pod logs, the challenge temporary HTTPRoute resource status, and the NGF NGINX logs for clues about where the issue might be occurring:

kubectl logs <cert-manager-pod> -n cert-manager

kubectl get httproutes
<...>
kubectl describe httproutes cm-acme-http-solverxxxxx

kubectl logs <ngf-pod-name> -n nginx-gateway -c nginx

[Curtingham]

Thank you so much @ciarams87 - I'm still working through it, but your response got me back on the right track and I have a better understanding of what's going on. I hadn't investigated the cm-acme-http-solver service closely enough to realize it listens on a separate port to the normal HTTP listener for NGINX Gateway Fabric. That alternate port is not forwarded, which is why the ACME challenge is failing.

That does lead into a more general question I have: What is a typical / recommended port forwarding configuration of the public endpoint that serves as a frontend to NGF? In my case, that public endpoint is HAProxy, and I had so far only made forwarding rules for the one HTTP and one HTTPS port that get bound via a NodePort service to NGINX Gateway Fabric. (I have it load balancing between the one control plane host and the two worker nodes in my Kubernetes cluster)

I had guessed (but haven't yet confirmed) that those two NGF service ports would be somewhat fixed for the lifetime of the gateway object. In the case of cert-manager though, the challenge service will be temporary, and so forwarding a single port will only work for one certificate issuance at best.

[Curtingham]

Thanks again @ciarams87. After working through it further and following the prompts you gave, I was able to get this working. My fix is a bit of a sidestep: I switched to using the DNS01 challenge mode, avoiding the need to port forward for the HTTP01 challenge mode at all. It took some other minor tweaks to the how-to guide, but I now have a valid (Staging) certificate and a properly working HTTPS example service running behind NGF. Thanks for pointing me in the right direction!

I'm still curious about what's typically recommended for port forwarding from the Public Endpoint to NGINX Gateway Fabric in production environments - I'm not yet running anything critical behind it, but I am building towards doing so. The holdup is just my own understanding of it all, so I'm excited to learn more about it.

Thanks to everyone involved in this project. I'm looking forward to using it more!

[ciarams87]

That's great you got it working!!

I'm still not really sure what the issue you were facing with the HTTP01 challenge was though. The cm-acme-http-solver service is fronted by a temporary HTTPRoute which proxies the challenge traffic through NGF (as configured in the Issuer). The challenge traffic should work on the HTTP port you have configured (assuming you have configured all public traffic coming in on port 80 to hit the HTTP port in NGF) without any extra configuration. Having said that, I've never tested NodePort + HAProxy setup, I normally use a LoadBalancer service, with a cloud provider LB, fronting NGF for my production environment setups.

Thanks again for your interest in our project, we're really excited to see people using it more!