Nginx container needs allowPrivilegeEscalation: true otherwise OPA is throwing an error, marking allowPrivilegeEscalation: false will result in unix:/var/run/nginx/nginx-config-version.sock failed (98: Address in use), 0.0.0.0:443 failed (13: Permission denied) in Nginx container #1957

[sja-commit]

As I discussed before I tried to recreate the nginx-gateway without adding any extra security in it but OPA gatekeeper throwing an error on nginx container.

I can see there are 2 containers in the nginx gateway installation one is nginx-gateway and another is nginx.

nginx-gateway container has below security context in the yaml manifest, so I understand by default this container doesn't require privileges.
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:

• KILL
  drop:
• ALL
  But Nginx container has below security context which doesn't specify the privilege as false.
  securityContext:
  capabilities:
  add:
• NET_BIND_SERVICE
  drop:
• ALL

When I install using this setup my OPA gatekeeper is throwing error "Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [Policy name] Privilege escalation container is not allowed: nginx
So I tried to add allowPrivilegeEscalation: false in the nginx container security context as well, which works and pods get created but problem starts when I create gateway with ssl termination and http route. Below is the sample gateway and http route I tried and I am getting error in nginx reload:
2024/05/13 08:52:29 [notice] 20#20: signal 1 (SIGHUP) received from 7, reconfiguring
2024/05/13 08:52:29 [notice] 20#20: reconfiguring
2024/05/13 08:52:29 [emerg] 20#20: bind() to unix:/var/run/nginx/nginx-config-version.sock failed (98: Address in use)
2024/05/13 08:52:29 [emerg] 20#20: bind() to unix:/var/lib/nginx/nginx-502-server.sock failed (98: Address in use)
2024/05/13 08:52:29 [emerg] 20#20: bind() to unix:/var/lib/nginx/nginx-500-server.sock failed (98: Address in use)
2024/05/13 08:52:29 [emerg] 20#20: bind() to 0.0.0.0:443 failed (13: Permission denied)

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: demo
namespace: demo
spec:
gatewayClassName: nginx
listeners:

name: https
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
kind: Secret
name: demo-tls
namespace: demo
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: demo
namespace: demo
spec:
parentRefs:

name: demo
sectionName: https
hostnames:
"example.com"
rules:
matches:
path:
type: PathPrefix
value: /
backendRefs:
name: demo #service name
port: 8080
Does nginx requires privilege container access?

[sjberman]

Privilege escalation is required by nginx to bind to ports under 1024, since these are considered privileged ports.

[sja-commit]

Correct me if I am wrong, As per my understanding nginx master process needs root access in order to bind the ports below 1024 but I can see nginx master process in ingress controller is running as non root user. In Nginx gateway config worker_connections 1024 but in nginx ingress config worker_connections uses 16384 (higher order) and also user nginx. I am not getting privileged container OPA gatekeeper error or port binding permission denied error on nginx ingress. Moreover in the nginx gateway deployment I can see the net bind service capability is added so why do master process require root access? If my understanding is correct in a nutshell both in nginx ingress and nginx gateway , nginx container does similar job. Please assist.

[brianehlert]

Within the container NGINX is running (or should be running as non-root) per best practice.

The Gateway deployment itself is being exposed on port 80 or 443 at the cluster will require allowPrivilegeEscalation: true because those are in the reserved port range of 1024 or lower.
This is the same for ingress controllers.

The way to have allowPrivilegeEscalation: false is to assign ports above 1024 and port forward at the load balancer layer.

[sjberman]

I'll also note that worker_connections have no relation to port number.

[brianehlert]

To add - worker_connections is a performance tuning value that we generally do not recommend be altered.
The default is recommended for nearly all cases. Sets the maximum number of simultaneous connections that can be opened by a worker process.

The NGINX Ingress Controller project from NGINX Inc does run as non-root by default. It runs under the nginx user.

[brianehlert]

In the original post I see: (98: Address in use)
This means that port 443 is already bound to another service in the cluster.
And thus the Gateway deployment is unable to bind to that port.

[sjberman]

This actually may be related to #1108. The same port can be used for multiple services in the cluster. It's some issue within the container itself.

[brianehlert]

Ah. That would be different indeed.

[sja-commit]

I use this template nginx-gateway-fabric/templates/service.yaml in order to create load balancer service. So as per my understanding 443 port binds with nginx-gateway load balancer service. I am not getting any error. But when I am creating application service and exposing it using gateway and http route and use the same load balancer IP, nginx reloads the config. This is where the permission denied issue starts. Now I am expecting nginx to use nginx user and create new worker process for the modified config.

[sjberman]

@sja-commit nginx is reloaded and configured with the ports defined in the Gateway resource. The service ports live only on the service and do not affect nginx configuration. They just need to have their targetPorts mapped to where the NGF pod has its ports exposed, which can be changed in the NGF deployment and Gateway resource to whatever you want them to be.