SocketException permission denied when trying to bind port 80 or 443 to Kestrel in Azure Container Instance with .NET 8

[Piedone]

I only see this in Linux Azure Container Instances (ACI) but I figured it's more appropriate to ask here instead of the Azure forums since the issue only appeared after upgrading the app to .NET 8; the exact same code works targeting .NET 6 (i.e. if I only change <TargetFramework> in the csproj and the publish profile).

I get the following exception when launching a .NET 8 ASP.NET Core app in an ACI:

fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.Net.Sockets.SocketException (13): Permission denied
         at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
         at System.Net.Sockets.Socket.Bind(EndPoint localEP)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.EndpointsStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
         at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
         at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
Unhandled exception. System.Net.Sockets.SocketException (13): Permission denied
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass28_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.EndpointsStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Program.<Main>$(String[] args) in E:\MyProject.Web\Program.cs:line 42

This happens if I try to make Kestrel listen on ports 80, 443, or both. I don't get any more information with Trace-level logging.

This is what I tried:

• Alternative configuration options for Kestrel, but it doesn't matter how I configure this, be it in appsettings.json under Kestrel__Endpoints, the ASPNETCORE_HTTP_PORTS and ASPNETCORE_HTTPS_PORTS environment variables, or with UseKestrel().
• Making Kestrel listen under the internal IP of the container, not any IP, but that yields the same result.
• Using different ports, like 8080 and 8443. That does work, but since I need this web app to be available under standard HTTP and HTTPS, 80 and 443 should be the ones publicly exposed. And I can't do that while using a different port internally, since ACI doesn't support port mapping.
• I'm aware of the default port having changed from 80 to 8080 with .NET 8, however, that should be addressed by setting the port to 80 explicitly as under the first point, nor should I affect port 443.
• ACI seems to reserve port 443 but neither should that affect port 80, nor should port 443 work with .NET 6. So, this is a .NET-related issue, not this port reservation.
• Using --privileged with az container create (as well as --allow-escalation) but nothing changed.

I can't reproduce this locally under Windows.

The container image is published self-contained.

Can somebody help me understand what I may be doing wrong? Is this perhaps an undocumented breaking change, in .NET 8 or how ACI supports .NET 8?

Update: cross-posted this to Stack Overflow since nobody replied here for a week.

Update 2: I cross-posted to Microsoft Q&A for visibility.

[elruss]

You can explicitly configure Kestrel to bind to port 80 during the WebApplication build at startup:

services.Configure<KestrelServerOptions>(options => { options.Listen(System.Net.IPAddress.Any, 80); // Listen on Port 80 });

In Container Apps, I found that all of ASPNETCORE_HTTP_PORTS, the ingress port, and the probe ports have to be aligned. If explicitly binding during build, environment variables port values are ignored.

Given what I understand, I'm surprised setting port 80 in the environment did not work for you. Are you able to console to the container to confirm that value?

[Piedone]

Thanks for your reply! I believe I tried that as well, and the below appsettings too, which is functionally the same:

{
  "Kestrel": {
    "Endpoints": {
      "HTTP catch-all": {
        "Url": "http://*"
      }
    }
    }
  }
}

What exactly do you mean by "Are you able to console to the container to confirm that value?"? What would I check in the console? However, I did try to use the console from the Azure Portal previously, and due to this issue making the startup of the app fail, and thus causing the container to continuously restart, I couldn't really (perhaps there are a few seconds one can catch during this when it works, or I could just wrap app.Run() into a try-catch to swallow the exception).

[elruss]

Confirm that ASPNETCORE_HTTP_PORTS is set to 80 by running env from the container console.

Check for any readiness/liveliness probes, too. If for some reason, ACI is detecting the 8080 binding, but not updating the probes (i.e. they are explicitly configured to health check on port 80), they will continually recycle the container. Could be vice versa, too...ACI is not detecting the explicit port 80 binding and the probes are still checking 8080.

[Piedone]

Thanks for the tip! Yes, the env vars are definitely set. Although I can't run env when the container start fails like this, adding ASPNETCORE_HTTP_PORTS and/or ASPNETCORE_HTTPS_PORTS to a previously starting app that doesn't have any configured endpoints both result in a SocketException of the above kinds (with messages like "Failed to bind to http://[::]:80 (IPv6Any). Attempting to bind to http://0.0.0.0:80 instead.").

[Piedone]

I dug some more and found that actually this might be the breaking change relevant to this issue, not the default port change: New non-root 'app' user in Linux images. Also see the blogpost it links. And sure enough, with .NET 6 the container's user is root, and with .NET 8 it's app.

I tried

az container create `
    --run-as-group 0 `
    --run-as-user 0 `
...

to run the container as root but it still runs as app. Adding USER root in the Dockerfile doesn't do anything either because the Dockerfile has no effect for Azure Container Instance, they use az container create as a substitute. I also tried setting ContainerUser with <ContainerUser>root</ContainerUser> but no luck.

I cross-posted to Microsoft Q&A for visibility.

[Piedone]

Found the solution. It was actually setting ContainerUser with <ContainerUser>root</ContainerUser> in the csproj. No other changes necessary.

So, all in all, compared to the .NET 6 version, my app has the following changes to target .NET 8:

• <TargetFramework>net8.0</TargetFramework> in the csproj.
• <TargetFramework>net8.0</TargetFramework> in the pubxml used by az container create.
• <ContainerUser>root</ContainerUser> in the csproj, see below the context.

Full section of the csproj:

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <UserSecretsId>MyId</UserSecretsId>
    <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
    <NoDefaultLaunchSettingsFile>true</NoDefaultLaunchSettingsFile>
    <EnableSdkContainerSupport>true</EnableSdkContainerSupport>
    <ContainerUser>root</ContainerUser>
  </PropertyGroup>

I did try this before, but apparently I messed up something during publishing, but now it's clear.

I also opened an issue to clarify this in the docs: dotnet/docs#39082.

Thanks for your help @elruss!

[dotfelixb]

isn't this dangerous running the app with root access?

[Piedone]

I guess theoretically it adds a risk, but there is no other way if you use Azure Container Instances, and you can't put a reverse proxy in front of your container (what's the case if your container is your reverse proxy, like in our case).

[bennean]

Is this really the best solution? I thought it was bad practice to run containers as the root user?

[Piedone]

As an Azure Container Instance, with ports 80 and 443?

[bennean]

Whoops, my bad, apologies, I checked and they run on port 44380 and we use nginx to redirect.

[Piedone]

No worries.

[ankense-cariad]

Whoops, my bad, apologies, I checked and they run on port 44380 and we use nginx to redirect.

@bennean How do you achieve this with nginx using the new app user instead of root inside these containers?

[bennean]

@ankense-cariad We use USER $APP_UID in the Dockerfile, as per https://devblogs.microsoft.com/dotnet/securing-containers-with-rootless/

[Amondale]

This is not just an Azure Container issue; Bare Metal Linux (haven't tried many flavors besides Ubuntu) also disallows running non-root on 443/80. Not a big deal though, as mentioned above :44300 or :8080 e.g. both work fine.

[josephrodriguez]

@Piedone I was able to run .NET 8 Web API containerized application, all is related to breaking change to use non-root user as you suspected.

https://devblogs.microsoft.com/dotnet/running-nonroot-kubernetes-with-dotnet/

In Dockerfile just expose the non-root port for HTTP and HTTPS:

EXPOSE 8080
EXPOSE 8443

And target such ports from the Service in Kubernetes.

[Piedone]

Great!