Server does not appear to open a listening port on Ubuntu 22.04 with Python 3.10.

[sxwebster]

I've set up Uvicorn on Ubuntu 22.04 and am getting the uvicorn process to start successfully using the below. It looks like Uvicorn isn't actually opening a local port to listen to, despite saying that it is, and ufw allowing port 8000.

Thoughts?

./execWxApi.sh

#contents of execWxApi.sh

#!/bin/sh

uvicorn app.wxapi:app --host 0.0.0.0 --reload

Output results showing port open on 8000

ubuntu@ip-172-1-1-1:~$ ./execWxApi.sh
INFO:     Will watch for changes in these directories: ['/home/ubuntu']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [5216] using StatReload
^CINFO:     Stopping reloader process [5216]

I'm not getting a response on port 8000. I've double checked firewalls, both in AWs and in ufw, and port 8000 is open to all addresses.

So, lets make sure that port 8000 is actually in use on the Ubuntu machine:

sudo netstate -tlnp

Output of netstat:

ubuntu@ip-172-1-1-1:~/app$ sudo netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      654/sshd: /usr/sbin 
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      403/systemd-resolve 
tcp6       0      0 :::22                   :::*                    LISTEN      654/sshd: /usr/sbin 

There's no port 8000 open. Huh?

#########################

I also tried a minimum viable install on Ubuntu 22.04 with Python 3.10 using the below, with the same results.

sudo apt update
sudo apt install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
[Enter]
sudo apt install python3.10
sudo apt install python3-pip to install pip
[Y]

Then

mkdir fastapi
cd fastapi
python3 -m venv env
source env/bin/activate
pip install fastapi uvicorn

Create main.py

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
    return {"message": "Hello World"}

Launch with

uvicorn main:app --reload

Loads successfully, green text indicating an open port on 0.0.0.0:8000.
netstat -a reveals no listenign port on 8000. There's no response on http://localhost:8000

###########

These examples are each on Amazon Web Services EC2 t3.small with Ubuntu 22.04 base image.

[sxwebster]

Additionally,

On this machine if I set up a very simply python script and execute it, I get a result in browser - confirming the machine is reachable, and ports are open all the way through.

test_server.py

import http.server
import socketserver

PORT = 8000

Handler = http.server.SimpleHTTPRequestHandler

with socketserver.TCPServer(("", PORT), Handler) as httpd:
    print("serving at port", PORT)
    httpd.serve_forever()

The execute with

python3 test_server.py

Then visit

http://my_internet_facing_ip:8000

[sxwebster]

Edit: Added additional content to demonstrate the machine is reachable and not firewalled on that port. Was able to use Python http server to deliver a simple directory via http.

[joonas-yoon]

I have been got in same trouble. In my case, when start simple server with python -m http.server <port>, the port is opened as well.
however when I try to run python -m uvicorn main:app --port <port>, nothing happened on ports.

FYI, I did run to see opened ports - netstat -tnlp.

I really hope anybody will resolve this problem..

[joonas-yoon]

and I have dependencies on fastapi==0.88.0 and uvicorn==0.20.0, on Ubuntu 18.04 LTS.

[sxwebster]

I found my answer:

What was happenign was that when I was using -reload the memory was being filled during script initialisation and the initialisation process was being killed. The killing of the process seemed to be silent, and application was still running, looking like it was working.

When I removed the reload I was able to see that during initialisation the process was exiting with a "killed message.

To work out what was happening I ran two SSH sessions side by side. One where I was using top to monitor memory usage, and the other to initialise and watch FastAPI and Uvicorn output on the CLI. I was able to see memory reaching maximum utilisation with top.

I hope this helps.

[joonas-yoon]

Thanks, I had been so struggled to find what causes error out.

As you said, something wrong is in script when initialize app.

...
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.schedulers.background import BackgroundScheduler

scheduler = BlockingScheduler()

app = FastAPI()

@app.on_event("startup")
async def startup():
    print('App started')
    scheduler.start() # Stuck in here by "blocking"

Thanks for your help :)

[sxwebster]

It looks like this issue might be related to the initialisation time of the script.
I have no idea why the second thing I tried has refused to work on the remote host at this stage.

My script attempts to load some data into memory with xarray so that it's sitting there at the time a query arrives and there's no querying unindexed data from disk. The principle issue in this case seems to be that when using the --reload flag that Uvicorn does not fail, it sits there and reports that the api endpoint is alive and well and the process is running. When the --reload flag is omitted the process takes 5-10 seconds to return a message, and that message is simply "Killed".

So, I believe it would be worth while understanding whether it's the memory limit on my instance that's the issue, or whether it's Uvicorn failing because the script is taking a while before it can accept a route due to pre-loading the data into indexed memory.

The challenge in identifying this is that I've had it running in a smaller container, and it hasn't had an issue.

[saper]

I believe it would be worth while understanding whether it's the memory limit on my instance

I read from the above that you are running it straight in the virtual machine (no containers involved), so you can check your ulimit -a as imposed by the shell and check the memory usage with something like vmstat 1. If it gets killed, a log entry should be generated (/var/log/messages or dmesg).

[saper]

We've had the same and it seems that uvicorn is really good at hiding things.

I believe this is caused by general inability to start the application. With --reload (big thanks to @sxwebster for the tip) it just tries to silently restart the app over and over and if it does not work, we do not get a listening socket open.

Without --reload it is easier to see why the application loading crashes, if this in a container - a whole container will stop and possibly restart.

I think this is a bad logging - why does it say it does listen on the port, when in reality it does not?