DJango container fails silently - Cannot deploy

[darkbox]

I'm not able to deploy the full stack of geonode. Django container keeps restarting in a loop, I guess because uwsgi fails silently, but there is nothing in the logs to point out what is the problem.

Here is my stack file (Docker Swarm):

# SWARM STACK

x-common-django:
  &default-common-django
  image: geonode/geonode:4.4.1
  env_file:
      - stack.env
  volumes:
    - statics:/mnt/volumes/statics
    - geoserver-data-dir:/geoserver_data/data
    - backup-restore:/backup_restore
    - data:/data
    - tmp:/tmp
  depends_on:
    - db
  deploy:
    replicas: 1

services:

  django:
    << : *default-common-django
    environment:
      - IS_CELERY=False
    entrypoint: ["/usr/src/geonode/entrypoint.sh"]
    command: "uwsgi --ini /usr/src/geonode/uwsgi.ini"
    #command: "python3 manage.py runserver 0.0.0.0:8000"  # Avoid uWSGI for now
    healthcheck:
      test: ["CMD", "curl", "-f", "-s", "http://localhost:8000/"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 10s
    networks:
      - geonode-network

  celery:
    << : *default-common-django
    environment:
      - IS_CELERY=True
    entrypoint: ["/usr/src/geonode/entrypoint.sh"]
    command: "celery-cmd"
    deploy:
      replicas: 1
    networks:
      - geonode-network

  memcached:
    image: memcached:alpine
    env_file:
      - stack.env
    command: memcached ${MEMCACHED_OPTIONS}
    healthcheck:
      test: ["CMD", "nc", "-z", "127.0.0.1", "11211"]
      interval: 30s
      timeout: 30s
      retries: 5
    deploy:
      replicas: 1
    networks:
      - geonode-network

  geonode:
    image: geonode/nginx:1.25.3-latest
    env_file:
      - stack.env
    environment:
      - RESOLVER=127.0.0.11
    ports:
      - ${HTTP_PORT-3080}:80
      - ${HTTPS_PORT-3443}:443
    volumes:
      - nginx-confd:/etc/nginx
      - nginx-certificates:/geonode-certificates
      - statics:/mnt/volumes/statics
    deploy:
      replicas: 1
    networks:
      - geonode-network

  letsencrypt:
    image: geonode/letsencrypt:2.6.0-latest
    env_file:
      - stack.env
    volumes:
      - nginx-certificates:/geonode-certificates
    deploy:
      replicas: 1
    networks:
      - geonode-network

  geoserver:
    image: geonode/geoserver:2.24.3-latest
    env_file:
      - stack.env
    ports:
      - "${GEOSERVER_LB_PORT:-8088}:8080"
    volumes:
      - statics:/mnt/volumes/statics
      - geoserver-data-dir:/geoserver_data/data
      - backup-restore:/backup_restore
      - data:/data
      - tmp:/tmp
    depends_on:
      - data-dir-conf
      - django
    healthcheck:
      test: ["CMD", "curl", "-m", "10", "--fail", "--silent", "--write-out", "HTTP CODE : %{http_code}\n", "--output", "/dev/null", "http://localhost:8080/geoserver/ows"]
      interval: 60s
      timeout: 10s
      retries: 2
    deploy:
      replicas: 1
    networks:
      - geonode-network

  data-dir-conf:
    image: geonode/geoserver_data:2.24.3-latest
    entrypoint: sleep infinity
    volumes:
      - geoserver-data-dir:/geoserver_data/data
    healthcheck:
      test: "ls -A '/geoserver_data/data' | wc -l"
    deploy:
      replicas: 1
    networks:
      - geonode-network

  db:
    image: geonode/postgis:15.3-latest
    command: postgres -c "max_connections=200"
    env_file:
      - stack.env
    volumes:
      - dbdata:/var/lib/postgresql/data
      - dbbackups:/pg_backups
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d postgres -U postgres"]
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    networks:
      - geonode-network

  rabbitmq:
    image: rabbitmq:3-alpine
    volumes:
      - rabbitmq:/var/lib/rabbitmq
    deploy:
      replicas: 1
    networks:
      - geonode-network

networks:
  geonode-network:
    driver: overlay

volumes:
  statics:
    driver: local
  nginx-confd:
    driver: local
  nginx-certificates:
    driver: local
  geoserver-data-dir:
    driver: local
  dbdata:
    driver: local
  dbbackups:
    driver: local
  backup-restore:
    driver: local
  data:
    driver: local
  tmp:
    driver: local
  rabbitmq:
    driver: local

And here are the last lines in django container logs (No migration errors, etc):

....
Found another file with the destination path 'lib/js/plugins/anchor/plugin.min.js'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.

Found another file with the destination path 'mapstorestyle/img/s2cloudless-s2cloudless.png'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.

0 static files copied to '/mnt/volumes/statics/static', 5009 unmodified.

statics tasks done

Executing UWSGI server uwsgi --ini /usr/src/geonode/uwsgi.ini for Production

-----------------------------------------------------

FINISHED DJANGO ENTRYPOINT --------------------------

-----------------------------------------------------

got command uwsgi --ini /usr/src/geonode/uwsgi.ini

[uWSGI] getting INI configuration from /usr/src/geonode/uwsgi.ini

The rest of containers are up and running and nginx returns 502 since uwsgi does not start? If I change uwsgi with python dev server geonode starts and do not reset, but the frontend has issues and obviously using the python server is not a good idea, but shows that must be something wrong with uwsgi.

[darkbox]

Update: I manage to start geonode with a hack. But I don't understand why I have to do this.

First, I modify the stack file django container:

django:
    << : *default-common-django
    environment:
      - IS_CELERY=False
    entrypoint: ["/usr/src/geonode/entrypoint.sh"]
    #command: "uwsgi --ini /usr/src/geonode/uwsgi.ini"
    command: "python3 manage.py runserver 0.0.0.0:8010" # <--- Add this dummy process to avoid the container to restart
    healthcheck:
      test: ["CMD", "curl", "-f", "-s", "http://localhost:8010/"] # <-- just so is marked as healthy
      interval: 30s
      timeout: 30s
      retries: 5
      start_period: 30s
    networks:
      - geonode-network

Start the stack, wait till is up and running, once is done open a shell and execute manually the command to start the web server uwsgi --ini /usr/src/geonode/uwsgi.ini and it works!

So if this works, why I need to do this hacky thing? The problem with this "solution" is that I have to manually intervene.

[mattiagiupponi]

is there something present in the geonode log? /var/log/geonode.log before the container dies?

[darkbox]

The last time I checked was empty (It's tricky since it restarts by itself). Even though may not be useful, this is what I get when starts manually:

[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
*** Starting uWSGI 2.0.25.1 (64bit) on [Mon Mar  3 13:15:24 2025] ***
compiled with version: 11.4.0 on 29 April 2024 10:58:00
os: Linux-6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec  5 13:09:44 UTC 2024
nodename: a49b2c7f9341
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /usr/src/geonode
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
chdir() to /usr/src/geonode/
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: enabled
[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
[busyness] backlog alert is set to 16 request(s), step is 2
[busyness] backlog non-zero alert is set to 60 second(s)
uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
Python main interpreter initialized at 0x6199b701a390
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 13105368 bytes (12798 KB) for 128 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x6199b701a390 pid: 337 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
spawned uWSGI master process (pid: 337)
spawned uWSGI worker 1 (pid: 347, cores: 1)
spawned uWSGI worker 2 (pid: 348, cores: 1)
spawned uWSGI worker 3 (pid: 349, cores: 1)
spawned uWSGI worker 4 (pid: 350, cores: 1)
spawned uWSGI worker 5 (pid: 351, cores: 1)
spawned uWSGI worker 6 (pid: 352, cores: 1)
spawned uWSGI worker 7 (pid: 353, cores: 1)
spawned uWSGI worker 8 (pid: 354, cores: 1)
spawned uWSGI worker 9 (pid: 355, cores: 1)
spawned uWSGI worker 10 (pid: 356, cores: 1)
spawned uWSGI worker 11 (pid: 357, cores: 1)
spawned uWSGI worker 12 (pid: 358, cores: 1)
spawned uWSGI worker 13 (pid: 359, cores: 1)
spawned uWSGI worker 14 (pid: 360, cores: 1)
spawned uWSGI worker 15 (pid: 361, cores: 1)
spawned uWSGI worker 16 (pid: 362, cores: 1)
[pid: 347|app: 0|req: 1/1] 10.0.15.4 () {52 vars in 1365 bytes} [Mon Mar  3 13:15:32 2025] GET / => generated 54494 bytes in 143 msecs (HTTP/1.0 200) 11 headers in 627 bytes (1 switches on core 0)
[pid: 348|app: 0|req: 1/2] 10.0.15.4 () {52 vars in 1385 bytes} [Mon Mar  3 13:15:34 2025] GET /api/v2/ => generated 984 bytes in 49 msecs (HTTP/1.0 200) 10 headers in 371 byt
s (1 switches on core 0)
Unauthorized: /api/o/v4/userinfo
[pid: 349|app: 0|req: 1/3] 10.0.15.4 () {52 vars in 1405 bytes} [Mon Mar  3 13:15:34 2025] GET /api/o/v4/userinfo => generated 88 bytes in 30 msecs (HTTP/1.0 401) 9 headers in 345 bytes (1 switches on core 0)
[pid: 350|app: 0|req: 1/4] 10.0.15.4 () {52 vars in 1540 bytes} [Mon Mar  3 13:15:35 2025] GET /api/v2/resources?page_size=4&page=1&filter%7Bfeatured%7D=true&api_preset=catalog_list => generated 87 bytes in 92 msecs (HTTP/1.0 200) 10 headers in 391 bytes (1 switches on core 0)
[pid: 351|app: 0|req: 1/5] 10.0.15.4 () {52 vars in 1554 bytes} [Mon Mar  3 13:15:35 2025] GET /api/v2/resources?api_preset=catalog_list&filter%7Bmetadata_only%7D=false&page=1&page_size=24 => generated 88 bytes in 87 msecs (HTTP/1.0 200) 10 headers in 391 bytes (1 switches on core 0)
[busyness] 1s average busyness is at 0%, cheap one of 16 running workers
[pid: 352|app: 0|req: 1/6] 10.0.15.4 () {48 vars in 1312 bytes} [Mon Mar  3 13:16:04 2025] GET / => generated 54494 bytes in 136 msecs (HTTP/1.0 200) 10 headers in 482 bytes (1 switches on core 0)
[pid: 353|app: 0|req: 1/7] 10.0.15.4 () {48 vars in 1332 bytes} [Mon Mar  3 13:16:05 2025] GET /api/v2/ => generated 984 bytes in 44 msecs (HTTP/1.0 200) 10 headers in 371 bytes (1 switches on core 0)
Unauthorized: /api/o/v4/userinfo
[pid: 354|app: 0|req: 1/8] 10.0.15.4 () {48 vars in 1352 bytes} [Mon Mar  3 13:16:05 2025] GET /api/o/v4/userinfo => generated 88 bytes in 29 msecs (HTTP/1.0 401) 9 headers in 345 bytes (1 switches on core 0)
[pid: 355|app: 0|req: 1/9] 10.0.15.4 () {48 vars in 1487 bytes} [Mon Mar  3 13:16:06 2025] GET /api/v2/resources?page_size=4&page=1&filter%7Bfeatured%7D=true&api_preset=catalog_list => generated 87 bytes in 89 msecs (HTTP/1.0 200) 10 headers in 391 bytes (1 switches on core 0)
[pid: 356|app: 0|req: 1/10] 10.0.15.4 () {48 vars in 1501 bytes} [Mon Mar  3 13:16:06 2025] GET /api/v2/resources?api_preset=catalog_list&filter%7Bmetadata_only%7D=false&page=1&page_size=24 => generated 88 bytes in 89 msecs (HTTP/1.0 200) 10 headers in 391 bytes (1 switches on core 0)
[busyness] 1s average busyness is at 0%, cheap one of 15 running workers