Higher demands on resources

[hufhend]

Versions

• Pi-hole: v6.0.4
• AdminLTE: v6.0.1
• FTL: v6.0.2

Platform

• OS and version: Armbian Linux 6.6.75-current-sunxi64

• Platform: Orange Pi Zero2 (1GB)

Expected behavior

Hi, in the latest version you've stretched the system resource requirements a bit and although it looks great, I can no longer fit a Pi-hole on one OrangePi along with Prometheus monitoring. Which used to be my favorite setup.

Actual behavior / bug

2025-02-25 23:00:02.093 CET [53/T189] ERROR: Cannot receive UDP DNS reply: Timeout - no response from upstream DNS server

Steps to reproduce

Steps to reproduce the behavior:

We will install Orange Pi Zero2 and Armbian. Deploy Pihole via Little-IaC (Ansible) in the following way:
ansible-playbook -l orange orangepi.yaml

where the orange file will contain this setup:

# keepalived: false
orange: true
firewall: true
mntd: "14"

# Deploy conternerized applications
gitea: false
pihole: true
prometheus: true
torproject: false
openvpn: true

Debug Token

• URL: https://tricorder.pi-hole.net/N5dv2a0b/

Screenshots

Additional context

Until Sunday it worked perfectly, today they don't fit together. The new version was only on Pi-hole.

[disnetclub]

Same problem
Host: Synology docker

[hufhend]

I'm attaching the monitoring, you can see it quite well here

[cliles]

I ended up rolling back to 2024.07.0 for stability.

[hufhend]

Confirmed, reverting to v2024.07.0 returns stability, but the new image must be deleted completely.

[PromoFaux]

Initial questions that come to mind:

• Are these observances from the Pi-hole container being on but otherwise sitting idle? (i.e just responding to DNS queries)
• Do you have the dashboard open for long periods of time while observing these issues? Do they go away if you leave the dashboard closed for a while?
  • Are you visiting the dashboard on port 80 or 443 (or, otherwise - are you using plain HTTP or HTTPS)?

[PromoFaux]

Supplementary - do you still see this same behaviour if you set the config value webserver.threads to a higher number - Up to 64, but anything above 16)

I've just tagged 2025.02.7 which includes the latest FTL patches , should be live once this finishes: https://github.com/pi-hole/docker-pi-hole/actions/runs/13588931892

[hufhend]

I noticed the problem even when the www panel was disconnected, i.e. when it was only answering DNS queries.
Actually, I only started exploring the site the morning after the nightly update, I usually run it as a service.
I don't think it's related to the web interface.

Otherwise, for me it's running on a reverse proxy, so only port 80 is coming from the Pi-hole

[hufhend]

I have deployed the latest version and so far it seems to be almost the same

[PromoFaux]

Would you be willing to try another image? You'll need to build it yourself locally.

git clone https://github.com/pi-hole/docker-pi-hole/
cd docker-pi-hole
./build.sh -f tweak/webserver_threads

This will create a local image named pihole:local, which you can use as a drop-in replacement for pihole/pihole:[tag] in your compose/run file.

I'm not 100% certain that you will see any improvements here, but would like to explore as many possibilities as possible.

Could you also tell me what monitoring software you are using? I'd like to try on my own machine

[hufhend]

I added it to Steps to Reproduction. I mean, if you take any Raspberry or VM with Ubuntu and run it, you get the same setup as I have here including Prometheus. That is, without the custom settings and data.

I have 6 Pi-holes in 3 locations, always master and slave, I'll try the local build on one and let you know how it turns out.

[hufhend]

So the build works nicely, but it's the same. I would guess that the problem would be somewhere in CORE or FTL.

I also noticed an interesting thing, on one site I reverted to v2024.07.0 and left the slave on the last one, there is no monitoring going on so it stalls. But the slave stopped getting queries and everything is being handled by the old version. I tried the new version and it is responding. But I guess it's slow. So nobody asks him.

[PromoFaux]

Core is basically the installation wrapper, the real workhorse here is FTL.

Can you maybe get some output from htop to show what processes/subprocesses are actually causing the CPU spikes?

[hufhend]

Looks like it's FTL

[PromoFaux]

If you look at it in Tree mode (F5), what do we see?

I'll admit I only half know what I'm looking for here. Tagging @DL6ER in case he can see any clues

[hufhend]

I didn't find anything interesting behind F5, but the docker stats pihole shows

And then there's this, but I don't really understand:

$ docker top pihole 
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                569574              569552              0                   21:07               ?                   00:00:00            /bin/bash /usr/bin/start.sh
root                569725              569574              0                   21:07               ?                   00:00:00            /usr/sbin/crond
hufhendr            569844              569574              0                   21:07               ?                   00:00:00            /bin/bash -c /usr/bin/pihole-FTL no-daemon >/dev/null
hufhendr            569847              569844              17                  21:07               ?                   00:10:31            /usr/bin/pihole-FTL no-daemon
root                570404              569574              0                   21:07               ?                   00:00:00            tail --follow=name -n +2 /var/log/pihole/FTL.log

[PromoFaux]

I guess what I was hoping to see was the subprocesses/threads of pihole-FTL (Sometimes I sound like I know what I'm talking about 😉)

Kinda a bit like this - except not as hard to read

[hufhend]

Oh, I guess this is it:

[rdwebdesign]

Yes, but it will help if you set htop to show "custom thread names".

Press F2 to open settings page, and use the arrow key to select "Display options" and then select "Show custom thread names". Press F10 to return to the main screen:

[hufhend]

Interesting, if it's a database, then that would explain the high IO pressure

[PromoFaux]

There are a lot of web workers there, too - do you have something polling the API?

[hufhend]

No, I don't think I'm using anything through the API. I just normally have it open in one window and see what it does.

But is the nginx proxy in front of it because of Let's Encrypt SSL, maybe those queries are breaking up somehow?

[4-FLOSS-Free-Libre-Open-Source-Software]

also observing database thread constantly high ressource demand.

i was required to set

  shm_size: "2G"

because:

RAM shortage (/dev/shm) ahead: 99% used
/dev/shm: 268.1MB used, 268.4MB total, FTL uses 268.1MB
Set the shared memory size to 1G
Resizing "FTL-queries" from 1007648768 to (17997824 * 56) == 1007878144 (/dev/shm: 1.0GB used, 1.1GB total, FTL uses 1.0GB)
WARNING: RAM shortage (/dev/shm) ahead: 93% is used (/dev/shm: 1.0GB used, 1.1GB total, FTL uses 1.0GB)
add_message(type=7, message=/dev/shm) - SQL error step DELETE: database is locked
Error while trying to close database: database is locked
Resizing "FTL-queries" from 1007878144 to (18001920 * 56) == 1008107520 (/dev/shm: 1.0GB used, 1.1GB total, FTL uses 1.0GB)
WARNING: RAM shortage (/dev/shm) ahead: 93% is used (/dev/shm: 1.0GB used, 1.1GB total, FTL uses 1.0GB)

[PREngineer]

This is running in Container Manager (Docker) in a Synology NAS.

RAM limited to 1 GB and no limits on CPU (8 accessible).

Why is it using 300%+ CPU?

[4-FLOSS-Free-Libre-Open-Source-Software]

Why is it using 300%+ CPU?

Its Status system usage, not pi hole alone.

[PREngineer]

Why is it using 300%+ CPU?

Its Status system usage, not pi hole alone.

Thanks for getting back to me. At this point, Pi-Hole v6 is unusable for me. I will uninstall and try it again once it matures and solves the current problems.

[hufhend]

I did a downgrade to 2024.07.0 on the master DNS and stayed on 2025.03.0 on the slave. OrangePi for slaves are less busy (less other applications), anyway I observe that most queries are handled by older Pi-holes.