Pi drops IPv4 address after DHCP lease expires

[geerlingguy]

After updating to the latest versions of Pi OS, Docker, etc. sometime in the last week or two, my Internet Pi seems to entirely drop its wired network connection after my network DHCP lease is up (literally 24h after boot). The only way to recover is a hard reboot (since this is a headless Pi).

In the syslog, I see:

Aug  6 01:34:42 geerli avahi-daemon[398]: Withdrawing address record for 10.0.100.3 on eth0.
Aug  6 01:34:42 geerli avahi-daemon[398]: Leaving mDNS multicast group on interface eth0.IPv4 with address 10.0.100.3.
Aug  6 01:34:42 geerli avahi-daemon[398]: Interface eth0.IPv4 no longer relevant for mDNS.
Aug  6 01:34:56 geerli systemd[1]: run-docker-runtime\x2drunc-moby-dffd0eb098dcc0dbd442b74566b64c4452df98d1675a513a7951463342bc256d-runc.1YatQ6.mount: Succeeded.
Aug  6 01:35:05 geerli dockerd[740]: time="2024-08-06T01:35:05.269649369-05:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:43159" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.1:43159->127.0.0.1:53: i/o timeout" question=";raw.githubusercontent.com.\tIN\t AAAA"
Aug  6 01:35:05 geerli dockerd[740]: time="2024-08-06T01:35:05.276761116-05:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:60157" dns-server="udp:127.0.0.1:53" error="read udp 127.0.0.1:60157->127.0.0.1:53: i/o timeout" question=";grafana.com.\tIN\t AAAA"

Initially I thought the network stack could be buggy, but after reading through Every 24h, network connectivity ceases, I'm beginning to think Docker's at fault, considering the interesting container networking layout that makes Internet Pi work...

I'll try to debug the problem in this issue, in case anyone else runs into it. First step, will try running a full upgrade today.

[geerlingguy]

No updates required (I'm on Bullseye on this machine still... need to upgrade it entirely. Maybe migrate to Pi 5 too ha!). Maybe I can try manually forcing a DHCP renewal? Will try that later as I have other things to get to this morning.

[geerlingguy]

Ah... from Stack Overflow:

This is an issue with DHCPCD & Docker on Raspberry, as you already found by yourself. The full discussion is here: Raspberry Pi 4: DHCPCD route socket overflowed. Basically, dhcpcd can be flooded when renewing IP addresses if too many interfaces are present. This is the case when Docker is installed and many containers/networks/services are running. In addition, docker take care of IP addresses and routing on its virtual network, so DHCPCD doesn't need to handle them.

Supposedly the fix is to add the following to the end of /etc/dhcpcd.conf:

denyinterfaces veth*

Then restart dhcpcd:

sudo systemctl restart dhcpcd.service

...this wouldn't apply to Bookworm, so at some point I just need to bite the bullet and upgrade my main network Pi :)

I will try this solution and see if it works after 24h.

[geerlingguy]

Solution was originally posted here: raspberrypi/linux#4092 (comment)

And only found relevant info once I looked at journal entries for dhcpcd and found things like:

Aug 08 22:22:29 geerli systemd[1]: Started DHCP Client Daemon.
Aug 08 22:22:33 geerli dhcpcd[736]: veth7e35d83: waiting for carrier
Aug 08 22:22:33 geerli dhcpcd[736]: veth05adbb6: waiting for carrier
Aug 08 22:22:33 geerli dhcpcd[736]: veth979fbed: IAID 09:c2:c3:98
Aug 08 22:22:33 geerli dhcpcd[736]: veth979fbed: adding address fe80::2694:f5d8:6724:c67f
Aug 08 22:22:33 geerli dhcpcd[736]: vethc5ab19f: IAID a3:64:28:4f
Aug 08 22:22:33 geerli dhcpcd[736]: vethc5ab19f: adding address fe80::3c58:cb2b:a9c3:a81f
Aug 08 22:22:33 geerli dhcpcd[736]: veth490bc1f: waiting for carrier
Aug 08 22:22:33 geerli dhcpcd[736]: veth979fbed: soliciting a DHCP lease
Aug 08 22:22:33 geerli dhcpcd[736]: veth53e2894: IAID 5b:50:44:e8
Aug 08 22:22:33 geerli dhcpcd[736]: veth53e2894: adding address fe80::1044:d2b2:407f:6351
Aug 08 22:22:33 geerli dhcpcd[736]: veth809dacf: waiting for carrier
Aug 08 22:22:33 geerli dhcpcd[736]: veth4530865: IAID d9:ba:24:16
Aug 08 22:22:33 geerli dhcpcd[736]: veth4530865: adding address fe80::6c0c:7ea9:ed6:720c
Aug 08 22:22:33 geerli dhcpcd[736]: route socket overflowed - learning interface state

[github-actions]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

Please read this blog post to see the reasons why I mark issues as stale.