ZHA - application controller is not running #130548
Comments
|
Hey there @dmulcahey, @Adminiuga, @puddly, @TheJulianJES, mind taking a look at this issue as it has been labeled with an integration ( Code owner commandsCode owners of
(message by CodeOwnersMention) zha documentation |
erkr
changed the title
|
My ZHA network has gone down twice now with similar issues, also starting with core 2024.11.1. |
|
@Nerwyn also using a zbdongle-e?! |
|
Nope, using the SONOFF Zigbee 3.0 USB Dongle Plus-E Gateway with Silicon Labs Multiprotocol. |
|
@Nerwyn Multiprotocol isn't really a supported configuration anymore. I strongly suggest you migrate back to Zigbee firmware and explore alternative options for Thread border routers (if you use Thread). |
|
@puddly For some reason I see ZHA resetting the dongle as it is reported in fail state. That happens a few times per week. Additionally I noticed this issue reported here after upgrading to core-2024.11.1 (from core-2024.10.4). But I can't really tell it's related to new core version as I upgraded my dongle as well two weeks ago Best Eric |
|
on core-2024.11.2 also have a SONOFF ZB Dongle-P, have the same error as the OP |
|
@nayefalebrahim please attach a ZHA debug log |
|
I'm seeing the same with a S31 Lite zb by SONOFF. Here's the logs: Here's where it fails in the system logs with the Zigbee debug enabled: |
|
@christabone |
|
@puddly Thank you for the insight on that issue, I'll be sure to check the coordinator! EDIT: Moving it further away with an extension cable fixed the issue, thanks again. |
|
I have only misery with the ZBDongle-E. I now upgraded to the latest ITead version 7.4.4.0 build 0 and will report back if that makes a difference |
|
Hi @puddly I'm now testing with the updated Firmware (iTead 7.4.4) and ZHA ran for almost 2 days without errors or warnings. It was running again without reseating the dongle 🤞 |
|
@puddly This realy an issue in ZHA!
This was found in the log |
|
@erkr Enable debug logging for a few minutes and post the full log when ZHA is in this bad state. |
|
@puddly 👌 |
|
@puddly I added the log ending with disabling the debug logging. Some timestamps:
|
|
Also seeing this issue sometime after upgrading to 2024.11. I'm runnning home assistant in an LXC. I'll try and grab some logs. |
|
This is what I saw. Everything is great, and then suddenly I get a "no such device" error. Looking in the container the device is still there. |
|
@puddly I created an automation that notified me when the last seen's stop updating (logging: Integration not responsive: Check zha)
The log doesn't contain a single ZHA related logging! |
|
Two days ago, something slightly different happened. The symptoms were identical, except that there was 1 error logged:
|
|
I'm going to try and will back to an older version I think and see if this problem goes away. Otherwise perhaps it's time to upgrade my controller. |
|
Ive rolled back to 2024.10.3, I'll update later this week. Grasping at straws, but I wonder if its related to this change? The logic for a lost serial connection has changed. |
|
A log will immediately download when you click the "Disable debug logging" button. You can speed up the OTA checks by explicitly clicking the refresh button, it'll ask that all devices check in immediately: |
|
Got it - Update on "Livingroom Zigbee Switch 2" started where it had left off (around 20%) and ran to about 40ish % then crashed. Double checked if I could restart and gave me errors and failure - then I tried just toggling one of the other switches that was within ear shot to hear if it toggled (with up to date firmware) and thats where I get the error "Failed to perform the action switch/turn_off. Failed to send request: ApplicationController is not running" See log |
|
I was away for the holidays and Santa apparently gave me the same bug. Throwing: @ryanwinter I'm also on 2024.10.3 now |
|
Same issue here with sonof dongle P. I replaced this with a connect ZBT-1 and it’s still giving me the same issue. Logger: homeassistant.components.websocket_api.http.connection [281472362328800] Unexpected exception The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): |
|
Also getting the following in the logbook Logger: homeassistant.components.websocket_api.http.connection [281472362322608] Error handling message: Unknown error (unknown_error) Peter from 192.168.0.123 (Mozilla/5.0 (iPad; CPU OS 18_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.12.2 (io.robbie.HomeAssistant; build:2024.1058; iPadOS 18.2.0) Mobile/HomeAssistant, like Safari) |
|
Attached is a debug log what I also noticed is that when I unplug the controller and plug it back in or when I do a full restart of the home assistant green device the zigbee switches work for about 10 minutes after which none of my zigbee devices are working anymore until I do the same thing again. This makes home assistant useless for me at this point as none of my automations that I need are working anymore. |
|
@petepall Can you enable ZHA debug logging, let it run for however long it takes to get How are you running Home Assistant? Docker? A VM? |
Hi puddly, my Zhang’s is not recovering anymore until I disconnect and connect the device or do a full system reboot. |
|
I tested connecting the controller directly to the home assistant green (with extension cable) and using a powered usb hub. Issue happens in both cases. |
|
Here is a new log that spans from the device working to it getting disabled and working again. |
|
@Ithrial It looks like the firmware on your stick partially crashes during OTA and the stick refuses to acknowledge any data sent to it. At the same time, it isn't hearing back any acknowledgements from ZHA. It's as if the serial connection's TX line disconnects at some point. This is all only happening when there is a lot of data flowing back and forth (i.e. OTA + controlling a device at the same time). I also see that you're running a firmware build with hardware flow control disabled. Can you try an alternative firmware for it that's configured like the one we use for the SkyConnect? sonoff-e_zigbee_ncp_hw_flow_7.4.4.0.gbl.zip (make sure to unzip it first). You can flash it the same way you probably upgraded your stick originally (via the web flasher), just pick this GBL file. Once that's done, re-configure ZHA to change the serial port's flow control to |
|
@petepall I see you're also running EmberZNet 7.4.4.0 on your ZBT-1. Do you remember when you installed it? How did you do so? |
|
@puddly I received the stick with this software on it. I today unboxed it and start using it as I thought the issue was with my sonoff dongle P |
|
@puddly - Thank you for looking at the logs. I'm installing the firmware that you linked now and I have some firmware updates needed on the plugs - will change to hardware flow control and reply back |
|
@puddly First follow-up: Hardware: reconfigure with hardware flow control hung on initializing. Rebooted HA after ZHA hung at Initializing. ZHA started and everything is detected but now the Third Reality plugs are showing as up to date, even after a manual updates check. Can confirm network is up by toggling the TR Plugs power on and off. Will report back with next firmware update |
|
@puddly: 2nd Follow-up: Just got the prompt for my 4x Third Reality Zigbee Plugs. I updated the 1st plug thats closest to the Coordinator. Completely 100% successfully with 0 crashes. I forgot to enable Debugging mode so thankfully it completed on the first click. I enabled Debugging mode and continued with the next closest which is about 12-15ft away. in the Kitchen - it too completed 100% in the first go. Next I tried upgrading the 2 farthest away (both equidistant from the Kitchen Plug). They're slowly but surely updating (update much slower when multiple devices are updating at the same time which makes sense in a network with limited bandwidth). The update of one of the 2 simultaneous updates did crash around 80ish percent, however I was able to resume it without having to restart HA, which is a first. The 2nd of the 2 simultaneous updates completed successfully with no issues. I'm not sure if hardware data flow control is enabled or no but I'm perfectly content in the state that its in right now, from a stability perspective The log file from the 3 updates is 155MB for some reason so it wont let me upload the thread so I had to upload to Google Drive - See Log |
|
Thanks! So as far as I can tell, the firmware doesn't crash at all once you use hardware flow control. OTA failing with concurrent updates is a different issue but I think we have the firmware problem figured out. What hardware and software are you running Home Assistant OS/container on? @petepall The ZBT-1 shouldn't ship with 7.4.4.0, it should come with 7.3.1.0. Was your ZBT-1 used? If not, which distributor did you buy it from? |
|
@puddly Thanks for looking at the logs. I'm running the HAOS via the generic x86 install options on a dedicated N97 based mini pc. |
|
@puddly the ZBT-1 was not used as far as I know. I bought it from HA shop. Today I changed again the extension cable and removed my powered usb hub and till now the devices are now working. |
|
@petepall Can you link to the store? |
Here is the link https://www.hashop.nl/Home-Assistant-SkyConnect#ProductReviewText |
|
My update after upgrading to HAOS 14.1 |
|
Hi all, I'm seeing similar after an update to 2025.1 today. Error "Failed to perform the action light/turn_on. Failed to send request: ApplicationController is not running" when trying to modify switches from HA. Python tracebacks in the debug logs too. I'm running ZHA with a remote (ser2net) dongle, but the service appears to be functional, seems to be an issue in ZHA itself. The dongle is the E-type: usb-ITEAD_SONOFF_Zigbee_3.0_USB_Dongle_Plus_V2_20220811144852 |
|
Have also experienced this issue twice, running ZHA with a skyconnect. Once last week on 2025.1.3 and once about a month ago. |
|
I reverted my setup back to my old conbee2, and the issue never occurred again. So it is a ember stack issue (I had an zbdongle-e, similar to skyconnect) |
|
I am getting the same issue and similar set of logs as the original poster. Sonoff 3.0 Zigbee Dongle with EZSP. I am using ZHA For me, this issue occurs specifically and exclusively when a new HA core update becomes available. Not when an update is run, but when one is available. I have auto-updates disabled, because I wanted to manage my own update schedule and (ironically) prevent breaking changes. Like clockwork with every notification of an HA Core update being available, the application controller stops responding. Sometime within the last year, sudden loss of functionality became a reliable prompt that an HA core update is available. This just happened again today as the newest HA core update was discovered. I cannot for the life of me figure this out. What I've tried:
Logs: |
|
Chiming in to report the same thing after a reboot of my Proxmox server. Home assistant will start and Zigbee devices will be controllable for a few minutes before becoming unavailable. ZHA reports a constant state of initializing, but never initializes. I was running core 2025.1.1, but updated to 2025.1.4 with the same behavior. I also updated to Home Assistant OS 14.1, but no change. I have confirmed that my ZBT-1 shows up in the Hardware section of settings, so it's not an issue with passthrough. |
|
Hi @bigdogdylan00 |
|
I should note that I actually did experience the exact same behavior that is described in this issue Zigbee devices all appeared to be online and functioning as normal, although sensors had not updated in three hours or more and any Zigbee command to turn a light on or off would eventually result in an error. Only after a few reboots did the devices begin showing as unavailable. I do have to wonder if it may have something to do with a hardware issue, As I am still having this issue, despite rolling back to a much older backup of Home Assistant
…On January 30, 2025 2:45:57 AM CST, Eric ***@***.***> wrote:
Hi @bigdogdylan00
That is not that same issue. This issue the controller suggests everything is working fine (initialised) but devices don't update and new settings don't work
--
Reply to this email directly or view it on GitHub:
#130548 (comment)
You are receiving this because you were mentioned.
Message ID: ***@***.***>
|
|
@bigdogdylan00 the issue exists for at least several months now. I bought a sonoff ZBDongle-E in November last year and ran into this issue. From the responses in this thread it became clear it occurs with EFR32MG21 based dongles of various brands only |
The problem
My ZHA entities couldn’t be controlled anymore although all ZHA entities seemed available and fine. No sensor updates and errors logged when sending commands (example logging included was a turn on service) .
Even reloading ZHA suggested that ZHA was fine, but the failure state remained. Shortly Un-plugging the ZBDongle-E (Sonoff stock Firmware: 6.10.3.0 build 297), resolved the issue.
So for some reason the ZHA watchdog didn’t notice this hanging dongle state for hours!
What version of Home Assistant Core has the issue?
Core-2024.11.1
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant OS
Integration causing the issue
ZHA
Link to integration documentation on our website
https://www.home-assistant.io/integrations/zha/
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Logger: homeassistant.components.websocket_api.http.connection Source: components/websocket_api/commands.py:245 integration: Home Assistant WebSocket API (documentation, issues) First occurred: 19:14:03 (2 occurrences) Last logged: 19:14:08 [546888653200] Unexpected exception Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 67, in wrap_zigpy_exceptions yield File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 85, in wrapper return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/util.py", line 136, in retry return await func() ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/zcl/__init__.py", line 375, in request return await self._endpoint.request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/endpoint.py", line 270, in request return await self.device.request( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zigpy/device.py", line 378, in request await send_request() File "/usr/local/lib/python3.12/site-packages/zigpy/application.py", line 841, in request await self.send_packet( File "/usr/local/lib/python3.12/site-packages/bellows/zigbee/application.py", line 700, in send_packet raise ControllerError("ApplicationController is not running") bellows.exception.ControllerError: ApplicationController is not running The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1335, in handler return await func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/zha/switch.py", line 57, in async_turn_on await self.entity_data.entity.async_turn_on() File "/usr/local/lib/python3.12/site-packages/zha/application/platforms/switch.py", line 95, in async_turn_on await self._on_off_cluster_handler.turn_on() File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/general.py", line 465, in turn_on result = await self.on() ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 84, in wrapper with wrap_zigpy_exceptions(): File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/__init__.py", line 76, in wrap_zigpy_exceptions raise ZHAException(message) from exc zha.exceptions.ZHAException: Failed to send request: ApplicationController is not running The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service response = await hass.services.async_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2802, in async_call response_data = await coro ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2845, in _execute_service return await target(service_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call single_response = await _handle_entity_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 628, in async_handle_light_on_service await light.async_turn_on(**filter_turn_on_params(light, params)) File "/usr/src/homeassistant/homeassistant/components/switch_as_x/entity.py", line 152, in async_turn_on await self.hass.services.async_call( File "/usr/src/homeassistant/homeassistant/core.py", line 2802, in async_call response_data = await coro ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/core.py", line 2845, in _execute_service return await target(service_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call single_response = await _handle_entity_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call result = await task ^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1337, in handler raise HomeAssistantError(err) from err homeassistant.exceptions.HomeAssistantError: Failed to send request: ApplicationController is not runningAdditional information
All last seen timestamps where not updated (3 hours old when I noticed). So all mains devices should have been made unavailable by ZHA (my timeout for mains devices is 5 minutes)
The text was updated successfully, but these errors were encountered: