[REQUEST] "ESP Zigbee NCP serial to IP proxy SDK" example project (TZ-648)

[Hedda]

Is your feature request related to a problem?

@lhespress (and @chshu) not sure if should post a request here or zigpy-espzb repo at https://github.com/lhespress/zigpy-espzb

Duplicate/cross-posting of lhespress/zigpy-espzb#4

(For bsck-story also see related dependecy here -> zigpy/zigpy-cli#45 )

This however is a feature request for "ESP Zigbee NCP serial proxy SDK" example similar to "Gateway Example" (Zigbee gateway with RCP firmware):

https://github.com/espressif/esp-zigbee-sdk/tree/main/examples/esp_zigbee_gateway

That is, please consider making and maintaining an "ESP Zigbee NCP serial proxy SDK" example design to be used with zigpy-espzb

The example should demonstrate how to build an ESP Zigbee NCP serial proxy server. Using a Wi-Fi SoC such as ESP32, ESP32-C3 or ESP32-S3 running a serial-to-network proxy relay server service (similar to "ser2net") allowing for a remote socket connection, used in combination with an 802.15.4 SoC like ESP32-H2 or ESP32-C6 running ESP Zigbee NCP to provide 802.15.4 radio via ZNSP (Zigbee NCP Serial Protocol) interface.

That is, a software solution that would make the two hardware designs from the "ESP Thread Boarder Router SDK" project usable as a network-attached remote Zigbee Coordinator NCP adapter, and as such instead offer a UART-to-IP serial streaming tunneling proxy relay solution (i.e. a remote Serial Port over TCP/IP) that works the same way as TubesZB’s Zigbee Gateways and ZigStar’s Zigbee Gateways solutions, meaning that it would present Espressif Zigbee NCP with ZNSP (Zigbee NCP Serial Protocol) interface as a remote serial communication port that can be used with Home Assistant's ZHA integration (which depends on zigpy):

https://www.home-assistant.io/integrations/zha

Such solutions based on ESP32 are commonly also referred to "Serial to TCP Bridge" and Tasmota is also a popular choice for it:

https://tasmota.github.io/docs/Serial-to-TCP-Bridge/

That is, just offering a pass-through tunneling of the UART port using a "ser2net" type Serial-to-IP proxy solution for relay connection:

• ESP32-C6/ESP32-H2 with Zigbee NCP firmware <-> ESP32-S3 with Serial-to-IP Proxy Server <-> Home Assistant's ZHA integration.

Preferably announcing the proxy service on the network via Zeroconf for automatic network discovery which is supported inside ZHA:

https://www.home-assistant.io/integrations/zha#discovery-via-usb-or-zeroconf

Otherwise, the end-user has to manually configure IP address and TCP port for the socket so Zeroconf makes it very user-friendly:

https://www.home-assistant.io/integrations/zha#zigate-or-sonoff-zbbridge-devices

FYI, Zeroconf (Zero-configuration networking) in the ZHA integration in turn depends on the Zeroconf integration:

https://www.home-assistant.io/integrations/zeroconf/

PS: TubesZB Zigbee Gateways and ZigStar Zigbee Gateways solutions as just examples as there are many ESP32-based solutions for this which shows that there are real-world use cases for "ESP Zigbee NCP serial-to-IP proxy" type products:

• https://tubeszb.com/

  • https://github.com/tube0013/tube_gateways

• https://zig-star.com/projects/

  • https://github.com/mercenaruss/zigstar_gateways

• https://shop.codm.de/automation/zigbee/40/zigbee-cc2652p2-tcp-ethernet-coordinator

• https://gio-dot.github.io/Z-Bee-Duo/

• https://smlight.tech/

  • https://github.com/smlight-dev/

• https://github.com/cyijun/OpenZ3Gateway

• https://shop68536829.taobao.com/

Describe the solution you'd like.

No response

Describe alternatives you've considered.

No response

Additional context.

No response

[chshu]

@Hedda Thanks for the suggestion, we are indeed working on a esp-zigbee-host example, which will run on a ESP Wi-Fi SoC and access the ESP NCP via UART. Although we still recommend our customer to use the Host+RCP mode if they use ESP Wi-Fi SoC as the host.

Generally, for the Zigbee Gateway solution, we will have two options for our customers:

• ESP Wi-Fi + Zigbee SoCs: https://github.com/espressif/esp-zigbee-sdk/tree/main/examples/esp_zigbee_gateway, It's the Host + RCP (Radio Co-Processor) design, application and Zigbee Stack run on an ESP32 series Wi-Fi SoC (ESP32, ESP32-C, ESP32-S, etc), with an 802.15.4 SoC (ESP32-H2, ESP32-C6) loaded with Zigbee RCP, the two communicate via UART
• ESP Zigbee + 3rd party host
  It's the Host + NCP (Network Co-Processor) design, Zigbee Stack runs on ESP 802.15.4 SoC, and user application runs on a 3rd party SoC. The host could access NCP via the ESP Zigbee serial protocol (https://docs.espressif.com/projects/esp-zigbee-sdk/en/latest/esp32/application.html#esp-ncp-guide). We will provide a esp-zigbee-host example as a reference for customer.

[xieqinan]

@teleger ，

The openthread is located within the esp-idf components. Could you please double-check the path to your esp-idf installation?

[Hedda]

@teleger that is off-topic for this specific request. Please start a new seperate issue instead to no hi-jack this.

[MartinXBcn]

@Hedda Thanks for the suggestion, we are indeed working on a esp-zigbee-host example, which will run on a ESP Wi-Fi SoC and access the ESP NCP via UART. Although we still recommend our customer to use the Host+RCP mode if they use ESP Wi-Fi SoC as the host.

Generally, for the Zigbee Gateway solution, we will have two options for our customers:

• ESP Wi-Fi + Zigbee SoCs: https://github.com/espressif/esp-zigbee-sdk/tree/main/examples/esp_zigbee_gateway, It's the Host + RCP (Radio Co-Processor) design, application and Zigbee Stack run on an ESP32 series Wi-Fi SoC (ESP32, ESP32-C, ESP32-S, etc), with an 802.15.4 SoC (ESP32-H2, ESP32-C6) loaded with Zigbee RCP, the two communicate via UART
• ESP Zigbee + 3rd party host
  It's the Host + NCP (Network Co-Processor) design, Zigbee Stack runs on ESP 802.15.4 SoC, and user application runs on a 3rd party SoC. The host could access NCP via the ESP Zigbee serial protocol (https://docs.espressif.com/projects/esp-zigbee-sdk/en/latest/esp32/application.html#esp-ncp-guide). We will provide a esp-zigbee-host example as a reference for customer.

Can you give an update of the status of the development of "esp-zigbee-host example, which will run on a ESP Wi-Fi SoC and access the ESP NCP via UART"?
Question for my understanding:
Why do you recommend the "RCP-solution" instead of the "NCP-solution"?
I would have thought that in case of the "NCP-solution" the workload for the host-ESP32-S3 is less compared to the "RCP-solution", i.e. more processor-capacity available on the host-ESP32-S3 for other user-tasks. Am I wrong?

[Hedda]

Why do you recommend the "RCP-solution" instead of the "NCP-solution"?

That might no longer be the recommation as the project by @lhespress that would be first to make use of it currently only support NCP for now, see:

https://github.com/lhespress/zigpy-espzb

[lhespress]

Can you give an update of the status of the development of "esp-zigbee-host example, which will run on a ESP Wi-Fi SoC and access the ESP NCP via UART"?

We have provide a esp-zigbee-host example as a reference.

Why do you recommend the "RCP-solution" instead of the "NCP-solution"?

This is recommend when ESP Wi-Fi + Zigbee SoCs, of course, you also can use "NCP-solution" on ESP Wi-Fi SoCs.

[MartinXBcn]

Can you give an update of the status of the development of "esp-zigbee-host example, which will run on a ESP Wi-Fi SoC and access the ESP NCP via UART"?

We have provide a esp-zigbee-host example as a reference.

Why do you recommend the "RCP-solution" instead of the "NCP-solution"?

This is recommend when ESP Wi-Fi + Zigbee SoCs, of course, you also can use "NCP-solution" on ESP Wi-Fi SoCs.

I have seen the example "esp-zigbee-host".
What I am missing in the example - as I do not have direct access to the ESP32-H2-MINI-1 but it is connected to the ESP32-S3-WROOM-1 via UART/SPI - is to upload the firmware of ESP32-H2 via the ESP32-S3 like it is realized in the example "esp-zigbee-gateway", i.e. the built NCP image will be automatically packed into the esp-zigbee-host-firmware and uploaded to ESP32-S3 and from the ESP32-S3 to ESP32-H2.
I started to add the upload-feature of "esp-zigbee-gateway" to "esp-zigbee-host", upload works, but I was struggling with other problems. That is why I asked if there is already a ready-to-run-example. Or is there a separate example only showing the feature of uploading a firmware from ESP32-S3 to ESP32-H2?

[chshu]

Or is there a separate example only showing the feature of uploading a firmware from ESP32-S3 to ESP32-H2?

The ESP RCP update is a standalone feature supported by the esp_rcp_update component, it used in several examples:

• Basic Zigbee Gateway
• Thread Border Router
• Rainmaker Zigbee Gateway

You may refer to these examples and integrate the feature to your project.

If you are using both ESP SoCs as Wi-Fi and Zigbee, the Gatway + RCP mode is recommended. The Host + NCP is only suggested when using ESP NCP with 3rd-party host SoC. The difference between RCP and NCP modes:

• RCP Mode: Zigbee Stack and application all run on host SoC, only 15.4 Mac layer frame transmit and receive actions are communicated between the two SoCs
• NCP mode: Zigbee stack runs on NCP, ZCL and Applications run on host, the protocols between Host and NCP is more complicate than RCP mode, more code needs to be implemented for each ZCL cluster behavior (commands/events/callbacks), all these are naturally supported if the application and stack run on the same SoC in RCP mode.

[Hedda]

@lhespress Do you have any more thoughts on these name change suggestions? -> lhespress/zigpy-espzb#11

Again, please see all related comments in your pull request for zigpy-cli here -> zigpy/zigpy-cli#45