How to get initial state for devices?

[gwww]

I'm working on some Python code that connects to the mosquitto broker (as a Home Assistant plugin). My initial subscribe is for zigbee2mqtt/#. I receive a number of messages from the bridge, for example zigbee2mqtt/bridge/devices among others. What I don't receive is the state of the devices.

I understood the state to be a retained message and should receive the state on connect.

Any pointers would be appreciated on how to get the device state on connect. Thank you!

[gwww]

I have much working, just not able to get the state on startup. Assuming that state is not a retained message, based on looking at incoming state messages, I tried to get the state. For example, I published a message zigbee2mqtt/Ikea bulb/get with payload {"state": ""} - which works.

However, when I try to get current state of an Aqara door sensor using the same payload I get the error No converter available for 'state', which is understandable since the door sensor doesn't have state as an attribute. So, I tried switching to an attribute that is part of the Aqara door sensor. I tried a number of them and got No converter available for '<whatever attribute I tried to retrieve'.

Here is the state message that I get from an Aqara door sensor. I get this message by activating the sensor.

{
  "battery": 80,
  "battery_low": false,
  "contact": false,
  "linkquality": 72,
  "power_outage_count": 0,
  "tamper": false,
  "update": {
    "installed_version": "unknown",
    "state": "idle"
  },
  "update_available": false,
  "voltage": 2970
}

So, any pointers would be appreciate on how to get the state of devices, since device state does not appear to be retrained! Thanks!

[gwww]

Last update for a bit. Curious, I looked at the front end extension code. If I understand correctly when the front end is started by z2m it pulls the state from internal z2m and sends them out for the front end to receive. I don't claim to fully understand if that's right but if true it explains how the front end has initial device state.

https://github.com/Koenkk/zigbee2mqtt/blob/master/lib/extension/frontend.ts#L113-L140

[gwww]

I have been working on this all day. I finally stumbled on a Reddit post that describes how to fix. I still have two questions though.

• How does the frontend have state if it doesn't get it through MQTT? Is it the code link above?
• I don't understand why the /get that I'm using doesn't work.

The fix, if others are wondering came from this post: https://www.reddit.com/r/homeassistant/comments/w9daui/some_zigbee2mqtt_devices_have_wrong_state_after/

Specifically, adding the following to your z2m configuration.yaml:

device_options:
  retain: true

[gwww]

Anybody? Should I even being turning on retain globally?

[MatAtBread]

I do this using a homebrew extension:

class QueryStateExtension {
    constructor(zigbee, mqtt, state, publishEntityState, eventBus, settings, logger) {
        this.state = state;
        this.zigbee = zigbee;
        this.logger = logger;
        this.mqttBaseTopic = settings.get().mqtt.base_topic;
        this.eventBus = eventBus;
        this.mqtt = mqtt;
    }

    start() {
        const mqttHandler = (topic, message) => {
            if (topic === `${this.mqttBaseTopic}/bridge/query`) {
                const what = JSON.parse(message).what;
                if (what === 'state')
                    this.mqtt.publish('bridge/query/' + what, JSON.stringify(this.state.state));
                else if (what === 'devices') {
                    const devs = this.zigbee.devices().map(device => [device.name,{defintion: device.definition, state: this.state.state[device.ieeeAddr] }]);
                    this.mqtt.publish('bridge/query/' + what, JSON.stringify(Object.fromEntries(devs)));
                }
            }
        };
        
        this.eventBus.onMQTTMessagePublished( this, data => mqttHandler(data.topic, data.payload), this.constructor.name);
        this.eventBus.onMQTTMessage(this, data => mqttHandler(data.topic, data.message), this.constructor.name);
    }
    
    async stop() {
        this.eventBus.removeListeners(this.constructor.name);
    }

}

module.exports = QueryStateExtension;

You can query the state (or all devices) or both device definition and state by sending:
mqtt.send("zigbee2mqtt/bridge/query", { what: 'state' })
or
mqtt.send("zigbee2mqtt/bridge/query", { what: 'devices' })

Responses look are posted back on topics zigbee2mqtt/bridge/query/state and zigbee2mqtt/bridge/query/devices

Note that the unloading/re-loading of extensions seems to have a bug, and you may end up with both the old and new extensions firing, until you restart the server.

[MatAtBread]

With a couple of extra lines, you query individual devices, if you have a huge network

[gwww]

Thanks! Very helpful. I will play around with it.

What I’ve been doing so far is I turned on retain.

[ajay10000]

Thanks @MatAtBread for this zigbee2mqtt Extension! I am trying to debug a custom device and work out how everything fits together.

I was stuck trying to send the commands you suggested above from the Mosquitto client in Windows. After getting it working in linux (RPi/debian) I eventually worked out how to escape the quotes in Windows. For anyone else trying this, here are the commands (with debug):
linux: mosquitto_pub -h my_server -p 1883 -t "zigbee2mqtt/bridge/query" -m '{"what":"devices"}' -d

windows:
mosquitto_pub -h my_server -p 1883 -t "zigbee2mqtt/bridge/query" -m {"""what""":"""devices"""} -d

[Josef-Friedrich]

zigbee2mqtt 2.1.3: this.zigbee.devices() -> this.zigbee.devicesIterator()

[Anzal3c3]

@MatAtBread hey thanks for sharing this . its a extension right? i havent worked with java script , i can add this extension to z2m using the instructions provided in the website right? Can anybody provide insight to create a custom converter to get sensor status . any help would be much appreciated

[MatAtBread]

It's a custom extension. You can load (copy paste) it remotely via the "extensions" menu item in the UI

[Anzal3c3]

okay i did and its working now. as i said that i dont have any experience in working with javascript . i need to change the subscribe topic to
'basetopic/device_id/get' and publish topic to 'basetopic/device_id' how can i do that and can you provide some resources which would be helpful for my case

thanks in advance

[MatAtBread]

The JS construct ${this.mqttBaseTopic} evaluates the expression within the {}

In almost all cases, this will be the default zigbee2mqtt topic, so you should send the message as per the earlier post:

You can query the state (or all devices) or both device definition and state by publishing the following message from your MQTT client code:
mqtt.send("zigbee2mqtt/bridge/query", { what: 'state' })
or
mqtt.send("zigbee2mqtt/bridge/query", { what: 'devices' })

Responses look are posted back on topics zigbee2mqtt/bridge/query/state and zigbee2mqtt/bridge/query/devices, which you should subscribe to in your MQTT client code in advance.

Note that the unloading/re-loading of extensions seems to have a bug, and you may end up with both the old and new extensions firing, until you restart the server.

As for learning JS, specifically within the NodeJS env (zigbee2mqtt runs in NodeJS, and the extensions run within that), try Google - there are literally thousands of sites with examples. What languages do you usually use?

[MatAtBread]

If you want to change the topics, you need to (a) make sure you're not over-loading an existing topic (I think your example is doing just that - you shouldn't attempt to modify an existing Z2M message handler), and (b) write something that handles the case.

It sounds like you don't really want an "extension" for this, but a custom device handler, which (I believe - I've never done it) is described here https://www.zigbee2mqtt.io/advanced/support-new-devices/01_support_new_devices.html

[MatAtBread]

If your "sensor" isn't really a zigbee device (with all the support that entails), and you just want to add a "new" message handler, you could simply write a separate server app that talks to your MQTT server, ie one that listens for a specific topic and publishes a response - no need to modify Z2M or herdsman at all. Obvs, it won't appear in the Z2M UI in that case

[Anzal3c3]

okay thanks for the help . appreciate it

[Anzal3c3]

If your "sensor" isn't really a zigbee device (with all the support that entails), and you just want to add a "new" message handler, you could simply write a separate server app that talks to your MQTT server, ie one that listens for a specific topic and publishes a response - no need to modify Z2M or herdsman at all. Obvs, it won't appear in the Z2M UI in that case

its a TuYa door sensor

[MatAtBread]

Check the instructions in the above repo. You'll need a new concertor. I'd say there's a 90% chance there already a definition for it, but maybe not your variant (assuming Z2M didn't identify it). In which case you'd only need to add the model or manufacturer's ID to the existing definition for Z2M to find it

[Anzal3c3]

thanks a lot brother i will try doing that

[Anzal3c3]

hello @MatAtBread i have impelemented the above extension to get the state of the device i have managed to get the state of individual devices by using ${this.mqttBaseTopic}/deviceID/get and publish the device state back in ${this.mqttBaseTopic}/deviceID now the problem is for powered devices also the state is being published back in the same topic for powered devices two response payload is being published in the above topic is there a way to identify the powered and non powered device from the extension itself please provide any info regarding this

[Anzal3c3]

here is the updated code suggest edits to perform the above

class QueryStateExtension {
    constructor(zigbee, mqtt, state, publishEntityState, eventBus, settings, logger) {
        this.state = state;
        this.zigbee = zigbee;
        this.logger = logger;
        this.mqttBaseTopic = settings.get().mqtt.base_topic;
        this.eventBus = eventBus;
        this.mqtt = mqtt;
    }

    start() {
        const mqttHandler = (topic, message) => {
            if (topic.startsWith(`${this.mqttBaseTopic}/`) && topic.endsWith('/get')) {
                const deviceId = topic.slice(this.mqttBaseTopic.length + 1, -4);
                const requestedKey = Object.keys(JSON.parse(message))[0];
                const queriedDeviceState = this.state.state[deviceId];
                console.log('Parsed message:', { topic, deviceId, });
                if (queriedDeviceState) {
                    const response = { [requestedKey]: queriedDeviceState[requestedKey] };
                    this.mqtt.publish(`${deviceId}`, JSON.stringify(response));
                } else {
                    this.logger.error(`Device with ID ${deviceId} not found.`);
                }
            }
        };

        this.eventBus.onMQTTMessagePublished(this, data => mqttHandler(data.topic, data.payload), this.constructor.name);
        this.eventBus.onMQTTMessage(this, data => mqttHandler(data.topic, data.message), this.constructor.name);
    }

    async stop() {
        this.eventBus.removeListeners(this.constructor.name);
    }
}

module.exports = QueryStateExtension;