Continuous updates (experimental feature)

[jeroenterheerdt]

Let's discuss the continuous updates feature here!

[jeroenterheerdt]

#363 and #436 let's move discussion about how to handle various sensors / aggregates to this conversation please!

[jeroenterheerdt]

Two things come to mind:

1. How about precipitation? Shouldn't that be the actual rain rate and not the last/max of daily rain rate when continuous updates are on?
2. Should we be using reimann integral for solrad sensor? If so, how?

[jata1]

@jeroenterheerdt - thanks for introducing this feature. I have it switched on on prod and test environments so happy to help investigate and test.

solar radiation is tricky and I am not getting bucket calculations like I would expect but it is winter here in Oz so that is a factor.

I think there is something not right with how I setup my reimann integral sensor - see below.

raw data from the solar rad sensor on my weather station:

daily solar rad using a reimann integral (curve shape is good but daily total is out maybe by a factor of 100 or 1000):

[jata1]

So as a next step, I will create a new riemann sensor x100 and one x1000 to see if this is closer.

For reference - before I used a riemann I had a senor that simply added up the sol rad sensor and it looks like this - but even this was underestimating watering so this approach did not work either (for me).

[jata1]

Tried changing the riemann to kilo but it only changes the unit so not helpful (for me)

[jata1]

I think I might have found my issue with solar rad being under reported. I found this on the interweb...

The daily irradiation in Wh/m2 will be obtained as the sum of all hourly values in W/m2. For instance, if the irradiance is constant at 100 W/m2 during 10 hours, the daily total irradiation is 1000 Wh/m2=1 kWh/m2.

The same principle applies to 10-min values, but then the total needs to be divided by 6. So if the irradiance is constant at 100 W/m2 during 10 hours or 60 10-min periods, the daily irradiation is 100x60/6=1 kWh/m2 again, as expected.

So I think you need to setup the riemann integral as an hourly calculation (rather than daily) - this means that my original setup was under reporting by a factor of 24. So

I will see what happens now

[pumabaer]

Hi,
Can you share your configuration for your Riemann sum integral sensor?
Did you solve it using a helper or in the configuration.yaml?

[posixx]

Just to understand continous updates; if using sensors only including ET, continuous updates are not possible? What is the advantage of continuous updates?

[jeroenterheerdt]

If you use sensors only you should be able to use continuous updates. If you rely on a weather service we block you right now. Continuous updates update as soon as your weatherstation gives new values so should be highly accurate to your weather as it's up to the minute or second updates depending on how quickly your weatherstation operates..

[posixx]

Ok so this also works using passthrough module with ET sensor available?

[jeroenterheerdt]

it should.

[posixx]

Nice. It would be nice you can fix #430 (comment) so my log isn’t flooded between calculation time and 1st ET sensor update

[pumabaer]

Hello,

I have enabled continuous calculation.
Now, when it calculates in the evening at 11:50 PM, the time increases again. However, the time then looks much higher than during the continuous calculation.
Example:
Over the day, the bucket empties to -0.5. In the evening, it then jumps to -2.5.

Does the evening calculation need to be turned off?

[jeroenterheerdt]

It should not be necessary to turn it off but there is a bug right now that seems to cause this

[pumabaer]

Ah okay.
Let me know if you need any more information.

[jeroenterheerdt]

I think I solved the bug in the latest update (v2024.7.10) so you should be able to keep both continuous updates and automatic updates on.

[pumabaer]

[jeroenterheerdt]

please open an issue for this.

[pumabaer]

Hi,
How should the "solar radiation" sensor be configured?
Is "average" the right option?

[jata1]

Hi, How should the "solar radiation" sensor be configured? Is "average" the right option?

I have made good progress on calculated daily solar radiation. Here is my setup but you will need to adjust for whatever your raw/source solar rad sensor is outputting.

I am using 2 helpers to get this working. One calculates the riemann integral for solar radiation hourly as my sensor in in watts (watts hrs). The other is a utility meter that resets the the riemann sum data every day so you have a good daily calculation of total solar radiation.

In smart irrigation setup you use the utility meter and set average as the calculation method.

sol rad sensor output units is W/M2:

Riemann helper:

Utility meter:

Here is what the solar output daily looks like from my system based on the above. Just plug this sensor into smart irrigation.

[jeroenterheerdt]

@jata1 how easy was this to set up? should we even try to implement this all in the integration itself?

[pumabaer]

Thanks for sharing!
I will test it as well.
If it works as expected, it would be good to incorporate it directly into the integration.

[jata1]

@jata1 how easy was this to set up? should we even try to implement this all in the integration itself?

The root issue in my case is related to the weather station integration/data as it provides real time solar radiation in W/M2. This data needs to be manipulated to be used in the ETO calcs in smart irrigation.

Now I know what I am doing it is easy but took a bit of working through (as you know).

I will contact the guys that maintain the integration for my weather station to see if I can get them to add an additional sensor for daily accumulated solar radiation. solar radiation needs to be handled the same as rain/precipitation.

[pumabaer]

Hi, How should the "solar radiation" sensor be configured? Is "average" the right option?

I have made good progress on calculated daily solar radiation. Here is my setup but you will need to adjust for whatever your raw/source solar rad sensor is outputting.

I am using 2 helpers to get this working. One calculates the riemann integral for solar radiation hourly as my sensor in in watts (watts hrs). The other is a utility meter that resets the the riemann sum data every day so you have a good daily calculation of total solar radiation.

In smart irrigation setup you use the utility meter and set average as the calculation method.

sol rad sensor output units is W/M2:

Riemann helper:

Utility meter:

Here is what the solar output daily looks like from my system based on the above. Just plug this sensor into smart irrigation.

Are you sure that "average" is the right setting for solar radiation?

If I have a counter, the average value is always below the actual (last counted) measured value, right?
Value 1 + value 2, etc. / 2

Just a thought, I might be wrong about this.

[jata1]

@jeroenterheerdt - the good news with all of this work is that I finally have smart irrigation working properly as I am handling solar radiation correctly so the daily ETO calcs are working. The sensor for total daily solar radiation looks correct when comparing to the solar radiation raw data sensor.

[jeroenterheerdt]

Love it. Would you be willing to write up a step by step process of what you did to configure your sensor group and input sensors for continuous updates? I am planning to rewrite the readme (move to wiki) and I think it would be good to have what you learned in there!

[jata1]

Happy to do a write up but not quite there yet.

My testing over the last week has not been using continuous updates as I wanted to test one thing at a time. I now have daily calcs working correctly with my solar radiation sensors. Now I am switching to continuous and I will see what I find and update this thread.

[jata1]

After switching to continuous updates, I have some new errors in the log and a couple of observations.

My setup has two zones (front and back) each with its own sensor group. The sensor groups are identical except one has precipitation set as a static value of 0 (as this zone is under cover). However, the amount of data collected for the two zones are very different - see image below from the last hour since I switched to continuous.

Diagnostic file attached...
config_entry-smart_irrigation-01J3NXJFTGQR2QDP3BMYX2SF83.json

[jeroenterheerdt]

Wait, that's weird! The sensor groups are the same except for the one static? I am wondering if somehow the static zone gets skipped?

[jata1]

So I have been playing with continuous updates and there is something strange going on. I have a feeling it is related to having two sensor groups that use the mostly the same sensors.

I had it working but only one zone was actually updating continuously. So I deleted the sensor group that I assumed was not working and set it up new.

Now this one works but the other zone is not updating. It's very confusing.

My next test is to use the same sensor group on both zones. I will report back

[jeroenterheerdt]

Is there anything in the logs that can help?

[jata1]

Nothing obvious in the logs. I don't understand why there is much more data being collected for one sensor group compared to the other. I think this is a clue to an underlying issue.

Right now I am running a different test. I have set both zones to use the same (newly created) sensor group so I will see what happens with this and report back.

[jata1]

Just an update from my experience over the last few days...

I can't get good results with continuous enabled. Not sure how to investigate further but happy to do so if helpful.

For now, I have switched to a daily calcs and this seems to be woking nicely now that I have fixed solar radiation by using the riemann sum helper/sensor setup.

[ChrisHomeAssistant]

Yes, definitely. Also transforming the daily "Wh" value to MJ/d makes sense. E.g. taking a value of 2000Wh you showed equals (*3.6/1000) 7.2MJ per day which is a plausible value.

For the ET calculation I would assume the input should be the last value in the sensor. Or does the formula in the calculation expect a different input?

[jata1]

Yes. I am using last or max in the sensor group config. I think either is fine.

[ChrisHomeAssistant]

I'm not completely sure about this. The input is converted with a factor of 0.0864, which also accounts for a "hour to day" conversion (0.0036*24).
Nevertheless, if it is already an aggregated figure using the Riemann sum it should only be converted by 0.0036 (from Wh to MJ), right?

[jata1]

I'm not sure to be honest. You might be right. If correct then this could be a bug.

At the moment, my calculated watering is about right and I spent a long time getting sol rad to work properly. I hope it does not have a calculation error x 24.

Suggest creating a new bug / issue on this topic so we can bottom it out with @jeroenterheerdt

[ChrisHomeAssistant]

Maybe it is related to the bug #461 (comment)

and the general question how the input is expected.

The calculation of the solrad itself looks absolutely fine for me.

[jata1]

see below. I had a powercut for 2hrs and you can see that riemann is flat during this time (as there is no data).

left chart is my source solar radiation sensor. right is riemann sum of the data...

[jeroenterheerdt]

slowly getting back to continuous updates. One thing I noticed is that there are no updates during the night. It looks like that is very dependent on your weather station, but mine for example doesn't provide a value for solar radiation during the night. Then the calculation is never done in PyETO because solar radiation (and whatever the weather station does not provide) is missing.
I see two options here:

• keep the current behavior. The calculations will happen again as soon as all data points are provided.
• when things are missing, set them to a default value so calculation can happen. Solar radiation is easy: default is 0. Wind speed same. But dewpoint? Temperature? Pressure? Not sure what to do with those.

Thoughts?

[jeroenterheerdt]

that's weird, I need to create a dev environment with just one zone then to see if I can reproduce this.
Can you turn on debug logging for a night and share that to see if the behavior you are seeing is the same as I described above? Also do you know if you weather station provides all values all the time or also sometimes skips some data points? I think I know why the calculations are paused: after a clear, until we have at least one value for each data point then we can't calculate using PyETO. So if you are not estimating solar radiation PyETO will only calculate once there is a value for it, so only after the sun is up.

I am thinking of solving this by always putting in default values if it's missing (like solar rad = 0 if it's not there) but not sure about the others (dewpoint? temperature?)... any ideas on what to do?

[pumabaer]

I’ve just enabled debug logging. I’ll provide it to you tomorrow.

Regarding the weather station:
It seems that not all values are always updated, only the changes. For example, right now the last value for solar radiation is from 21:07.

Setting the solar radiation to “0” makes sense.
But is there maybe a way to delete the weather data except for the last data point? So, to keep the last measured value of each sensor.

Example:
Measurement 1
23 degrees
Measurement 2
24 degrees
Measurement 3
21 degrees
—————
After deletion:
Measurement 1 (3)
21 degrees

This would avoid the problem with temperature, etc.

[jeroenterheerdt]

oh, that is likely what is happening: you only get new values when it has changed - if that's indeed what is happening here then would indeed make sense to keep the last measurement for all around and just override with whatever just came in. I think we can then do the same for solar radiation as well (as it should be 0 anyway). I'll experiment with that!

[jeroenterheerdt]

I was not able to reproduce the issue where you need two zones - my continuous update works fine with just one zone, so not sure if what is going on there...

[pumabaer]

I will set up a second HA (Home Assistant) when I get the chance and test this as well.

For now, it works as is, and I’d rather not change it. :D

[pumabaer]

Just to clarify:
If the total rainfall is used in the continuous calculation (which is reset to 0 at midnight), will the subsequent calculation still be accurate?

I also have a sensor that gives me the current rainfall rate. The value is in mm/h (naturally much higher than the total rainfall, but only for the duration that it’s raining).

It might rain in the next few days. I’ll keep an eye on it.

[jeroenterheerdt]

I think it should still be accurate.

[jeroenterheerdt]

ok, so what's the consensus now for which aggregates to use when continuous updates are on?

• solar radiation: seems like we are converging on taking the average of the riemann integral
• precipitation: needs to be the actual rain rate, not the total precipitation. aggregate needs to be last. I think for this we need to add a new data point to the sensor group that allows me to configure current rain rate.
• any other data point (dewpoint/temperature/humidity...): last?

Either we need to instruct users to set this up or we just take care of it automatically once you have continuous updates on. We can go either way, I think it would be confusing if I can set up a sensor group with whatever aggregation I want and then continuous updates would just "override" the aggregate to whatever it needs. However, this might be the easiest for the user (more done automatically).

The alternative is that the user has to set up two sensor groups: one for zones that can be continuously updated and one for zones that cannot (rely on weather service, for example because they use forecasting). This means the user needs to do more work and likely they will not read the docs instructing them to do so.

Conclusion: I think continous updates should just apply the right aggregates. If you all agree, that just leaves the other question: which aggregates to use (see above).