Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Frequent data gaps with MS600 (GEN3) inverter #68

Open
s-allius opened this issue May 10, 2024 · 8 comments
Open

Frequent data gaps with MS600 (GEN3) inverter #68

s-allius opened this issue May 10, 2024 · 8 comments
Labels
enhancement New feature or request

Comments

@s-allius
Copy link
Owner

With an MS600 (GEN3) inverter with DSP version 5.1.09, I regularly see gaps in the data transmission. An initial analysis has shown that the TCP connection is stable and TCP keep-alive packets are also being answered.

We may be able to get the inverter out of this state by reestablishing the connection. I am not aware of a way to restart the DSP using a magic packet.
I will try to recognize the situation by time monitoring and then close the TCP connection from the proxy side (sending a FIN packet) in the hope that the inverter will then send data again immediately.

Bild 10 05 24 um 20 23
@s-allius s-allius added the enhancement New feature or request label May 10, 2024
@roethigj
Copy link

roethigj commented May 10, 2024 •
edited
Loading

I see the same gaps with my MS600 (using dev-0.8.0, 10 sec interval. I switched off cloud. The gap is very regularly: every ~13 min, 55-58 request (or similar). Looks like your gaps have the same frequency - interval 1 min, gap each hour.
Interesting point: the gaps are shorter too -> same amount of request before connection reset?

Maybe the MS600 accepts only a max number of requests per connection. Reset connection by default after 50 requests?

Bildschirmfoto vom 2024-05-10 21-57-54

Same with 20 sec request interval. It works for 55-58 requests. Followed by 11-18 unanswered requests. Than:

  1. a new connection (new port) is accepted.
  2. old connection is reset by peer. (server loop stops.)
  3. modbus 0x70 requests are send to now connection and are answered
  4. (BUG: 0x70 requests are send to closed connection too)

@s-allius
Copy link
Owner Author

s-allius commented May 11, 2024 •
edited
Loading

That is interesting. Today I also saw very long gaps (up to 15 minutes). So not only the Modbus responses are missing, the inverter itself is not sending any packets either.
During the analysis, I noticed that I had forgotten to call the close() function to delete internal structures during a graceful disconnect. Half-open TCP connections could be the result. Fix is checked in the 0.8.0 branch. (That should fix your seen bug - point no 4)

Today I also implemented a monitoring function that resets the connection after 2.5 minutes without receiving any packets. This will hopefully keep the dropouts to a minimum.
However, the actual cause is not yet clear to me.
Which DSP firmware do you have? I have V5.1.09 on it after an update to 800VA.

@s-allius
Copy link
Owner Author

s-allius commented May 12, 2024 •
edited
Loading

I have an idea.

Before the Modbus payload, there are four bytes (offset 0x17 directly after the message ID), which could be a packet counter or a reset timestamp. I have set this byte to a fixed value of b'\x00\x01\xa3\x28'.
But in my traces, I saw that this value changes with every request from the TSUN cloud. I have five examples:

2024-04-30 10:32:37   00 01 a3 06
2024-04-30 10:37:41   00 01 a3 08
2024-04-30 10:52:54   00 01 a3 0e
2024-04-30 11:33:28   00 01 a3 14
2024-04-30 15:20:23   00 01 a3 28

And you are wright, the resets are very regular. We can see it in the Power On Timegraph:
Bild 12 05 24 um 10 50

Now I check, what will happen with all four bytes set to 0x00.

@s-allius
Copy link
Owner Author

Today I saw a gap from 13:15 until 18:00. My timeout handler worked as expected and close the TCP connection properly (server loop stoped). But the inverter didn't establish a new connection.

Maybe I have a network problem. It is possible, that the inverter roamed to another Accesspoint in that moment. (I have three APs here) But wenn the inverter don't make a reconnection, then I see no chance to mitigate the problem from the proxy side.

I will invest some more time to analyse it.

@jkrwdf
Copy link

jkrwdf commented May 13, 2024 •
edited
Loading

I also had gaps larger than usual (AVM AP) this afternoon, so likely no local issue on your side.

I attach my report from TSUN website.

device-report-2024051319514161.xlsx

BTW: I do not use the proxy.

@s-allius
Copy link
Owner Author

Today a German TSUN handler wrote on his web page, that beginning with the 20th of May customer can switch to the Smart App even for GEN3 inverters. Maybe this will include a new firmware which supports the SolarmanV5 protcol.

The article is only in German available: https://priwatt.de/blog/tsun-wechselrichter-update-800-watt/

@s-allius
Copy link
Owner Author

s-allius commented Aug 10, 2024 •
edited
Loading

With the proxy version 0.10 the connection is really stable here. But I have to deactivate the polling mode. It seems to me, that the inverter has a problem with the frequent MODBUS request every minute.

To deactivate the polling mode add a line to your config.toml and set modbus_polling to false for the GEN3 inverter

See also Inverter config

@Phil1pp
Copy link

Phil1pp commented Feb 12, 2025

Did you try to further analyze this?
I am also seeing data gaps and reconnects every 2-3 hours on my TMS800 (Gen3).
Disabling the polling_mode solves it, but I would like to have more frequent updates.

It seems that there is still something wrong with accepting new connections and closing old connections.
Because less than a minute after the disconnect, the inverter is trying to connect again.
However it seems the proxy isn't sending any data and closing the connection again after some time.
It then takes another connection approach from the inverter to successfully start the proxy again.

´´´
'2025-02-07 10:47:39 INFO | data | [balcony/] MODBUS: grid.Output_Power : 484.0W'
'2025-02-07 10:48:37 INFO | data | [balcony/] MODBUS: grid.Output_Power : 502.2W'
'2025-02-07 10:49:39 INFO | data | [balcony/] MODBUS: grid.Output_Power : 520.4W'
'2025-02-07 10:51:04 INFO | data | [balcony/] Modbus timeout (FCode: 3 Reg: 0x300f, 7)'
'2025-02-07 10:51:39 WARNING | conn | [balcony/:22] Dead connection timeout (120s) for ('172.23.0.2', 5005)'
'2025-02-07 10:51:39 INFO | conn | [balcony/:22] Server loop stopped for r('192.168.100.180', 54350)'
'2025-02-07 10:51:39 INFO | conn | [balcony/:22] disc client connection: [balcony/:23]'
'2025-02-07 10:51:39 INFO | conn | [balcony/:23] Peer closed. for ('172.23.0.2', 60370)'
'2025-02-07 10:51:39 INFO | conn | [balcony/:23] Client loop stopped for l('172.23.0.2', 60370)'
'2025-02-07 10:52:20 INFO | conn | [G3:24] Accept connection from ('192.168.100.180', 54351)'
'2025-02-07 10:54:45 INFO | conn | [G3:24] Peer closed. for ('172.23.0.2', 5005)'
'2025-02-07 10:54:45 INFO | conn | [G3:24] Server loop stopped for r('192.168.100.180', 54351)'
'2025-02-07 10:54:52 INFO | conn | [G3:25] Accept connection from ('192.168.100.180', 57773)'
'2025-02-07 10:55:11 INFO | root | [balcony/] Connect to ('logger.talent-monitoring.com', 5005)'
'2025-02-07 10:55:11 INFO | root | [G3:26] Connected to ('logger.talent-monitoring.com', 5005)'
'2025-02-07 10:55:12 INFO | data | [balcony/] GEN3: collector.Collector_Fw_Version : RSW_400_V1.00.17'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: env.Inverter_Status : 1'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_Alarm : 0'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_Fault : 0'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_Bitfield_1 : 0'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: events.Inverter_bitfield_2 : 0'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: grid.Output_Power : 606.1W'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: total.Daily_Generation : 0.53kWh'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: total.Total_Generation : 44.6kWh'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: inverter.Max_Designed_Power : 800W'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: other.Prod_Compliance_Type : 2'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: inverter.Output_Coefficient : 100.0%'
'2025-02-07 11:00:19 INFO | data | [balcony/] GEN3: inverter.Version : V5.0.17'
'2025-02-07 11:01:03 INFO | data | [balcony/] MODBUS: grid.Output_Power : 634.7W'
'2025-02-07 11:02:03 INFO | data | [balcony/] MODBUS: grid.Output_Power : 678.2W'
´´´

´´´
'2025-02-12 13:47:38 INFO | data | [balcony/] MODBUS: grid.Output_Power : 246.0W'
'2025-02-12 13:48:40 INFO | data | [balcony/] MODBUS: grid.Output_Power : 262.3W'
'2025-02-12 13:49:11 INFO | data | [balcony/] GEN3: grid.Output_Power : 255.7W'
'2025-02-12 13:49:11 INFO | data | [balcony/] GEN3: total.Daily_Generation : 1.2kWh'
'2025-02-12 13:49:11 INFO | data | [balcony/] GEN3: total.Total_Generation : 54.54kWh'
'2025-02-12 13:49:55 INFO | data | [balcony/] MODBUS: grid.Output_Power : 259.1W'
'2025-02-12 13:51:21 INFO | data | [balcony/] Modbus timeout (FCode: 3 Reg: 0x300f, 7)'
'2025-02-12 13:51:55 WARNING | conn | [balcony/:239] Dead connection timeout (120s) for ('172.23.0.2', 5005)'
'2025-02-12 13:51:55 INFO | conn | [balcony/:239] Server loop stopped for r('192.168.100.180', 56679)'
'2025-02-12 13:51:55 INFO | conn | [balcony/:239] disc client connection: [balcony/:240]'
'2025-02-12 13:51:55 INFO | conn | [balcony/:240] Peer closed. for ('172.23.0.2', 58116)'
'2025-02-12 13:51:55 INFO | conn | [balcony/:240] Client loop stopped for l('172.23.0.2', 58116)'
'2025-02-12 13:52:26 INFO | conn | [G3:241] Accept connection from ('192.168.100.180', 56680)'
'2025-02-12 13:54:51 INFO | conn | [G3:241] Peer closed. for ('172.23.0.2', 5005)'
'2025-02-12 13:54:51 INFO | conn | [G3:241] Server loop stopped for r('192.168.100.180', 56680)'
'2025-02-12 13:54:58 INFO | conn | [G3:242] Accept connection from ('192.168.100.180', 55067)'
'2025-02-12 13:55:17 INFO | root | [balcony/] Connect to ('logger.talent-monitoring.com', 5005)'
'2025-02-12 13:55:18 INFO | root | [G3:243] Connected to ('logger.talent-monitoring.com', 5005)'
'2025-02-12 13:55:19 INFO | data | [balcony/] GEN3: collector.Collector_Fw_Version : RSW_400_V1.00.17'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: env.Inverter_Status : 1'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_Alarm : 0'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_Fault : 0'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_Bitfield_1 : 0'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: events.Inverter_bitfield_2 : 0'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: grid.Output_Power : 490.0W'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: total.Daily_Generation : 1.26kWh'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: total.Total_Generation : 54.6kWh'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: inverter.Max_Designed_Power : 800W'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: other.Prod_Compliance_Type : 2'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: inverter.Output_Coefficient : 100.0%'
'2025-02-12 14:00:25 INFO | data | [balcony/] GEN3: inverter.Version : V5.0.17'
'2025-02-12 14:01:09 INFO | data | [balcony/] MODBUS: grid.Output_Power : 415.5W'
'2025-02-12 14:02:10 INFO | data | [balcony/] MODBUS: grid.Output_Power : 601.7W'
´´´

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@roethigj @jkrwdf @Phil1pp @s-allius