Evo2 appears to drop power on the molex, rebooting OGS -- as a way to watchdog the MobileLink module

[glenprescott]

Checklist

• I made sure that there are no existing issues or discussions - open issues,closed issues,open discussion, closed discussion - which may answer my question or that I could contribute my information to.
• I have read the Known Issues Wiki Page and my question / issue is not listed.
• I have provided the version of the software and the operating system in my post.
• If I have an issue with genmon not communicating with the controller, I have read and followed the serial troubleshooting guide.
• I have included the generator make and model in my submission.

Version

V1.19.05, under python3.12 on NetBSD

Have you submitted your log files from the About page? Note that outbound email must be working for logs to be submitted.

I have not submitted logs

What is/are your question(s)?

I have what I think is a normal setup, and have discovered strange Evo2 behavior, dropping power on the molex. I didn't notice this for a really long time, and I wonder if anyone else see it. (It's not really a gennmon issue, but relevant to many who are trying to use genmon, so it seemed ok to ask here.)

My system:

• Generac 22 kW liquid cooled, propane, installed late 2023
• Generac firmware 1.18.25
• Generac MobileLink module (ML) connected and working
• OpenGenSet (OGS, from pintsize.me), wifi AP just inside wall, good signal strength, esphome api to Home Assistant (HA)
• genmon v1.19.05 running on a computer in the house, computer and all networking on UPS
• MQTT to HA

Generally, everything works well. The genmon web page works, values make sense, remote start/stop works, notifications work. MQTT values to HA work. I get alerts from HA on loss/restore of utility, output active, state changes. Other than what I describe below, the OGS module is very reliable.

I noticed that sometimes, genmon had no data, and that opengenset is offline. Usually it is gone for 23h. Occasionally for 46h, exactly twice. as long. I started looking into it, and will spare you the mis-steps.

For instrumentation, I added a boot time sensor in esphome, and in HA an automation to notify me. I have also been looking at DHCP logs for reboot behavior from both OGS and ML.

My findings so far:

• every 23h and about 2 minutes, like clockwork, ML and OGS both reboot. Even if there are other reboots of ML or OGS, the 1-hour-earlier each day pattern is clear.
• occasionally there are other reboots. Most of my other esphome devices stay up much longer -- one, powered by lead acid batteries, has been up since mid February.
• once ,I had no internet for several hours (unrelated to generator/power). OGS rebooted every few minutes during this outage. My firewall doesn't allow OGS to talk to the Internet, and because it's esphome it doesn't try anyway. Logs showed frequent ML reboots.
• I redid my firewall for ML, blocking everything and then enabling things one at a time. When NTP doesn't work, the ML reboots after only a few minutes. When NTP works and the generac servers are blocked, it seems. to reboot after 20 minutes. I left it set to repoint ntp to my own server; usually it stays up 23h.

My theory is that there is a watchdog in the evo2, and that the ML reports in with various kinds of status, like "time sync and basic internet ok" and "talked to generac cloud ok". If those watchdogs fire, the evo2 blips power on the molex to reboot the ML. And, it does this every 23h no matter what.

I speculate that the power blip is short. Obviously one should turn off power for at least 10s to reboot devices. I am guessing that sometimes the ESP in the OGS ends up in a bad state from power dropping below safe operational levels but not entirely draining. It seems to vary over time whether this is likely, occasional or almost never.

A while after formulating this theory, my OGS went offline, at exactly one of the 23h intervals. The next day, 23h later, it was still offline. I changed my firewall to block the ML and after most of an hour the OGS came back, and I unblocked. OGS was offline again, so I re-blocked and unblocked immediately as soon as it came back. I confirmed ML reboots in the log.

My controller could be defective, but this really feels like an intentional design. OGS comes back very quickly almost always, genmon reconnects, and there are no alerts or missing data -- so this could be happening to many without being noticed.

So, my questions:

• Does this happen to you?
• If you have an OGS and could add a boot time sensor and watch it, please post what happens.
• Same if you have some other kind of computer powered from the Molex connector.
• If you have an ML, and you save its DHCP traffic , do you see DHCPDISCOVER every 23h?
• If you firewall your ML, does that cause OGS reboots?
• If you have some other kind of device powered by the molex, does it restart?

With the warming weather, I'll look at the molex power with a scope. I know actually measuring power is the next step.

Additional information

No response

[skipfire]

@glenprescott I know we've discussed this directly, but I've not noticed a power drop when using the Molex 12v powering a Pi Zero 2. I don't use MobileLink though (don't want to risk a breaking firmware coming to my generator), so the rest of the data I don't have.

[glenprescott]

Is your generator configured so that it doesn't think it has a ML, or you don't have one plugged in, or never had one? That could explain the difference. I hope we'll hear from someone else with a ML.

[skipfire]

I have one, it just isn't plugged in.

[glenprescott]

The weather and spare time coincided Monday, so I opened up the side panel. As found, the OGS and the ML connector were plugged into a pintsize splitter, which was plugged into the generator's control port. I unplugged the ML from the splitter. After a bit (most of 15 minutes), OGS rebooted, and after another few minutes it went offline. I noticed that the red power light was out. After disconnecting and reconnecting, it came back and then disappeared again. I reconected the ML and it booted but that didn't help OGS.

I then removed the splitter and plugged the OGS in directly. It has been up since Monday afternoon with zero reboots and no trouble. I noticed that the other LED was bright purple (looks like an LED with red and blue with both lit); it's much much brighter than the usual SMD chip LED that tells you an ESP32 is powered. I have no memory of seeing this before. The third thing is that the battery charging behavior is different. Now the charger current goes to zero, and after maybe 6h the charger turns on and battery is at 14, then high 13, and after 18h charger turns off and goes to 12.9 or so and then it repeats. Before, it would sit at 12.9V and 50-100 mA charge current for days.

I wonder if the ML got damaged; I had an outage last June and the disappearing OGS started then. And the charging behavior changed then but I thought maybe it was about temperature. Next step is to get the splitter and look it over carefully.

I am still really curious if anyone else has seen this 23h reboot of either OGS or ML, from looking at DHCP logs. I could well have a one-off strangeness.