[Bug]: WiFi only works for a couple of seconds after reconnecting

[stylesuxx]

Category

WiFi

Hardware

T-Beam

Firmware Version

#3026

Description

Follow up to #2796

After flashing artifact from #3026 WiFi is now reconnecting correctly after some time, but it only does work for a couple of seconds. I verified this by pinging the device, also DNS resolution fails when connecting to the MQTT server as can be seen from the serial log:

DEBUG | 12:05:58 50540 [Power] Battery: usbPower=1, isCharging=0, batMv=4182, batPct=100
INFO  | 12:06:11 50553 [mqtt] Using non-TLS-encrypted session
INFO  | 12:06:11 50553 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[50556820][I][WiFiClient.cpp:253] connect(): select returned due to timeout 3000 ms for fd 52
ERROR | 12:06:14 50556 [mqtt] Failed to contact MQTT server directly (1/5)...
DEBUG | 12:06:18 50560 [Power] Battery: usbPower=1, isCharging=0, batMv=4182, batPct=100

---SNIP---

DEBUG | 12:08:35 50697 [Power] Battery: usbPower=1, isCharging=0, batMv=4183, batPct=100
INFO  | 12:08:45 50707 [mqtt] Using non-TLS-encrypted session
INFO  | 12:08:45 50707 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[50721932][E][WiFiGeneric.cpp:1578] hostByName(): DNS Failed for mqtt.meshtastic.org
ERROR | 12:08:59 50721 [mqtt] Failed to contact MQTT server directly (5/5)...
DEBUG | 12:08:59 50721 [Power] Battery: usbPower=1, isCharging=0, batMv=4182, batPct=100
INFO  | 12:08:59 50721 [WifiConnect] Reconnecting to WiFi access point Takbir
[50721980][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[50721984][W][WiFiGeneric.cpp:1057] _eventCallback(): Reason: 8 - ASSOC_LEAVE
DEBUG | 12:08:59 50721 [WifiConnect] WiFi-Event 5: Disconnected from WiFi access point
WARN  | 12:08:59 50722 [WifiConnect] WiFi lost connection. Reason: 8
[50727751][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 4 - STA_CONNECTED
DEBUG | 12:09:05 50727 [WebServerThread] WiFi-Event 4: Connected to access point
[50735300][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
[50735301][D][WiFiGeneric.cpp:1098] _eventCallback(): STA IP: 192.168.1.35, MASK: 255.255.255.0, GW: 192.168.1.1
DEBUG | 12:09:12 50735 WiFi-Event 7: Obtained IP address: 192.168.1.35
INFO  | 12:09:12 50735 Using non-TLS-encrypted session
INFO  | 12:09:12 50735 Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: xxxxxxx
INFO  | 12:09:13 50735 MQTT connected

This is the approximate ping section belonging to the above serial log

From 192.168.1.3 icmp_seq=252 Destination Host Unreachable
From 192.168.1.3 icmp_seq=253 Destination Host Unreachable
From 192.168.1.3 icmp_seq=254 Destination Host Unreachable
64 bytes from 192.168.1.35: icmp_seq=255 ttl=255 time=2505 ms
64 bytes from 192.168.1.35: icmp_seq=256 ttl=255 time=1483 ms
64 bytes from 192.168.1.35: icmp_seq=257 ttl=255 time=459 ms
64 bytes from 192.168.1.35: icmp_seq=258 ttl=255 time=6.48 ms
From 192.168.1.3 icmp_seq=266 Destination Host Unreachable
From 192.168.1.3 icmp_seq=267 Destination Host Unreachable
From 192.168.1.3 icmp_seq=268 Destination Host Unreachable

Relevant log output

No response

[GUVWAF]

Can you try with a static IP and set a DNS server (e.g. 8.8.8.8) yourself? Seems there is an issue with the DNS server your DHCP server is giving to the device. It then eventually get disconnected when connecting to the MQTT server fails.

[stylesuxx]

I tried setting up static details, unfortunately I don't fully understand how to do so. I can change address_mode but how am I supposed to set network.ipv4_config? If I try to do it via web UI, it does not seem to take the settings and returns 0 for all fields. I could only find this thread of another person not being able to set it either...

[GUVWAF]

Do you have an Android or Apple device? You should be able to set it with either of those apps.

[stylesuxx]

Thank you, setting static details via Android worked fine. Unfortunately the error still persists, connection does only work for a very limited time after re-connection:

INFO  | 15:10:38 630 [mqtt] Using non-TLS-encrypted session
INFO  | 15:10:38 630 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[633020][I][WiFiClient.cpp:253] connect(): select returned due to timeout 3000 ms for fd 52
ERROR | 15:10:41 633 [mqtt] Failed to contact MQTT server directly (2/5)...
DEBUG | 15:10:51 642 [Power] Battery: usbPower=1, isCharging=0, batMv=4184, batPct=100
INFO  | 15:11:08 660 [DeviceTelemetryModule] (Sending): air_util_tx=2.457250, channel_utilization=0.000000, battery_level=100, voltage=4.184000
DEBUG | 15:11:08 660 [DeviceTelemetryModule] updateTelemetry LOCAL
DEBUG | 15:11:08 660 [DeviceTelemetryModule] Node status update: 70 online, 100 total
INFO  | 15:11:08 660 [DeviceTelemetryModule] Sending packet to phone
INFO  | 15:11:08 660 Telling client we have new packets 152
INFO  | 15:11:08 660 [SerialConsole] getFromRadio=STATE_SEND_PACKETS
DEBUG | 15:11:08 660 [SerialConsole] phone downloaded packet (id=0x45c5cd7b fr=0x94 to=0xff, WantAck=0, HopLim=3 Ch=0x0 Portnum=67 rxtime=1703257868 priority=1)
DEBUG | 15:11:08 660 [SerialConsole] encoding toPhone packet to phone variant=2, 51 bytes
DEBUG | 15:11:11 662 [Power] Battery: usbPower=1, isCharging=0, batMv=4183, batPct=100
INFO  | 15:11:11 663 [mqtt] Using non-TLS-encrypted session
INFO  | 15:11:11 663 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[670048][E][WiFiGeneric.cpp:1578] hostByName(): DNS Failed for mqtt.meshtastic.org
ERROR | 15:11:18 670 [mqtt] Failed to contact MQTT server directly (3/5)...
DEBUG | 15:11:31 682 [Power] Battery: usbPower=1, isCharging=0, batMv=4185, batPct=100

Sometimes it's the timeout, sometimes it's the DNS error that hits first. The logic seems to be to restart the WiFi device after 5 failed connection attempts to the MQTT server. The first connection to the MQTT server after restarting the WiFi device works for a couple seconds

[GUVWAF]

Did you try different DNS servers? Maybe it's a firewall issue?

The logic seems to be to restart the WiFi device after 5 failed connection attempts to the MQTT server.

Indeed:

firmware/src/mqtt/MQTT.cpp

Lines 326 to 328 in db8f8db

	if (reconnectCount >= reconnectMax) {
	needReconnect = true;
	wifiReconnect->setIntervalFromNow(0);

So, yeah, if the MQTT connection fails due to DNS failures, it is expected that it tries to reconnect WiFi afterwards.

Edit: Also, you mention the first connection to the MQTT server works for a couple seconds. Are you using the default channel (LongFast)? If so, there's a lot of traffic there and it might be that that is causing problems. You could try changing the channel and see if it stays connected for longer.

[stylesuxx]

I tried 4 different routers, 2 mobile APs, different DNS servers and couple different WAN - I'd say I can exclude any "external" networking issues at this point.

Maybe I was not completely clear - the absolutely first connection after a reset works fine for minutes, up to hours (max was probably around 2 hours). Every subsequent re-connect after the first disconnect will only work for seconds. To me it feels like a buffer filling up and causing issues from there.

Yes, I am using the default channel. I will try on a different channel and see if things change.

Update:
Disabled the primary channel as a base line. WiFi connection was not lost once within 4 hours.

[major-tweaks]

Side comment: As can be seen on the official Discord Channel Announcements,
from User MC Hamster
—
21.12.2023 02:53
MQTT server maintenance tomorrow, Thursday, starting at 9 PM Pacific.

https://discord.com/channels/867578229534359593/871558934916390953

It may be on and offline sporadically.

[stylesuxx]

@major-tweaks thanks for the heads up. I'll probably stop being a pain in the ass over the Christmas holidays anyway.

[major-tweaks]

What I can say, I've got two HelTec V3s ESP32, one via WiFi connected (WebApp), one via BT (iOS App).
The one with WiFi connection to the official MQTT is dead actually at this very moment.
Just turned MQTT on the BT one on, received new nodes and messages (via iOS WiFi Connection), hoping that there are no issues relating to that WiFi connection.. running the latest stable version 2.2.15 beta

[stylesuxx]

Should you run into an infinite WiFi reconnection loop, flash the artifacts of #3026 ;-)

[major-tweaks]

Disabled MQTT on the WiFi connected one, disabled it also on Channel 0, switched off BT also, results in a much much more responsive client.. so there IS actually some issue with WiFi and MQTT.. no problem with just BT and MQTT.

[stylesuxx]

I don't think that the issue is with WiFi and MQTT per say, I think the issue is "just" when you are subscribed to a busy channel. I now disabled the default channel and things are working fine - admittedly with only very little load on that channel. Would be interesting what the "breaking" point is exactly.

[GUVWAF]

Just turned MQTT on the BT one on, received new nodes and messages (via iOS WiFi Connection), hoping that there are no issues relating to that WiFi connection..

Not sure what you mean here. You cannot connect to the iOS app with WiFi, but there is an MQTT proxy via Bluetooth. You also cannot use WiFi and BT simultaneously. If you turn on WiFi, BT is turned off and vice versa.

I don't think that the issue is with WiFi and MQTT per say, I think the issue is "just" when you are subscribed to a busy channel. I now disabled the default channel and things are working fine - admittedly with only very little load on that channel.

Yes, when connected to the public MQTT server on the default channel, the device is basically receiving messages non-stop and may not function properly anymore. I have already opened a PR to the documentation to warn users about this (meshtastic/meshtastic#901).

Since there is not really much we can do about this, I'm closing this and converting it to a discussion if you want to discuss further.