The problem
A Wi-Fi hub in the house talks over 915 MHz LoRa to nodes at the equipment pad that switch the pump and heater. One hub in the field was stuck in Wi-Fi reconnect loops, and every reboot dropped the LoRa link to its nodes. The dashboard showed hubs online that were dead, so nobody could trust device health. About 5,400 lines on the hub and 1,860 on the node had been written before I arrived, and the bench work had to happen on a shipped kit with mains, relays and a 24 VAC input.
What I did
The first deliverable was a review, not code: eight risks ranked, an estimate of 27 to 49 hours, and a fixed price plan in five parts the client could buy one at a time. Reading the hub firmware closely showed it already sent an MQTT Last Will and a state message every 30 seconds. The online when dead problem was in how the cloud listened, not in the device. The presence backend is two AWS IoT rules into a dependency free Node.js 20 Lambda that patches the device row in Supabase.
Every write carries its event time so an older event can never overwrite a newer one. A node goes offline after 660 seconds, just above the firmware's own 600 second cutoff, and a scheduled sweep catches anything stale every minute. It shipped as a pull request with the SQL migration and went live on both production hubs with nothing flashed in the field.
The reconnect loop turned out to be three different loops that looked the same from outside: a setup portal timeout that rebooted every 180 seconds forever, a router outage that restarted every 5 minutes, and a cloud unreachable case that restarted every 8 minutes. After the fix a one hour router outage causes one restart instead of twelve, a 150 second Wi-Fi drop causes one reconnect and no reboot, and a node holds its relays through a 200 second hub loss and recovers alone.
The bench also found a current sense fault firmware cannot fix. A workaround was written and then withdrawn, because it would have hidden the fault, and the finding went to the board designer with probe points marked on photos.
Decisions that shaped it
- Read before writing. The review found the cloud was the problem and saved a fleet wide flash.
- Never beat the device to a verdict. The offline threshold sits just above the firmware's own.
- Blank beats wrong. A faulted sensor shows no wattage instead of the last good number.
- Say what was not proven. The test report separates proven on hardware, proven from code, and not yet tested.
Result
Presence live on both production hubs, twelve reboots an hour became one, and a validation milestone proposed for the parts not yet proven on hardware.
