Telling a Dead MQTT Gateway from a Quiet One on the HMI
Using Last Will, birth messages and retained state so a SCADA screen shows offline, stale and healthy as three things — not one green icon that lies.
The complaint that starts most of these jobs: "the gateway shows green but the tank level hasn't moved in an hour." The green came from a TCP socket that was still open. The level came from a PLC behind that gateway that had stopped answering twenty minutes earlier. One icon was answering two different questions, so it was wrong about at least one of them.
MQTT gives you the pieces to keep those questions separate — Last Will, birth messages, retained state, and a timestamp in the payload. Wire them up carefully and the HMI can say offline, stale, and live as three distinct states. Wire them up lazily and you get one green light that lies whenever the failure isn't a clean socket drop.
The one MQTT mechanism people misread
Last Will and Testament (LWT) is not something the client publishes. You register the will topic and payload in the CONNECT packet, and the broker holds it. If the client vanishes without sending a proper DISCONNECT, the broker publishes the will on your behalf. That's the whole trick: the message announcing "I'm gone" is stored ahead of time, because a device that just lost power can't compose its own obituary.
The part that trips people up is when the broker decides you're gone. Per MQTT 3.1.1 (OASIS, section 3.1.2.10), if the broker receives no packet from the client within 1.5 × keepalive, it must drop the connection and fire the will. With the common keepalive of 60 s, that's up to 90 seconds of silence before the will appears. Pull a network cable and your HMI stays green for a minute and a half. If that's too slow for a critical skid, drop the keepalive — 20–30 s is reasonable on a stable LAN — and accept the extra PINGREQ traffic.
A clean shutdown is the opposite path: the client sends DISCONNECT, the broker discards the will without publishing it, and nothing announces the stop. So if you want planned stops to show on the HMI, the client has to publish its own online=false message before it disconnects. The will only covers the ungraceful exits.
What actually goes on the wire
Keep the online-state payload small but not empty. A bare true/false throws away everything you need at 2am:
{
"online": true,
"source": "packaging-gw-02",
"ts": "2026-06-11T14:08:31Z",
"seq": 18422,
"reason": "birth",
"sw": "1.7.4"
}
ts— when the gateway generated the status, in UTC. Not when the HMI noticed. This is the field that lets you compute staleness later.seq— a counter that moves on every publish. If it stops moving whileonlinestaystrue, the source is quiet, not dead. If it jumps backward, the gateway restarted.reason—birth,planned_stop,unexpected_disconnect,maintenance. The will payload should carryunexpected_disconnect; your own pre-shutdown message carriesplanned_stop. The HMI alarms on the first and stays silent on the second.
Retain the status message (retain=true) so a fresh HMI subscriber gets the current state on connect instead of waiting for the next publish. Do not retain high-rate process values the same way — a subscriber that reconnects six hours later will happily paint a retained level from this morning as if it were live. Retain state; timestamp data.
Topic layout
Consistency matters more than the exact hierarchy. The one rule I hold to: online/offline state lives on its own topic, never buried inside a metric blob the HMI has to special-case.
| Topic | Purpose | Retain |
|---|---|---|
site/area/line/gw/status | Gateway online/offline (this is the will topic) | Yes |
site/area/line/gw/birth | Startup details, capabilities, sw version | Yes |
site/area/line/gw/dev/{id}/status | Downstream PLC / meter / skid health | Yes |
site/area/line/gw/dev/{id}/data | Process values | No |
site/area/line/gw/event | Connection changes, diagnostics | Short or none |
The dev/{id}/status row is the one that answers the tank-level complaint. Gateway health and source-device health are separate topics because they fail separately.
If you're on Sparkplug, this is already solved — differently
Sparkplug B builds the whole birth/death lifecycle into the spec, so don't reinvent it on top. The node registers an NDEATH as its MQTT will, and on connect publishes NBIRTH; devices behind it get DBIRTH/DDEATH. The bdSeq number ties a death back to the birth it belongs to, so a stale will from a previous session can't be mistaken for the current one — and a per-message seq (0–255, wrapping) lets the consumer detect a missed message and re-request a birth. If you're running Sparkplug, use its state model and its STATE/ primary-host topic; don't bolt your own online boolean beside it. If you're on plain MQTT, none of that exists and you build it yourself — which is what this article is about. The mistake is assuming plain MQTT gives you Sparkplug's guarantees for free.
Settings that bite during commissioning
- Keepalive — sets your detection floor (1.5× it, above). Tune per criticality.
- Session expiry (MQTT 5.0) — replaces 3.1.1's
cleanSession. Set it so a brief reconnect keeps the subscription and queued QoS 1/2 messages, but a device gone for hours doesn't hoard a giant offline queue. - Will Delay Interval (MQTT 5.0, section 3.1.3.2) — holds the will for N seconds after disconnect. Set it just above your reconnect time and a 3-second Wi-Fi blip stops generating a false death-then-birth flap on the HMI.
- QoS — higher QoS improves delivery, it does nothing for freshness. A QoS 2 message about a value from an hour ago is still an hour old.
- Client ID — duplicates are the classic trap. Two gateways cloned from the same config share an ID and kick each other off the broker all afternoon; the symptom looks exactly like flaky network.
Field note: the duplicate-client-ID case is worth calling out because it doesn't look like a config error. It looks like intermittent packet loss, and people chase switches and cables for a day before noticing the two devices' disconnect timestamps line up perfectly.
On the HMI, derive staleness — don't trust the boolean
online=true is a hint, not proof of fresh data. For anything that matters, derive a Data_Stale tag from timestamp age or seq movement: if now - ts > 3 × publish_interval, flag stale even while online is still true. A device can be perfectly connected and publishing nothing new because its source PLC is stopped, the poll driver faulted, or the gateway app is wedged.
The rest of the subscriber-side rules:
- Offline sources get a distinct stale/comms color, never a normal process color that reads as "fine."
- Alarm on
unexpected_disconnectafter a short debounce; treatmaintenance/planned_stopas an event, not an alarm. - Log every online↔offline transition to the historian — that record is what turns "it's been flaky" into a timestamp you can correlate.
- Show broker-path failures separately from device-level wills. "The broker is unreachable" and "one PLC died" are different work orders.
The six tests I actually run
Watching a console is not a test. Run these with a timestamped checklist:
- Start the publisher — birth appears with the retain flag set.
- Connect a fresh subscriber — it immediately sees the retained current state.
- Stop the app cleanly — planned-offline shows; the will does not fire.
- Kill power / pull the cable — after 1.5× keepalive, the broker fires the will.
- Restart the broker — clients recover and old retained data is not painted as live.
- Drop the downstream PLC while the gateway stays up — source-device status changes on its own, gateway status doesn't.
Case 6 is the one everyone skips and the one the original tank-level complaint was about. If your test plan doesn't separate case 4 from case 6, your HMI won't either.