How to Catch a Frozen SCADA Value Before the Operator Trusts It

The worst comms failure I have chased was a suction pressure that read 4.2 bar, steady and green, while the transmitter had been offline for 20 minutes. The last polled value just sat there. The operator had no reason to doubt it — the color was calm, the trend was flat, and nothing on the screen said "this number is dead." That is the failure mode heartbeat tags exist to kill: a frozen value that still looks alive.

A heartbeat is any value the remote system changes on a schedule. A watchdog is the logic that alarms when that change stops. The pair answers one question the raw value cannot: is this number current, or am I looking at a corpse?

Your protocol probably won't tell you on its own

The first mistake is assuming the driver already covers this. It depends heavily on the protocol, and the differences matter.

Modbus has no data quality. None. A Read Holding Registers either returns 16-bit words or it times out. When the TCP socket stays up but the PLC task stops executing, the last values keep coming back and the driver reports "good." If your device speaks Modbus, you must add a heartbeat — the protocol will never tell you the logic died.

DNP3 (IEEE 1815) carries per-point flags, and they are genuinely useful. Every static point ships a flags octet with ONLINE (bit 0), RESTART (bit 1), and COMM_LOST (bit 2). If your master exposes those, a point going COMM_LOST or losing ONLINE is a real quality signal — but it still tells you about the link to the outstation, not whether the outstation's own downstream logic is running.

OPC UA gives you a StatusCode and two timestamps per value. Good is 0x00000000; the ones you care about are Bad_NoCommunication (0x80310000), UncertainLastUsableValue (0x40900000), and Bad_OutOfService. Just as important: every DataValue carries a source timestamp and a server timestamp. Trend the source timestamp. If the value is Good but its source timestamp hasn't moved in 30 seconds, the server is faithfully republishing a stale sample — good StatusCode, dead data. That specific trap catches a lot of people who trust Good and stop there.

So the layered truth is: link up ≠ poll succeeded ≠ controller executing ≠ value fresh. Do not collapse all of that into one green "Comms OK" lamp. During an outage the technician needs to know which layer broke, and a single boolean throws that information away.

The tag set I actually deploy

For each remote controller, skid, gateway, or edge node, I build a small block rather than a single flag:

Tag	What it holds	Alarming
Heartbeat_Raw	Counter, bit, or timestamp straight from the device	Not alarmed directly
Heartbeat_Age_s	Seconds since Heartbeat_Raw last changed	Alarm above the engineered limit
Poll_OK	Driver/protocol status (StatusCode, DNP3 flags)	Event or alarm when false
Data_Stale	Derived boolean the graphics and alarm logic key off	Suppresses dependent process alarms
Last_Good_Update	Timestamp of the last accepted change	Shown on the diagnostic popup

The one that earns its keep is Heartbeat_Age_s. Alarming on the raw counter is useless — you'd have to know its expected value. Alarming on age is self-explanatory and reads naturally on a popup: "last good update 14 minutes ago" ends the argument about whether it's a process problem or a comms problem.

A note on the raw heartbeat itself: watch for wrap, and detect change, not increase. A UINT16 counter rolls 65535 → 0; an INT16 rolls 32767 → −32768. If your stale logic is new > old, it fires a false alarm on every wrap. The correct test is new != old plus an age timer. And a toggling bit needs its own guard — a bit stuck permanently true looks identical to a healthy bit if you only check "is it 0 or 1," so you have to confirm it actually transitioned within the window.

For a plain PLC-to-SCADA link, an incrementing unsigned integer plus whatever quality tag the driver exposes is a solid baseline. If you're on Sparkplug B, you get this partly for free — bdSeq and the MQTT Last Will NDEATH/DDEATH give you a real death certificate — but the age check still belongs on top of it.

Pick the timeout from the process, not from a template

The stale threshold follows the update rate and the cost of being wrong. There is no site-wide number that's correct.

• Local PLC on the plant LAN, 1 s heartbeat → stale at 5–10 s. Tight enough to be useful, loose enough to ride out one dropped scan.
• Cellular telemetry, 30 s publish → stale at 2–3 missed publishes, not seconds. Set this in "missed intervals," not absolute time.
• Batch/phase interface → base it on the longest legitimate silence between phases, or you'll nuisance-alarm every hold step.
• Historian collector, 5 s scan → loose enough to survive a service restart without flagging every tag stale.

The two ways to get this wrong are symmetric. Too tight and normal network jitter cries wolf until operators mute the alarm — at which point the tag is worse than useless because it's trained them to ignore it. Too loose and they run the plant on dead data for minutes. When unsure, I start on the loose side of nuisance and tighten during commissioning while watching the real jitter, not a spec sheet's.

Make "stale" change the picture, not just raise an alarm

A stale-data alarm buried in the list is easy to miss. The stale state should visibly change how the value is drawn:

• Stale analog values get a distinct quality style — hatching, a "?" badge, greyed text — never the normal live green. A low-pressure reading that's actually a failed read must not look identical to a real low-pressure trip.
• Inhibit or qualify the process alarms that depend on a stale measurement. Otherwise a dead transmitter reading 0 generates a phantom low-low that the operator chases in the field.
• Keep the comms/stale alarm in its own group, separate from process alarms, so a network event doesn't drown the real process list.
• Put Heartbeat_Raw, Heartbeat_Age_s, the StatusCode, and Last_Good_Update on the diagnostic popup. Hiding all of it behind engineering tools means every outage starts with someone RDP-ing into the SCADA server.

Prove it before handover

Reading the code is not proof. Force each condition and watch the HMI:

1. Step the heartbeat by hand and confirm SCADA sees it move and Heartbeat_Age_s resets.
2. Freeze the heartbeat while leaving the network path up — this is the case Modbus can't catch on its own, so it's the one that matters most. Confirm Data_Stale sets within the expected time.
3. Pull the network and confirm the driver-level quality (Poll_OK, StatusCode) changes, distinct from the frozen-logic case above.
4. Confirm dependent process alarms are actually suppressed or marked, not still firing.
5. Confirm the event log records both the stale transition and the recovery — you need both timestamps to compute outage duration.
6. Repeat after a PLC restart, a SCADA service restart, and a network reconnect. Restart is where wrap bugs and stuck-bit bugs surface.

Capture an event export for one healthy case, one stale case, and one recovery. Those three exports are what you hand the next engineer when they ask whether the diagnostics were ever really tested.