Designing SCADA Alarm Callouts That Actually Wake Someone Up

A lift station lost both pumps at 02:14. The SCADA server raised a critical alarm, the notification service sent an SMS to the on-call technician, and the wet well overflowed at 04:50. Everyone assumed the technician had slept through it.

He had not. The message reached his phone at 04:31 — two hours and seventeen minutes after it was sent, sitting in a carrier queue the whole time because the site's callout ran through an email-to-SMS gateway. The alarm system worked perfectly. The delivery channel was a best-effort mail path with no receipt, no timeout, and nothing watching it.

That is the thing most callout designs get backwards. The engineering effort goes into which alarms get sent, and almost none goes into proving they arrived and that somebody answered. Both halves fail in the field, but the second half fails silently.

Everything that pages out is part of the alarm system

ISA-18.2 puts remote notification inside the alarm system lifecycle, which is not a bureaucratic detail. It means every alarm on the callout list inherits rationalization, documented operator response, testing, and performance monitoring. A notification list assembled by copying the "Priority 1" filter into a config screen has been rationalized for a control room with a human sitting in it, not for a phone on a nightstand.

Those are different questions. The control room asks is this important enough to interrupt the operator. The callout asks is this important enough to get someone out of bed and into a truck. A compressor discharge temperature high alarm is a genuine Priority 1 during the day, when someone can walk over and unload the machine. At 02:00 with the plant down and the compressor already tripped, it is a thing to read over coffee.

My filter for the page list is three questions, all of which have to be yes:

Does something bad get worse if nobody acts before the next shift?
Is there actually no one on site who can act?
Can the person being paged do something about it remotely or within their drive time?

Question three kills more entries than the other two combined. Paging someone about a condition they cannot influence for ninety minutes trains them to read the message and go back to sleep — and that habit does not distinguish between the alarm they could ignore and the one they could not.

Budget the pages, not just the alarms

EEMUA 191's steady-state target of roughly one alarm per ten minutes per operator is a control-room number, and even that is generous in practice. The callout equivalent is harsher, because an overnight page has no acknowledged-and-moved-on state. It has a person waking up.

I aim for zero pages on a normal night and treat two in a week as a signal to go re-rationalize something. Once a site normalizes three or four pages a night, the on-call phone is on silent within a month, and no amount of escalation logic recovers from that. The fix is upstream: fix the chattering level switch, add the debounce, raise the trip point that was set optimistically at commissioning. It is never "add a louder ringtone."

One config detail that matters more than it looks: send the return-to-normal only for alarms you actually paged on, and send it as a separate short message. Suppressing the RTN entirely means the technician drives in for a condition that cleared itself; sending RTN for everything doubles the message volume for no benefit.

Escalation timers are human timers

The common mistake is starting the escalation clock at alarm onset. It should start at the delivery attempt, and it should be long enough for a real person to wake, find the phone, read a message, and press something.

A tier structure that has held up for me on unattended sites:

T+0 — primary on-call. One message, one voice call if the alarm is in the "get out of bed" class.
T+10 min, unacknowledged — repeat to primary, then secondary on-call.
T+20 min — supervisor or duty manager.
T+40 min — whatever the site's genuine last resort is, which on a water utility is often the operations centre and on a plant is often nobody, and it is better to know that than to leave a fourth tier pointing at a phone number that belonged to someone who left in 2023.

Five-minute escalation sounds attentive and is usually wrong. It escalates through the whole roster before the primary has their glasses on, and then three people are driving in for one alarm. Where the process genuinely cannot wait ten minutes for a human, the answer is not a shorter timer — it is an automatic action in the PLC, because no notification path is deterministic enough to be a protection layer.

Also: escalation must stop on acknowledgement, not on the alarm clearing. Those come apart constantly. A technician who acknowledges from the truck at 02:20 and arrives at 03:05 should not have woken the supervisor at 02:35 because the wet well was still high.

The delivery path has to survive the thing it is reporting

Here is the failure that embarrasses people: the notification service runs on the SCADA server, as a service on the same VM, sending through the site's internet link. Now list what it cannot tell you about — the server going down, the VM host going down, the site link dropping, the whole rack losing power. Which is a fair share of the events you most want a call about.

Two rules keep this honest:

Something outside the system has to be watching the system. A heartbeat that the SCADA writes outward every few minutes to something off-site, with a watchdog that pages when the heartbeat stops. Silence has to be an alarm condition. If your only detector of "SCADA is dead" is SCADA, you have no detector.

The notification path should not share a failure with the site link. A cellular modem on the same carrier as the site's backup LTE is one tower away from being the same path. On small remote sites I still like a standalone cellular autodialer wired to a hard contact off the UPS or the PLC's watchdog relay — it is crude, it says almost nothing useful, and it works when everything else is dark.

While you are at it, check what happens on power loss. A UPS that holds the servers for fifteen minutes but not the network switch or the modem gives you a SCADA system that is alive, alarming, and mute.

Write messages for a phone, not a log file

Assume the message is read on a lock screen, by someone half awake, possibly out of order relative to other messages.

Lead with the site and the condition, then the value and the time: PS4 WETWELL HIGH-HIGH 4.82m 02:14 - both pumps failed. Not ALM: TAG PS4_LT101_HH ACT PRI 1. The tag name belongs at the end, if at all — the person needs to decide whether to get dressed, not to look something up.

Keep it inside one SMS segment. GSM-7 gives you 160 characters, and concatenated messages drop to 153 per part and can arrive split or reordered. Always include the alarm's own timestamp in the body, because the phone's received-at time can be hours off — as the lift station found out.

And treat delivery status as data. A provider that returns delivery receipts lets you log sent, delivered, and acknowledged as three separate events. Without receipts you cannot tell a slept-through alarm from an undelivered one, and those need completely different corrective actions.

Acknowledging from the phone is a security decision

Inbound acknowledgement is the feature everyone asks for and the one I push back on hardest, because it changes the direction of traffic into the control network.

If it is genuinely needed, it belongs in the DMZ, not on the SCADA server. IEC 62443's zone-and-conduit model is the argument to make in the design review: the callout service sits in the DMZ zone, holds the only inbound conduit, and writes an acknowledgement — one bit, one alarm, per authenticated user — through the same broker or historian interface that already crosses the boundary. What it must never be is a port forwarded to the SCADA server so a mobile client can log in from the highway.

Acknowledgement from a phone also does not mean what people assume. It means "I have seen this," not "the alarm is handled." Keep them as separate states in the event log, or your alarm performance report will show a beautifully responsive night shift that never actually did anything.

When someone tells me the callout did not work, the order I check is: did the alarm actually raise in SCADA, did the notification service pick it up, did the provider accept it, did the provider report delivery, and was the on-call roster pointing at the right person that night. Four out of five times it is the last one — a shift swap that never made it into the notification config. The escalation logic gets the blame and the phone list is the bug.