How to Test SCADA Server Switchover Without Fooling Yourself

The failover that passed on paper

The switchover took about four seconds. The standby went green, operator screens repainted, everyone signed the test sheet and went to lunch.

Three weeks later the monthly production report came back with a 26-minute hole in flow totals on the night of the test. The standby node's historian collector had been installed but never pointed at a valid store-and-forward buffer path — the folder didn't exist, the service logged one warning at startup that nobody read, and it discarded everything it collected until someone restarted it. The redundancy diagnostic was green the whole time. It was telling the truth: the two servers could see each other's heartbeat. That is all that icon has ever meant.

So the useful question for a switchover test isn't "did it switch." It's "which of these survived the switch": operator visibility, alarm state, data continuity, background jobs, and the external interfaces nobody owns.

Write down what kind of redundancy you actually bought

Vendors reuse the same five words for very different architectures. Before touching anything, record what is installed — half the surprises in a failover test are architectural, not runtime.

Area	What to pin down
Server role	Hot / warm / cold standby, active-active, active-passive
Client routing	Virtual IP, DNS name, load balancer, or a client-side primary/standby list
Data source routing	PLC drivers on both nodes, a shared OPC server, or a gateway in front
Database path	Local replication, one shared SQL instance, or a collector pair
File path	Shared project folder, replicated runtime files, or per-node local copies
License model	Floating, dongle, node-locked, or a specific redundancy SKU

The recurring failure mode here is asymmetry. The standby was built six months after the primary, by a different person, and it is missing exactly one thing: an ODBC driver, a client certificate in the trusted store, an outbound firewall rule to the MES box, a service account that was never granted "Log on as a service." None of that shows up until the node becomes active. Diff the two machines properly — installed features, driver versions, certificates, firewall rules, environment variables, mapped drives — rather than eyeballing a services list.

Clock sync belongs in the same pass. If the pair drifts a couple of seconds apart, the event journal goes non-monotonic across the switchover, and some historians will reject or silently overwrite out-of-order samples. Both nodes on the same NTP source, and if they are domain-joined remember Kerberos gives you a 5-minute skew tolerance by default — you can be badly out of sync and still authenticate fine.

Where the client actually reconnects

The most visible failure is at the operator console, so put an engineer there — not only at the rack. Watch one normal operator station and one engineering station, because they often have different login scopes and different cached connections.

If clients follow a virtual IP, the takeover depends on a gratuitous ARP being sent and believed. Cisco switches default to a 4-hour ARP cache timeout while the MAC address table ages at 300 seconds; when the GARP gets lost, some clients happily keep unicasting to a MAC address that moved. If clients follow a DNS name, the Windows DNS client honours the record TTL, so a 3600-second TTL on that A record means an hour of clients talking to a dead server. Set it to 30–60 seconds. Both of those are network-team problems that only ever surface during a SCADA failover test.

For OPC UA, don't leave the redundancy mode implicit. IEC 62541-4 defines transparent and non-transparent redundancy, and in the non-transparent case the client is responsible for picking a server — it reads ServiceLevel and the ServerUriArray under the ServerRedundancy object and moves to the healthiest one. If your client doesn't implement that, "redundant OPC UA" means "somebody restarts the client." Also check what the session timeout was actually revised to; a client that reconnects inside RevisedSessionTimeout can reactivate its session and keep subscriptions, and one that reconnects after it starts from scratch and re-browses.

Things worth timing with a stopwatch rather than describing as "fast":

• Seconds until tag quality leaves bad/uncertain on a representative screen.
• Whether commands are blocked while quality is uncertain — they should be.
• Whether the login session survives or the operator has to re-authenticate mid-upset.
• Whether the operator can see which server is in control without opening a diagnostic page.

Write the acceptance criterion as a number before the test, not after. Mine is usually: live data back on operator screens within 30 seconds, no operator action, no re-login.

Alarm state is the part that quietly breaks

Alarm state is more than on/off. IEC 62682 (ISA-18.2) gives you a state machine with unacknowledged, acknowledged, shelved, suppressed-by-design and out-of-service, and every one of those is replicated state that the standby may or may not have received before it took over.

The symptom operators report is "old alarms came back." What actually happened is that acknowledgements from the last few seconds — or the last few hours, if replication was broken and nobody noticed — never made it across, so the standby's alarm list is a stale snapshot. Shelved alarms are the worst version of this: an alarm shelved for a known maintenance job reappears in the middle of an upset.

So capture, before and after: active alarm count, acknowledgement state for a named handful, the full shelved and suppressed list, and the timestamps and sequence order around the switchover. Then confirm the switchover itself is in the event journal — if it isn't, the next person investigating a 3am incident has no way to correlate.

If alarm history lands in SQL, check the standby's connection string and credentials specifically. A missing database driver on the standby looks perfectly healthy right up until it is active and dropping every event.

Historian gaps show up in next month's report

Live trends look fine after a switchover far more often than the stored data is fine. Query the historian after the test rather than trusting the pens.

Compare expected against stored sample count for the test window on a fast tag, and inspect raw timestamps around the switchover on a slow one. You're looking for four things: a flatline that's really a stale cached value, duplicate timestamps with different values, store-and-forward buffers that never drained, and calculated tags that restarted with the wrong context or a partial input window. Batch and lot identifiers deserve their own check — if the context attaches on the active node only, samples during the gap can end up orphaned or, worse, attributed to the previous lot.

A 60-second gap on a building utility tag is a non-issue. The same gap on a custody transfer flow or a regulated process variable is a deviation you have to write up. Decide which tags are in which category before the test, so you're judging against a rule instead of arguing afterwards.

The scariest bug is the job that runs twice

Redundant nodes carry background logic: report generation, shift summaries, email notification, cleanup jobs, interface exports, custom scripts. Failover breaks these in two directions — nothing runs, or everything runs twice.

Nothing running is a Monday-morning annoyance. Both nodes running is a production incident when the job is a recipe download, a setpoint write, or an MES transaction. For each background task, find out concretely what makes it single-instance: is it gated on the product's own "am I primary" flag, a database semaphore, a file lock, or nothing at all? "It only runs on the primary" is an assumption until you've seen the gate in the code.

Duplicate-execution risk is also the one thing worth designing for rather than testing around. Make the receiving side idempotent — a transaction key the MES rejects on repeat — and the failover question stops mattering.

Pull the plug, not just the button

A graceful role switch exercises the happy path only. If site risk allows, get at least these into commissioning:

Scenario	What it proves
Manual role switch	Product logic and the written operator procedure
Stop the primary's application service	Standby detects application death, not just host death
Reboot the primary	Client reconnect plus service startup order and dependencies
Lose the heartbeat network path	Heartbeat design and split-brain protection
Bring the failed node back	That the recovered node rejoins without grabbing control at the wrong moment

Be careful with the network isolation test specifically. If the heartbeat rides a single path with no quorum or witness, isolating it gets you split brain — two nodes both convinced they're active, both writing to the historian, both talking to PLCs. Confirm what the product does about that before you unplug anything, and if the answer is vague, run that test with the PLC link disconnected first.

Keep the evidence: start and end time, active node at each step, measured client reconnect time, alarm counts and states, the historian query results, screenshots of the diagnostics, and every issue with an owner and a retest result. Not for the audit file — for the person doing this again in two years who needs to know what "normal" looked like.

The step teams skip is the last row of that table. Failing over is the easy half; the recovered node rejoining, resynchronising its alarm and historian state, and not taking control back at a bad moment is where I've seen more real outages than in the failover itself. If you only have one test window, spend the tail of it there.