How to Test OPC UA Redundant Failover Before the Plant Depends On It

Two green icons prove nothing

I have signed off on plenty of OPC UA pairs where both servers showed "online" during the walkdown, and I have also been called back a month later because the HMI sat on a three-hour-old flow value while the standby quietly ran the plant. Both servers were up the whole time. Nobody had ever pulled a cable.

That is the trap with redundancy commissioning: the steady-state picture is easy, and it lies to you. What you actually have to prove is what happens in the fifteen seconds after one server stops answering — whether the client finds the other one, whether the subscription comes back with real timestamps, and whether an operator can tell the difference between "current" and "frozen." None of that shows up on a status icon.

Here is the procedure I run now, and why each step is there.

First, write down which redundancy model you actually have

OPC UA Part 4 defines redundancy in the information model through Server.ServerRedundancy.RedundancySupport, and the value it reports changes what you are testing. The enum (RedundancySupportType) is None, Cold, Warm, Hot, Transparent, and HotAndMirrored (the last added in 1.04). The line that matters for commissioning is who does the failover:

• Transparent — the server pair moves a virtual endpoint/IP and the client is supposed to notice nothing. Your test targets are the network layer: ARP/DNS update time, and whether the certificate presented after the move still validates.
• Non-transparent (Cold/Warm/Hot) — the client owns the failover. It reads Server.ServiceLevel (a Byte, NodeId i=2267) from each server in ServerUriArray and connects to the healthiest one. Here you are testing the client's logic, not just the servers.

Read RedundancySupport yourself with a generic client (UaExpert, opcua-client) instead of trusting the datasheet — I have seen products advertise "Hot" and report Warm, which means the standby isn't tracking subscription state and every failover is a full resubscribe. Record the endpoint URLs, hostnames, IPs, SecurityPolicy, MessageSecurityMode, and certificate thumbprints for both nodes while you are in there. "Primary" and "secondary" labels on a vendor screen are not network facts.

ServiceLevel is the number to trend

For non-transparent pairs, ServiceLevel is the whole game. It is a 0–255 byte where higher means "more able to serve data," and a well-behaved client watches it and switches when the active server's value drops below the standby's. The useful test is not "does failover work" — it is what makes ServiceLevel move.

Put ServiceLevel from both servers on a trend, then degrade the active one in ways short of a hard stop: pull its upstream PLC link, fill its session table, stall its historian feed. A server that keeps reporting 255 while its own source data is dead will never trigger a client failover, and no amount of network testing catches that. If the vendor lets you configure the ServiceLevel calculation, this is where you find out whether they wired it to anything real.

Test subscriptions, because that is what production clients use

Browse and read tests pass on almost anything. Your SCADA driver, HMI, and historian collector don't browse — they hold subscriptions, and subscriptions have state that a failover can quietly drop.

Two numbers decide whether a subscription survives a gap: the publishing interval and the lifetime count. The server deletes a subscription if it goes LifetimeCount × PublishingInterval without a successful Publish. So a 250 ms publishing interval with a lifetime count of 60 gives you a 15-second window — reconnect slower than that and the subscription is gone server-side, monitored items and all. Check what your client actually negotiated (the revised values, not what you requested), because servers clamp them.

During the failover itself, trend four points together — a fast-changing test tag, a slow analog, an alarm/event source, and a server-side heartbeat — and watch for:

• how long monitored values stall,
• whether the client flags quality Bad during the gap or holds the last good value,
• whether TransferSubscriptions is even attempted (in non-transparent pairs it almost never works across servers, so the client should be recreating subscriptions from scratch),
• whether event subscriptions come back at all — this is the one clients forget. Data resubscribes, alarms don't, and you lose the CommunicationFault event that would have told the operator anything was wrong.

A single static tag "recovering" proves nothing. Nothing changed, so nothing had to.

The failure that hurts is stale data, not a red X

An honest disconnect is the good outcome — the operator sees it and calls someone. The expensive one is an HMI holding a three-hour-old value at good quality, with alarm logic still evaluating it and the historian dutifully logging the same number every scan.

This is why the server-side heartbeat is non-negotiable. Have the server increment a counter tag every second. If that number freezes while quality still reads Good, you have found the failure mode that costs money, and you have found it on the bench instead of during an event. Watch specifically for the quality-laundering path: the raw OPC UA item goes Bad, but a derived HMI or calculation tag built on top of it keeps its last computed value and shows Good. That derived tag is lying to the operator, and it will pass every test that doesn't include a heartbeat.

Break it the ugly ways, not just a clean service stop

A graceful Stop-Service is one scenario, and it is the politest one. TCP gets a clean RST and most stacks reconnect fast. Real failures don't send an RST:

What you do	What the client sees	Why it's different
Stop the OPC UA service	TCP RST	Fast, clean — the easy case
Reboot the active node	RST, then dead port	Adds standby-startup race
Pull the cable / kill the switch port	Nothing — silent timeout	Client waits out session timeout; slowest recovery
Firewall-block the endpoint port	SYN with no response	Looks like a hung server, not a down one
Break DNS for the active endpoint	Resolution failure on reconnect	Exposes whether the client cached the IP

The cable-pull is the one that finds bugs, because there is no RST — the client only knows the server is gone once its session timeout expires. If RevisedSessionTimeout came back as 120 seconds, that is your worst-case detection time no matter how fast the standby is ready. Set an acceptance number here: "active-server failure detected within X seconds, values updating from standby within Y." "Failover works" is not an acceptance criterion.

Certificates, the quiet failover killer

Certificate problems love to hide until the exact moment you switch servers. The primary is trusted, everything is fine, and the standby presents a cert the client has never seen — or the right cert with the wrong name inside it.

The one that catches people: in OPC UA the certificate's ApplicationUri (the uniformResourceIdentifier entry in the Subject Alternative Name) must match the applicationUri in the server's ApplicationDescription. If the standby was commissioned from a cloned config but issued its own certificate, those can disagree, and the client rejects it as BadCertificateUriInvalid only when it finally tries to connect. Run down the list on both nodes:

• both server certs are in the client's trust list,
• the client cert is trusted by both servers,
• endpoint hostname matches the cert SAN where the product enforces it,
• ApplicationUri in each cert matches that server's ApplicationDescription,
• validity periods are current on both — and the clocks agree, because a node a few minutes fast can make a perfectly good cert read not yet valid,
• you know how each stack handles CRL/revocation checks on an air-gapped control network (many block for seconds trying to reach a CRL endpoint that isn't routable).

If a virtual hostname is in play, confirm whether the cert carries the virtual name, the physical names, or both, and write down which name the client uses on reconnect.

Leave the site something to read at 3 a.m.

The commissioning record isn't for you, it's for whoever is standing in front of the HMI during the first real failover at night. Capture the client-side settings that decide recovery — endpoint priority order, SecurityPolicy and MessageSecurityMode, session timeout and reconnect interval, subscription publishing interval and lifetime count, monitored-item queue size and discard policy, and whether failback is automatic or manual. Export the config files; during an upgrade two years out, a diff beats anybody's memory.

Then write four lines an operator can act on: what normal active/standby indication looks like, which alarms are expected during a failover, roughly how long the gap lasted in commissioning, and how to confirm which node is live. Redundancy is only worth the hardware if the site can tell a normal switchover from a real outage. Without that note, every planned failover gets logged as a mystery, and the mystery ones get ignored.