Pull the Cable: Proving an Industrial Ethernet Ring Actually Fails Over

The ring converged in 200 ms and the HMI was blank for 30 seconds

Five of six link pulls recovered before the ping test dropped a packet. The sixth — the fiber between the two cabinet switches nearest the root — took about thirty seconds, and Area 2's faceplates sat greyed out for all of it.

The cause was a media converter someone had left in that span from an earlier fiber repair. It presented the port as half duplex, so RSTP could not treat the link as point-to-point, and without a point-to-point link there is no proposal/agreement handshake. The protocol fell back to the 802.1D timers: max age 20 s plus forward delay 15 s. The switch logs said "topology change" and nothing else. Nobody would have found it from the drawings, because on the drawing it was one line.

That is the whole argument for commissioning a ring by breaking it. Green link LEDs and a loop on a P&ID-style network sketch prove the cable route. They say nothing about convergence time, blocked ports, VLAN carriage on the alternate path, or how the SCADA client behaves during the gap.

Name the protocol in the record, not "Ethernet ring"

The failure modes barely overlap, so "ring redundancy: yes" in a commissioning record is worth nothing a year later.

Method	Spec	Realistic recovery	What actually bites you
RSTP	IEEE 802.1D-2004, now in 802.1Q	Sub-second on point-to-point links; 30 s+ when it falls back to legacy timers	Root bridge chosen by MAC address, non-p2p links, unmanaged switches eating BPDUs
MRP	IEC 62439-2	200 ms profile (20 ms test frames, up to 50 nodes); 30 ms and 10 ms profiles exist	Wrong or duplicate MRM role, devices attached outside the ring domain
Vendor ring	Proprietary	Usually tens of ms, per datasheet	Replacement switch ships with default firmware and no ring license
PRP / HSR	IEC 62439-3	Zero — the duplicate is already there	End devices must be DANPs or sit behind a RedBox; nobody watches the discard counters

PRP is the one people misjudge. It has no recovery time because nothing recovers — both LANs carry every frame and the receiver discards the duplicate. The corollary is that a PRP network can run for months with one LAN completely dead and give no operator-visible symptom at all. If the design is PRP, the commissioning deliverable is a monitored duplicate-discard counter, not a stopwatch.

What the ring drawing has to carry

Ring ports and non-ring uplinks. Hostname, management IP, firmware version, cabinet. Which switch is root / MRM / ring master, and which is the intended backup. VLANs per trunk. Attachment points for HMI servers, historian, PLCs, remote I/O, and the engineering workstation.

Then the part that gets left off and causes the reruns: every unmanaged switch, media converter, wireless bridge, and vendor panel switch inside the protected path. A five-port unmanaged switch added so a local HMI could share a drop will silently swallow BPDUs — the 01:80:C2:00:00:00 reserved multicast is supposed to be consumed by a bridge, and cheap devices handle it inconsistently. Downstream of one of those, the ring protocol is blind.

The test sequence

Do this with the plant in a state where a thirty-second communication gap is acceptable, because you may discover you have one. Keep a real operator client, a SCADA server, a controller, and the historian collector online — testing against a bare laptop tells you about ICMP, not about the system.

1. Sync the switch clocks first. NTP against the same source the SCADA servers use. Comparing logs across switches whose clocks differ by four minutes is a waste of the outage window.
2. Back up every managed switch config and record firmware versions.
3. Record the steady-state topology: blocked port, root bridge ID, MRM/client roles. Photograph or export it — this is the reference you compare against after every future switch swap.
4. Start a continuous ping to a PLC and to an HMI server, timestamped, from a fixed test station.
5. Break one ring link. Physically unplug it; do not administratively shut the port. A shutdown skips the link-loss detection path you are actually trying to test.
6. Record recovery time, and record it twice: Layer 2 convergence from the switch logs, and the operator-visible gap from the HMI.
7. Restore the link and watch for a second interruption. Reversion is where poorly planned RSTP topologies hurt — the return to the preferred path is another topology change.
8. Repeat on a different link, including one adjacent to the root or MRM. Breaks near the root behave differently from breaks at the far side of the ring.
9. If the design claims switch-failure tolerance, power-cycle a non-critical switch. A link break and a switch death are not the same test.
10. Export switch event logs while they are still in the buffer.

Measure what the operator saw

Ping loss is the weakest signal in this test, because every protocol above it has a different timeout and a different recovery cost.

OPC UA. A short gap costs nothing; a long one costs a subscription. The server tears down the subscription when no PublishRequest arrives within RevisedLifetimeCount × publishing interval, and Part 4 requires that lifetime to be at least three times the keep-alive count. With a 1 s publishing interval and a lifetime count of 60, you have 60 s of slack. With a 250 ms interval and a lifetime of 30, you have 7.5 s — and past that the client has to recreate every monitored item, which on a 20,000-item subscription is where the thirty-second faceplate freeze comes from, not from the switch.

Modbus TCP. Drivers typically time out at 1000 ms with 2–3 retries, so the device flips to bad quality in about three seconds. The nastier case is the half-open socket: the client keeps writing to a TCP connection whose peer is now unreachable by that path, and on Linux the default tcp_retries2 lets that limp along for roughly fifteen minutes before the stack gives up. The ring recovered in 200 ms; the driver did not, because it was never told the socket was dead. If you see one device stay bad long after everything else came back, look at the socket, not the ring.

Historian. Store-and-forward buffering means the trend fills in later and looks clean the next morning. Check for duplicate samples and for timestamp source — collector time versus device time — before you sign it off.

So write the result in operator terms. Not "RSTP converged." Something like: "SW-03 port 7 unplugged 14:22:31. Area 2 HMI bad quality 3–5 s, command buttons correctly disabled throughout. OPC UA subscription survived. Historian backfilled 4 s, no duplicates. Reversion at 14:31 caused a further 2 s gap." That is a record someone can compare against next year.

Four things that break it

Root bridge by default MAC address. Leave every switch at priority 32768 and the root is whichever box has the lowest MAC — usually the oldest one, often the one in the least accessible cabinet. Replace it and the active topology moves. Set the intended root to 4096 and its backup to 8192, and re-verify the actual root after every switch replacement.

Access ports treated as switch links. An operator station, a printer, or a laptop plugged into a non-edge port generates a topology change every time it boots, which flushes MAC tables ring-wide. Configure edge/portfast on access ports, and pair it with BPDU guard so that "edge port" does not become "place to plug in a loop."

A second path outside the ring. A maintenance switch, a forgotten commissioning patch cable, an office-network uplink. The intended ring protocol does not control that loop and cannot break it. Walk it down physically, then confirm against LLDP neighbours and MAC address tables — and pull every temporary cable before the final test, not after.

VLAN missing on the alternate path. Layer 2 recovers, HMI traffic works, and historian or engineering traffic is gone, which nobody notices until the next shift report. Test each VLAN and subnet the SCADA system actually uses. The management VLAN being fine proves only that you can still reach the switch.

Before you close the file

Keep the config backups, firmware versions, role assignments and priorities, the normal port-state table, the break points you tested with measured times, and the list of devices that cannot do fast recovery. Add one page nobody thinks to write: the setup steps for a replacement switch before it touches the live ring. That is the document a night-shift technician will be reading at 02:00, and if it does not exist they will plug in a factory-default switch with priority 32768 and hand you a new root bridge.