How to Catch an Intermittent SCADA Comms Fault in a Packet Capture

A driver alarm at 02:14, cleared by 02:15, and by morning the trend looks fine. That ran three nights straight on one site I worked, and every meeting landed on the same two words: network issue. It stopped being a guess the night we left a capture running on the SCADA server NIC and found the PLC answering in 8 ms while the reply never appeared on the firewall's outside interface. Four packets ended a week-long argument.

A capture does not replace switch counters or driver logs. What it does is draw the line between "the application is broken" and "the path is broken" — and that line is usually exactly where the argument is.

Get it running before the fault clears

The single biggest failure mode is capturing after the event. If the problem is active and observing it is safe, start now; do not wait for the maintenance window. Five minutes taken during the fault beats a day of hypothesis.

Where to hang the capture, by symptom:

• HMI runtime or SCADA server NIC when screens freeze or the driver reports timeouts.
• PLC uplink port when several clients lose the same controller at once.
• OPC UA server interface when sessions reconnect or certificate errors appear.
• Modbus TCP gateway port when exactly one slave or one unit id goes quiet.
• Firewall inside and outside interfaces after any rule, route, or NAT change.

For something that fires once a night, use a ring buffer instead of one enormous file:

tcpdump -i eth1 -w /var/tmp/plc-%Y%m%d-%H%M%S.pcap -C 200 -W 24 -Z root \
  'host 192.168.10.25 and (port 502 or icmp)'

-C counts in millions of bytes, -W is the number of files kept, so that line costs you about 4.8 GB of disk and keeps roughly the last day. Files of 100–200 MB open fast; a 6 GB single capture does not. On tcpdump older than 4.0 add -s 0, or you will truncate the Modbus payload you came for and only notice two days later.

Write the command, the interface, the host's time zone, and the reason into the work log at the same time. Six weeks on, a pcap with no context is a file nobody can testify about.

The same timeout means four different things

Where you tapped decides what the timeout tells you:

• Request left the server, no reply came back → the fault is downstream.
• Request never left the server → driver, OS routing table, local firewall, or NIC.
• PLC replied but the HMI never saw it → VLAN, ACL, NAT, duplicate IP, asymmetric route.
• Both sides look clean → look at the application, not the wire.

Symptom	First capture point	Compare against
One HMI client is slow	HMI client NIC	SCADA server NIC
All clients lose one PLC	SCADA server / driver NIC	PLC switch port mirror
OPC UA reconnects every few minutes	Client and server NICs	Firewall session table
Exception vs timeout unclear	Gateway or PLC side	SCADA driver side
Only the remote site fails	Both ends of the WAN/VPN	Router and firewall logs

Two synchronized captures on opposite sides of a firewall or radio link are worth far more than one capture from the "obvious" side. Point both hosts at the same NTP source first, and ping across the link at the start so you have a common marker — if the clocks still drift apart, editcap -t -0.412 a.pcap a-shift.pcap will realign them for comparison.

SPAN ports lie when you oversubscribe them

Mirroring both directions of a gigabit access port onto a gigabit monitor port gives the switch up to 2 Gbps to squeeze into 1 Gbps. It drops the excess silently, and those drops show up in Wireshark as gaps that look exactly like packet loss on the network. I have watched a team chase a "loss problem" that lived entirely inside their own SPAN session.

Two habits fix it: mirror one direction at a time when the link is busy, and check the switch's monitor-session drop counter before you trust the file. For anything where the timing matters — sub-10 ms request/response, PROFINET, motion — use a passive TAP. A SPAN port also reorders and re-times frames; a TAP does not.

First pass: timing and direction, not fields

Do not start decoding protocol fields. Start with who spoke, who answered, and how long it took.

• Are requests leaving the client at the expected poll rate, or did the client go quiet first?
• Does every request get an answer?
• Are there TCP retransmissions before the application timeout, or does the timeout arrive out of a clean stream?
• Does a reset follow a specific packet size or a specific command?
• Are ARP requests repeating because the target MAC never resolves?
• Is a DNS or NTP lookup stalling an application that should be talking to a fixed IP?
• Did the source IP change because NAT, a VPN, or a second NIC got in the path?

In Wireshark, tcp.analysis.flags for the first sweep and Statistics → Conversations sorted by duration will usually put you within a few packets of the interesting moment. If the client stopped sending on its own, stop looking at the network — that is a SCADA server problem wearing a network costume.

Modbus TCP: an exception is not a timeout

These get merged in reports constantly, and they lead to opposite fixes.

An exception response means the slave received the request, understood it, and refused. Exception 02, Illegal Data Address, is the one you will see most, and nine times out of ten it is the 40001-vs-0 offset: someone configured holding register 41035 where the device wanted offset 1034. That is a register map correction, not a network ticket.

A timeout means the request, the response, or the session path failed. Different ticket entirely.

Filter with mbtcp and check the MBAP transaction identifier — it pairs each response to its request, and a gateway that reuses transaction id 0 for everything is a good reason to stop trusting the gateway's queueing. Also confirm the unit identifier: the Modbus Messaging on TCP/IP Implementation Guide says a device addressed directly by IP should be reached at unit id 255, but plenty of gateways and PLCs also answer on 0 or 1, and a driver set to the wrong one produces silence that looks like a dead link.

OPC UA: three failures that look identical from the HMI

The HMI says "disconnected" for all of them. On the wire (port 4840, opcua in Wireshark) they are clearly separate, and IEC 62541-6 defines the order they must happen in:

1. HEL / ACK — the transport handshake. If you see HEL with no ACK, you have a TCP or firewall problem, not a certificate problem. Also read the EndpointUrl in the HEL and the endpoint the server returns: a server advertising its own internal hostname to a client that cannot resolve it is a recurring, maddening failure with a one-line DNS or hosts-file fix.
2. OPN / OpenSecureChannel — security policy and certificates. A rejected certificate dies here, before any session exists. Check both trust stores; the rejection is frequently one-sided.
3. CreateSession / ActivateSession / Publish — the application layer. Slow or starved Publish responses are a server load or subscription-tuning issue and have nothing to do with the two steps above.

Treating a failed secure channel as "slow subscriptions" is how people end up tuning publishing intervals for a week to fix an expired certificate.

The patterns that keep coming back

Retransmissions stacked up before every timeout

Look at duplex mismatch first if the link is old copper — the half-duplex side logs late collisions and FCS errors, and large frames suffer worse than small ones, so the symptom often looks like "only the big reads fail." After that: wireless link quality, firewall deep inspection, switch queue drops.

Do not raise the SCADA timeout as the first move. It buries the symptom and makes operators wait longer for data that is already wrong.

Duplicate IP

Two different MACs answering ARP for one address. The symptom looks random by design: the HMI reaches the PLC, then reaches a spare controller or a contractor's laptop, then reaches the PLC again. Fix addressing and DHCP exclusions before touching a single driver setting.

Reset after an idle period

The capture shows normal traffic, a quiet stretch, then a reset or a failed session reuse. Firewalls and NAT devices commonly expire idle TCP sessions at an hour or less; Windows will not send its first keepalive until KeepAliveTime elapses, and the default is 7,200,000 ms — two hours. The session is already gone by then. Shorten the keepalive on the SCADA host, lengthen the firewall idle timeout, or make the driver reconnect cleanly. Pick one and write down which.

Broadcast and multicast noise

Older PLCs, panel HMIs, and embedded gateways have small CPUs and no traffic policing. A VLAN with no IGMP querier will flood multicast to every port, and the capture shows the device receiving streams it has no business seeing. That is a VLAN boundary, IGMP snooping, or unmanaged-switch-placement problem — not an application problem.

Asymmetric route

The request arrives, the reply leaves by a different path. Only paired captures show the missing half. Look here first after anyone adds a second NIC, a VPN, a temporary engineering laptop route, or swaps a firewall.

A pcap from a plant is evidence, not a screenshot

Captures carry IP addresses, hostnames, tag names, account names, and often live process values and recipe data. IEC 62443-2-4 places requirements on service providers for handling customer site data, and a pcap is squarely that kind of data.

Before you capture on a production site, settle: which interface and time window, whether office or internet traffic gets excluded, which controlled folder it lands in, whether the payload contains process or recipe values, and when the file gets wiped off the temporary laptop. Keep the original untouched and analyze a copy.

When vendor support asks for a capture, send the smallest file that still contains the fault plus surrounding context, with a short note: device IPs, expected behavior, actual symptom, local time of the event. That note is what saves you three rounds of email.

Write the finding as a packet fact

The analysis only matters if it changes the action list. Replace "network issue" with something a different engineer could verify:

• PLC replied in 8 ms; response absent on firewall outside interface → firewall and routing work.
• Modbus slave returned exception 02 for 41035 → register map correction.
• OPC UA connection reset during OPN after certificate rejection → trust store on both ends.
• Client sent no requests for 14 s while server CPU was pegged → SCADA server performance.

Then check the thing captures cannot see. A clean file on both sides with no loss, no resets, and answers inside 10 ms means the wire is fine and the problem is above it: driver thread starvation, a blocked poll queue, or a PLC scan that stalls under a specific recipe. That is the next place to look, and it needs different tools.