Commissioning SCADA Firewall Rules: Test the Application, Not the Port

The screen worked all through FAT. On the production network the operator logs in, the graphic opens, and half the tags go stale after ninety seconds. Nobody touched the SCADA config. What changed is that the historian query now crosses a firewall the FAT bench didn't have, and the return path for that one collector port was never in the rule set.

That is the shape of most firewall problems at commissioning. Not "the firewall is blocking everything" — it's one missing supporting path behind an application that mostly works. So the rule set is worth treating like part of the control system, not a generic network chore. Each rule maps a conversation: who talks to whom, on which port, and why the process needs it. IEC 62443-3-3 calls these zones and conduits for a reason — a firewall rule is a conduit with an owner and a purpose, or it's tech debt waiting to be blamed.

Write the data flows before you write ports

A port list on its own is unreadable six months later. Allow 502 tells you nothing about which of eleven PLCs it's for. Start from the flow and let the port fall out of it:

Flow	Direction	What actually has to be open
HMI → SCADA server	Workstation → server	Web/thick-client port, plus the auth dependency (AD, local, or the SCADA's own service).
SCADA driver → PLC	Server → controller	Modbus TCP 502, DNP3 20000, EtherNet/IP 44818+2222, or the vendor's driver port.
SCADA → historian	Server → historian	DB or API port and the store-and-forward callback the collector uses.
OPC UA client → server	Client → server	Endpoint port (4840 default, often reassigned) plus DNS, NTP, and cert exchange.
MQTT publisher → broker	Edge → broker	8883 for TLS (not 1883), plus DNS and any bridge to a second broker.
Engineering PC → PLC	Workstation → controller	Time-boxed and role-restricted, not standing.

For every row, record source zone, destination zone, protocol, port, the system that owns it, and the process reason. A rule with no owner is the one nobody dares delete, so it lives forever.

Client-initiated is a default, not a guarantee

Most industrial protocols are client-initiated over TCP, and a stateful firewall handles the return traffic for you. Then a driver opens a callback, a license check phones home, or a discovery packet goes out UDP, and the tidy one-direction rule isn't enough.

OPC Classic (DA/DCOM) is the classic trap: it negotiates on TCP 135 and then jumps to a dynamically allocated high port, so a rule that only permits 135 lets the connection start and then hang. You either pin the DCOM port range or you don't get it working through a firewall. So before you sign off a rule, confirm on the wire:

• Which side actually opens the TCP connection — verify it, don't assume the datasheet.
• Whether the firewall is stateful for this rule, or whether you need the return direction explicitly.
• Whether the UDP discovery (OPC UA LDS, EtherNet/IP ListIdentity) is a setup-only thing or runs during operation.
• Whether a redundant pair originates traffic from both physical IPs, not just the active one.
• Which address lands in the firewall log — the node IP, the virtual IP, or a NAT translation.

And don't approve a rule because a ping got through. ICMP and your application port are unrelated: ping can be blocked while Modbus reads fine, or ping can answer while 502 is still denied.

The supporting services are where cutover dies

The application port is rarely the whole story. When something worked at FAT and fails in the isolated production zone, it's usually one of these — check them before you touch the SCADA driver:

• DNS (53). Hostname-based OPC UA endpoints and MQTT brokers resolve on the office bench and fail in a zone with no resolver path. Symptom: connect times out at name resolution, not at the port.
• NTP (123/UDP). Drift past a few minutes and TLS handshakes reject certs as not-yet-valid, event ordering scrambles, and historian timestamps lie. The failure looks like a certificate problem; it's a clock problem.
• AD / LDAP (88, 389, 636 + RPC). The HMI page loads because the SCADA server is reachable; the operator can't log in because the domain controller isn't. Two different conduits.
• Certificate services (OCSP/CRL, usually 80/443). If the client does a revocation check and the CRL distribution point is unreachable, the TLS connection stalls or drops — intermittently, which is worse.
• License server. Runtime licenses that validate at FAT quietly fail in the segmented production network. This one always surfaces at 2am on cutover night.

If policy says a dependency stays blocked, design the replacement deliberately — local accounts, a local NTP source, offline/stapled cert validation. Choosing that up front is engineering. Discovering it at startup and punching a hole to get past it is not.

NAT hides the address you're troubleshooting

NAT between plant zones, vendor gear, and the enterprise is sometimes unavoidable, and it makes every packet capture ambiguous. The addresses that break things:

• An OPC UA server advertises an internal endpoint URL in its GetEndpoints response; the client reaches the NAT address, gets handed an unreachable internal one, and the session dies after the handshake.
• The PLC rule is written for the real address, but the log shows the translated one, so it looks like no rule matched.
• The historian collector records the NAT address as the device identity, and now your point provenance is wrong.
• A failover moves the active node to an IP with no matching rule, so redundancy "works" until the day it's tested.

When NAT is in the path, put both the real and translated address in the commissioning sheet. That one column turns a two-hour capture-both-sides session into a glance.

Prove it with the protocol, not with telnet

telnet host 502 or an nmap open port proves the TCP three-way handshake completes. It does not prove the protocol works — a stateful firewall can permit the SYN and still mangle the session. Test at the application layer and keep the evidence:

• Modbus TCP: read a known register, confirm the value and the quality.
• OPC UA: browse, subscribe to a live tag, and do one allowed write if writes are in scope.
• MQTT: publish and subscribe through the real broker listener with TLS on — not a plaintext shortcut you'll forget to close.
• Historian: store a test value and read it back through the reporting path, which often crosses a different conduit than the collector.
• HMI: log in as an operator role and open a graphic backed by live data.
• Engineering access: go online or download only inside the approved maintenance window.

Grab the firewall log for the same timestamp as each test. That log line is what proves the intended rule carried the traffic — not an any-any startup rule someone left open.

The seven patterns you'll actually hit

Field troubleshooting collapses to a handful of causes. When a "working" path misbehaves, it's almost always one of:

1. Right rule, wrong source — the test laptop is permitted, the production server isn't.
2. Right rule, wrong direction — return traffic allowed, client initiation blocked (or the reverse for a callback).
3. Primary only — secondary IP, virtual IP, or failover path never got its own rule.
4. Name resolution missing — direct IP works, hostname endpoint doesn't.
5. Clock blocked — NTP denied, so TLS/OPC UA certs read as invalid.
6. Temporary rule masking — the any-any commissioning permit stays open and hides every missing detailed rule beneath it.
7. Logging off — nobody can prove which rule accepted or denied anything, so every dispute is a guess.

Number 6 is the dangerous one, because everything appears to work right up until the day the broad rule gets cleaned up. Which is why the last two jobs before turnover are: retest every path after you remove the startup permit, and confirm logging is on.

What the turnover package needs

Before handover, the firewall package should carry — for every rule — the approved source, destination, protocol, port, purpose, and approver; application-level test evidence for each critical flow; separate entries for redundant nodes, virtual IPs, gateways, and NAT addresses; the supporting-service conduits (DNS, NTP, auth, certs, license); expiration dates on temporary vendor and engineering rules; a rollback plan for changes made during production cutover; and log samples showing both accepted traffic and the deny you expect to see. Last of all, name who may request, approve, and remove rules after startup — because the rule set decays the moment that's ambiguous.