The Modbus TCP Bug Where the Socket Is Fine but the Values Are Wrong

The socket can be healthy while the replies are wrong

Modbus TCP looks simple: open port 502, send a request, read a response. On real sites the hard part is not opening the socket. It is proving that each response belongs to the request the driver thinks it sent.

That proof lives in two bytes. The MBAP header — the seven-byte prefix in front of every Modbus TCP PDU — starts with a transaction identifier at offset 0–1, big-endian. The client picks the value; the server is required to copy it back unchanged. The MODBUS Messaging on TCP/IP Implementation Guide v1.0b is explicit about this: the server "recopies" the transaction identifier of the request into the response so the client can pair them when several requests are outstanding on one connection.

Plenty of devices honor that. Some do not. Older PLC Ethernet modules, embedded serial-to-TCP gateways, cheap protocol converters, and simulators built for bench testing behave as if requests always arrive one at a time. Point one client at them slowly and they look perfect. Add load and the transaction ID stops meaning anything.

When that happens the symptoms look random:

A trend briefly shows the value from a neighboring register block, then corrects itself on the next scan.
One polling group times out while another group on the same device succeeds.
A packet capture shows responses arriving, but the driver still reports mismatched or stale data.
The trouble only starts after you add tags, connect a second HMI, or move a device to a faster scan class.

"Port 502 is open" is where the diagnosis starts, not where it ends.

Capture the MBAP header, not just the register value

Most people capturing Modbus grab the decoded register value and stop. The value is the last thing you want — by the time it is wrong, the evidence is already in the header. Pull these fields for each request/response pair:

Field	Why it matters
Transaction ID	Confirms which response belongs to which request
Unit ID	Shows the gateway target or logical slave behind the TCP endpoint
Function code	Separates FC 03/04 reads, FC 06/16 writes, diagnostics, and 0x80+ exceptions
Starting address + quantity	Shows whether block optimization changed the request shape
TCP stream	Identifies which client socket carried the exchange
Response time	Surfaces slow devices before they turn into timeouts

The tell is simple: if a response's transaction ID does not equal the request's, a correct client rejects it. A weak server gives itself away three ways — it always returns 0, it repeats a stale ID from an earlier exchange, or it swaps replies between two sockets. The dangerous case is the last one, because the register value that comes back is real data; it just belongs to a different read. The driver has no reason to flag it, so it lands on the screen as good quality.

Wireshark decodes the MBAP fields natively — filter on mbtcp and add mbtcp.trans_id as a column. Save ten seconds around the fault, not a full shift. A tight capture with the bad behavior in it beats a gigabyte nobody will open.

Does your driver pipeline requests?

This is the setting that decides whether the transaction ID matters at all. A driver that sends one request, waits for the reply, then sends the next never has two IDs in flight — so a broken server never gets a chance to swap them. A driver that keeps several requests outstanding on one connection to raise throughput exposes the weakness immediately.

Pipelining is legal Modbus TCP. It is only safe when the server actually tracks each transaction, and a lot of gateways do not. When commissioning a link to a device you do not trust yet:

Find the driver's control — it is usually called maximum outstanding requests, concurrent transactions, or parallel polls.
Pin it to one outstanding request per device (or per socket) and confirm the symptoms disappear.
Raise it only when the device manual claims support and a capture proves the IDs stay matched.
Re-test after enabling block optimization — request size and timing both change.
Re-test through any VPN, firewall, or cellular link that will exist in production, because those add reordering and latency the bench never showed.

One trap: if a gateway fronts a slow RS-485 segment, pipelining on the TCP side does nothing but grow a queue inside the gateway. The 9600-baud serial hop downstream is still the bottleneck. You get more outstanding requests and the same throughput, now with worse latency.

More clients, weaker server

A device that is solid against one SCADA server can fall apart the moment engineering tools, a redundant server, a historian, and a local HMI all connect. The Implementation Guide leaves the number of simultaneous connections up to the implementation, and cheap gateways set that number low — sometimes one clean session, with extra sockets silently accepted and served garbage.

Patterns worth ruling out:

Each client is fine alone, but their combined polling exceeds the gateway's serial capacity.
A vendor tool left a diagnostic poll running since commissioning and nobody closed it.
Two redundant SCADA nodes both poll actively instead of running active/standby.
A NAT or firewall drops idle sockets, so a client reconnects in a loop and the gateway never reclaims the old session.

Build a client inventory before you blame the device. For every IP hitting port 502, write down the polling interval, the function codes it uses, and whether it can write. An unknown client on 502 stays on the suspect list until you have accounted for it.

Set timeouts from measured response time

Copy a timeout from another site and you buy misleading diagnostics. Too short rejects slow-but-valid responses; too long lets a dead device hold a scan group for seconds while operators stare at stale data.

Measure under the load the link will actually see — normal production scan for a baseline, every important display open at once, a historian catching up after an outage, the slowest slave behind a serial gateway, and a forced network drop to time the first good response after reconnect. Then set the timeout longer than the worst-case valid response you measured, but short enough that bad quality reaches the operator quickly. Same logic on retries: one dropped packet should not raise an alarm, but three retries at a two-second timeout hides a dead device for six.

Failure modes worth memorizing

Symptom	Likely cause	Field check
Random values in one block	A response accepted for the wrong request	Compare transaction IDs and addresses in the capture
Timeouts only at high scan rate	Device/gateway can't process concurrent requests	Drop outstanding requests to 1, slow the polling group
Errors appear after a second client connects	Server connection limit or serial-side overload	Disconnect engineering tools and redundant clients one at a time
First poll after reconnect is wrong	Gateway returns a stale buffered response	Clear the socket, disable reuse, verify the first transaction ID
Writes succeed but reads time out	Read block is heavier than the light write path	Capture read and write streams separately

The instinct to bump every timeout until the alarms stop is the exact wrong move here. It quiets the alarm and leaves the wrong value on the screen — you have hidden the fault, not fixed it.

For anything you touch, leave a two-line comms note against the device: IP, port, and unit-ID rules; how many clients it really tolerates; whether concurrent transactions are safe; the timeout and retry values you landed on; and the filename of the baseline capture. The next project that bolts on a historian or an MES connector will not touch the PLC logic at all — and can still break the link purely by changing how much Modbus transaction pressure the device sees.