SCADA Things

← Articles
LangEN한국어
SECS/GEM/10 min read/— views

The S6F12 Nobody Sends: How a Missing Ack Loses a GEM Event Silently

Skip or malform the S6F12 and the equipment drops the event when T3 expires — no retry, no S9F9, nothing back to the host. Four captured rounds.

SECS/GEMMESSCADATroubleshooting

The S6F11 arrived. It is in the host log. On the equipment side the same event is recorded as timed-out. The socket never dropped, and the S2F41 sent afterwards was answered normally. From the host there was nothing to see. This is what it looks like when you skip S6F12 — the one reply a GEM host owes the equipment.

The equipment sends S6F11 three times. The first gets no reply and is dropped when T3 expires, the second gets an A-format S6F12 that leaves T3 running until it also expires, and only the third is acknowledged by a B 0 ack HostEquipment S6F11 W · sys 1 no S6F12 S6F11 W · sys 2 S6F12 · A 'OK' malformed S6F12 · T3 still armed S6F11 W · sys 3 S6F12 · B 0 acknowledged T3 expirydropped T3 expirydropped

The equipment opened this transaction

Nearly all the code a host developer writes has the host sending the primary and the equipment returning the secondary. S1F13/S1F14, S2F41/S2F42, S2F13/S2F14. The T3 reply timeout sits on the host, because the sender is the one holding the clock.

S6F11 runs the other way. The equipment opens a primary, sets the W-bit, and waits for the host's S6F12. So T3 for this transaction runs on the equipment side. SEMI E37's timeout parameter table lists T3 over a 1–120 s range with a typical value of 45 s — a typical, not a mandated default. Either way, the host is not the one counting those seconds. Miss the S6F12 and no timer fires on your side and no line lands in your log.

The capture below splits that situation four ways.

One socket, four S6F11s, four different answers

The equipment is secs-gem-equipment 0.1.0 from the simulator repository, run as a passive listener on 127.0.0.1:5599, SessionID 0. One thing was changed: timers.t3 set to 6 s. At 45 s a single capture takes over three minutes. No fault injection was used. The host is a Python socket client that frames and decodes every byte itself.

Define, link, enable and S1F17 go up normally first. That ordering has its own article — S2F33, S2F35 and S2F37 read as bytes.

TX S2F33 W   00 00 82 21 00 00 00 00 04 04
             01 02 B1 04 00 00 00 01
                   01 01 01 02 B1 04 00 00 02 BC 01 02 B1 04 00 00 00 01 B1 04 00 00 00 02
RX S2F34     00 00 02 22 00 00 00 00 04 04  21 01 00
TX S2F35 W   00 00 82 23 00 00 00 00 04 05  ...  CEID 10001, 10002 → RPTID 700
RX S2F36     00 00 02 24 00 00 00 00 04 05  21 01 00
TX S2F37 W   00 00 82 25 00 00 00 00 04 06  01 02 25 01 01 01 02 B1 04 00 00 27 11 B1 04 00 00 27 12
RX S2F38     00 00 02 26 00 00 00 00 04 06  21 01 00

DRACK 0, LRACK 0, ERACK 0. RPTID 700 holds VID 1 (CONTROL_STATE) and VID 2 (PROCESS_STATE), linked to CEID 10001 (START) and 10002 (STOP).

1. No reply at all

Send S2F41 START and the S6F11 follows HCACK 0 immediately.

00:05:15.361 TX S2F41 W START  00 00 82 29 00 00 00 00 04 08  01 02 41 05 53 54 41 52 54 01 00
00:05:15.362 RX S2F42         00 00 02 2A 00 00 00 00 04 08  01 02 21 01 00 01 00
00:05:15.362 RX S6F11 W       00 00 86 0B 00 00 00 00 00 01
                              01 03 B1 04 00 00 00 01
                                    B1 04 00 00 27 11
                                    01 01 01 02 B1 04 00 00 02 BC 01 02 A5 01 05 A5 01 03

Read the header: byte 2 is 86 — W-bit 1 plus stream 6, byte 3 is 0B, function 11. SystemBytes are 00 00 00 01, not the 00 00 04 xx range the host was using. This is a primary the equipment opened with its own numbering. The body is <L [3] <U4 1> <U4 10001> <L [1] <L [2] <U4 700> <L [2] <U1 5> <U1 3>>>>>: DATAID 1, CEID 10001, RPTID 700 carrying 5 and 3.

The host then sent nothing for 9 s. So did the equipment. No retransmission, no S9F9, no socket close. The host log is completely empty across those 9 s.

Only the equipment log records it:

00:05:15.362Z  collection-event ceid=10001 sys=1  outcome=sent
00:05:21.369Z  collection-event ceid=10001 sys=1  outcome=timed-out
00:05:21.369Z  anomaly  T3 (S6F11) timeout sys=1

6.007 s — the configured T3. The event is discarded at that moment and never re-sent. MES will not have it, ever, and the only side that knows is the equipment.

2. A reply with the wrong body format

This is the expensive one. The second S6F11 (sys 2, CEID 10002) got an S6F12 whose ACKC6 went out as ASCII instead of binary. That is a common real-world host bug: the ack value is carried around as a string and serialised as one.

00:05:24.374 RX S6F11 W    00 00 86 0B 00 00 00 00 00 02  ...
00:05:24.374 TX S6F12      00 00 06 0C 00 00 00 00 00 02  41 02 4F 4B

SystemBytes 00 00 00 02 is exactly right. The W-bit is correctly 0. One thing is wrong, the body: 41 02 4F 4B is <A "OK">.

The equipment takes it like this:

00:05:24.374Z  anomaly  malformed S6F12 sys=2
00:05:30.379Z  collection-event ceid=10002 sys=2  outcome=timed-out
00:05:30.379Z  anomaly  T3 (S6F11) timeout sys=2

It received an ack and did not stop T3. Six seconds later the transaction expires and the event is dropped. Nothing came back to the host — no S9F7, no SxF0, nothing. The host log holds one line reading "S6F12 sent" and that is all.

From host code, rounds 1 and 2 are indistinguishable. Both are silent. The only difference is that round 2 leaves evidence in the host log that acks are being sent, which is what sends the investigation off in the wrong direction.

3. The correct reply

00:05:33.385 RX S6F11 W    00 00 86 0B 00 00 00 00 00 03  ...
00:05:33.385 TX S6F12      00 00 06 0C 00 00 00 00 00 03  21 01 00

21 01 00 — a one-byte binary item, value 0. The equipment logs outcome=acknowledged within 1 ms. Nothing answers the S6F12; a secondary has no secondary of its own.

4. ACKC6 non-zero

00:05:36.389 TX S6F12      00 00 06 0C 00 00 00 00 00 04  21 01 01
00:05:36.389Z  collection-event ceid=10002 sys=4  outcome=rejected ackCode=1

T3 stops. The reply counted as a reply. But that is the end of it — eight further seconds produced no retransmission. A non-zero ACKC6 is a record that the host refused, and the only way back to that data is a separate path such as S6F19/S6F15. ACKC6 has no field for a reason, which is covered in the article on event report structure.

All four look the same from the host

Nothing in the bytes the host received separates rounds 1, 2 and 4. The equipment never complains to the host. So what has to change is on the host side, and it is a log line.

  • Record the arrival time, SystemBytes and CEID of every S6F11.
  • Record the time the S6F12 went out and the bytes actually written to the socket. Log only "ack sent" and round 2 stays invisible forever.
  • The only key that joins those two lines to the equipment's own event log is SystemBytes. CEIDs repeat and the clocks differ.

If a database commit sits inside the S6F12 path, it blows up here too. Even at 45 s, a synchronous host commit can run longer, and then the equipment drops events while the host is perfectly alive. Send the ack first and persist afterwards. A failed write stays a host-side problem; a T3 expiry is unrecoverable loss.

What I did not verify

  • What E5 requires of equipment that receives a non-zero ACKC6 is not something I checked in the standard text. The behaviour above is this implementation's.
  • This equipment sent no S9F9 on T3 expiry. E5 has an S9F9 for a transaction timer timeout, but whether any equipment sends one is not something this capture can tell you. Why stream 9 cannot be relied on is separate.
  • Whether GEM spooling covers this case was not checked. Spooling assumes a lost connection, and here the socket stayed up throughout. What spooling actually protects is its own article.
  • The public demo listener (EQ1, 5501) cannot produce this capture. It does not send S6F11 to a socket client.

For the host side

  1. Always answer an S6F11 with an S6F12. A primary with the W-bit set demands a reply, and this is the one reply the host owes.
  2. ACKC6 is one binary byte: 21 01 00. Not a string, not a U1, not a list.
  3. Copy the SystemBytes from the S6F11 into the S6F12. Looking for that value in the host's outstanding transaction table will not find it.
  4. The W-bit on S6F12 is 0.
  5. If you are going to send a non-zero ACKC6, log the reason on your own side. The equipment never learns it.
  6. Keep slow I/O out of the ack path. T3 here is being counted by the equipment, not by you.
  7. Keep an S6F11/S6F12 record paired on SystemBytes and reconcile it against the equipment log periodically. Missing events show up nowhere else.

Every frame above is a byte that actually crossed a loopback socket. Both sides' logs and the raw export are in content/demos/s6f12-missing-ack-loses-event.json. To reproduce it, point a socket at the SECS/GEM simulator and run the path where your host deliberately does not answer an S6F11. That is the only way to find out whether that branch in your code has ever executed.