What SECS/GEM Spooling Actually Saves When the Host Drops for an Hour

An MES restart ran long on a Saturday night — about forty minutes of no host. The tool kept processing, three lots finished, nobody noticed anything on the floor. Monday morning the lot-complete records were all there in MES, and every single one carried the same timestamp: 09:14, the moment the link came back. Protocol-wise nothing failed. The equipment spooled, reconnected, drained the queue, and the host acknowledged every message. The history was still garbage.

That is the normal outcome of spooling that was switched on but never specified.

Spooling is host-configured, and most projects skip that step

GEM (SEMI E30) does not let the equipment decide what to spool on its own. The host declares which streams and functions are spoolable using S2F43 Reset Spooling Streams and Functions. Send S2F43 with an empty list and you have just disabled spooling for everything.

The answer comes back in S2F44. Its first item is RSPACK: 0 accepted, 1 rejected, everything else reserved. Don't stop reading there. S2F44 also carries a list of the entries that failed — STRID, a STRACK code, and the FCNID list at fault. STRACK is 1 = spooling not allowed for this stream, 2 = unknown stream, 3 = unknown function, 4 = secondary function.

STRACK 4 is the most common and the most trivial: the host driver put S6F12 in the list instead of S6F11, and secondaries are never spoolable. Five minutes in the code. But a driver that reads the RSPACK byte and logs "spool setup failed" turns those five minutes into a two-day argument with the tool vendor.

Only primary messages the equipment originates are candidates — S6F11 event reports, S5F1 alarms, S6F1 trace data. Nothing the host sends is spooled, and secondary replies are not spooled either. So the question "what gets spooled?" is really "which SxFy pairs did we enable, and which CEIDs are linked to reports that ride on them?"

Enable S6F11 and S5F1. Think hard before enabling S6F1 trace reports — a trace with a 1-second interval will bury four hours of lot events under 14,400 samples if the spool discards by age.

My rule for picking CEIDs: spool what breaks genealogy if it is missing, not what is nice to have. Carrier in/out, lot start/complete, process start/complete/abort, recipe selected and recipe verification result, alarm set and alarm clear as a pair. Chamber-level diagnostic events usually stay out.

The state model decides what you see on reconnect

E30 defines spooling as a state machine, and it is not a two-position switch:

SPOOL INACTIVE — link is up. Messages go straight out, never touching the spool.
SPOOL ACTIVE / SPOOL LOAD — the side that puts messages into the spool. It splits again into SPOOL NOT FULL (append to the end of the spool) and SPOOL FULL (the spool is at capacity).
SPOOL ACTIVE / SPOOL UNLOAD — the side that sends spooled messages out to the host, or throws them away. Which of the two happens is the host's choice, made in the RSDC byte of S6F23, not the equipment's.

LOAD and UNLOAD are not sequential phases. They run independently over the same spool, which is exactly why an event generated halfway through a drain does not lose its place: UNLOAD empties the front while LOAD keeps appending to the back.

Vendor HMIs label the drain with their own words — "Spool Output", "Transmitting". Talk to the equipment vendor in E30's names, not in screen labels, or you will spend an afternoon discovering you meant different states.

What happens at the ceiling is set by OverWriteSpool, an E30 equipment constant: true means the oldest spooled message is dropped to make room, false means new messages are discarded and the old ones are kept. The capacity — how many messages fit — is not a host-settable constant in E30. It is vendor configuration, often a service-menu field rather than an ECID, so get the number in writing during interface design; you may not be able to read it back over the link.

What you can read back is E30's spool status variables: SpoolCountActual (messages sitting in the spool right now), SpoolCountTotal (how many have been spooled since spooling last became active), SpoolStartTime and SpoolFullTime. Sample the first one. A host that trends SpoolCountActual knows it is in an outage before anyone calls it in.

The third constant worth knowing is MaxSpoolTransmit, which caps how many spooled messages the equipment sends in reply to one S6F23. Set it and the host paces a long drain across several requests instead of taking three thousand messages in one burst. Almost nobody sets it, which is why the burst failure below keeps happening.

Neither OverWriteSpool default is right for every tool. If a tool runs 2-hour batches and the host outage window in your site's change policy is 4 hours, sizing the spool for one batch and leaving OverWriteSpool true means you keep the tail and lose the lot starts — you end up with completes that have no matching starts, which is worse for MES than losing both. I would rather set OverWriteSpool false and raise the overflow alarm, so the gap is one clean, visible hole.

The nastiest failure in this whole area is on the host side. After reconnect the equipment waits for the S6F23 Request Spooled Data that moves it into SPOOL UNLOAD: RSDC = 0 transmits, RSDC = 1 means purge. Some MES recovery routines send the purge as a housekeeping step to get the interface clean and responsive again. One S6F23 with the wrong byte and forty minutes of production history is gone, with a perfectly successful S6F24 RSDA = 0 in the log. Check what your host sends before you tune anything on the tool.

RSDA has three values too: 0 = OK, 1 = denied, busy — try later, 2 = denied, spooled data does not exist. Hosts routinely conflate them. I have seen a driver treat RSDA 1 as a hard failure and never retry, which is the opposite of what the code asks for. RSDA 2 is the useful one: after a reconnect where nothing arrives, it separates "the tool never spooled" from "my host never asked". An S6F23 against an empty spool does not quietly succeed — it comes back 2.

While the spool drains, newly generated messages get appended to the spool rather than sent live, so ordering is preserved end to end. That is the standard's behaviour, not a vendor courtesy — if your tool interleaves live and spooled messages, it is not compliant and your host's dedup logic will need to cope.

No clock variable in the report means no event time

This is the one that bit the Saturday-night case above.

An S6F11 carries CEID and the report bodies linked to it by S2F35 — defining, linking and enabling those reports is its own three-step handshake. It does not carry an event timestamp on its own. If the report definition sent with S2F33 does not include the equipment's clock SV, the only time the host ever sees is arrival time — and after a spool drain, arrival time is meaningless.

So: put the clock SV (whatever the vendor calls it — Clock, TimeStamp, EventTime) into every report you intend to spool. The TimeFormat equipment constant picks the encoding, and there are three, all ASCII:

TimeFormat	Form	Length
0	`YYMMDDhhmmss`	A[12]
1	`YYYYMMDDhhmmsscc`	A[16]
2	`YYYY-MM-DDThh:mm:ss.s` plus `Z` or `±hh:mm`	variable

Never 0 — no century, no hundredths. Take 2 if the tool has it: it is the only form that carries an offset, and it is the only one that ends the argument about what a timestamp meant when a spool drains across a DST change or across fabs in different time zones. Otherwise 1. Sync the tool with S2F31 Date and Time Set Request on every host connect, and log the delta you corrected — a tool that drifts 4 seconds a day is telling you its RTC battery is going.

Also include something that identifies the source uniquely. A shared local sequence number across two chambers is a classic: both chambers emit sequence 1047 for the same CEID within the same second, and the host's dedup logic silently drops one real lot event.

Testing it: pull the cable, don't be polite

A test that only uses an orderly HSMS Separate.req proves very little, because the equipment knows instantly that the link is gone and transitions to SPOOL LOAD cleanly. Real outages are ugly. The tool sits in NOT SELECTED or waiting on a reply while the E37 timers run out — typical defaults are T3 reply 45 s, T6 control transaction 5 s, T7 NOT SELECTED 10 s, T8 network intercharacter 5 s. Events generated inside that window are the ones most likely to fall between "not spooled yet" and "already sent, no reply."

The sequence I run at commissioning:

Record the spool capacity, OverWriteSpool, MaxSpoolTransmit, TimeFormat, and the S2F43 spoolable list. Screenshot it — this is the thing nobody can find two years later.
Sync clocks with S2F31 and note the correction applied.
Yank the network, mid-process, not between lots. Confirm the equipment enters SPOOL ACTIVE.
Run representative traffic: a lot complete, an alarm set and its clear, a recipe download and verification.
Watch SpoolCountActual climb on the equipment HMI. If you cannot see it there, that is a finding in itself.
Reconnect. Capture the whole recovery on the wire — you want the S6F23 with its actual RSDC value, not the host's summary of it.
In MES, check the recovered records for original event time, order, and lot/carrier/recipe context. Not "did the rows arrive" — "would a quality engineer trust these rows."
Repeat with the outage long enough to fill the spool, and confirm the overflow behaviour and alarm match what you configured in step 1.

Step 8 is the one that gets cut for schedule. It is also the only one that tests the branch you have never executed.

The S2F43 bytes, on a real socket

The two messages this whole article turns on are small enough to hand-build. Putting them on a socket once teaches more than reading the body layout out of E5.

Below is a real transcript against the SECS/GEM simulator's HSMS listener on 127.0.0.1:5501, driven from outside by a host client. TX is what the host sent, RX what came back. SessionID is 11 throughout.

TX  00 00 00 0A  00 0B 00 00 00 01  00 00 02 01     Select.req
RX  00 00 00 0A  00 0B 00 00 00 02  00 00 02 01     Select.rsp

Header byte 3 of the reply is 00 — SelectStatus 0, accepted. Everything below is a data message on that session.

TX  00 00 00 0C  00 0B 82 2B 00 00 00 00 02 02
    01 00

Bytes	Field	Value
`00 00 00 0C`	Length	12 — the 10-byte header plus a 2-byte body
`00 0B`	SessionID	11
`82`	Header byte 2	W-bit `0x80` set, Stream 2
`2B`	Header byte 3	Function 43
`00`	PType	0, SECS-II
`00`	SType	0, data message
`00 00 02 02`	SystemBytes	514
`01 00`	Body	`L,0` — the empty list

Two body bytes, and spooling is off for every stream on the tool. This is the message that belongs in every host connect sequence and usually is not there — and it is also what a "reset the interface" cleanup script sends without anyone reading the body.

The version that actually declares something:

TX  00 00 00 19  00 0B 82 2B 00 00 00 00 02 03
    01 01  01 02  A5 01 06  01 02  A5 01 0B  A5 01 0D

01 01 is a list of one entry, and 01 02 opens that entry's two fields — for a list, E5's length byte counts elements, not bytes. A5 is format code 51 octal (U1) with one length byte, so A5 01 06 is STRID 6. Then 01 02 again with A5 01 0B and A5 01 0D: FCNID 11 and 13. Spool S6F11 event reports and S6F13 annotated event reports, nothing else.

Then the drain request:

TX  00 00 00 0D  00 0B 86 17 00 00 00 00 02 04
    A5 01 00

86 is the W-bit plus Stream 6, 17 is Function 23, and the entire body is A5 01 00 — RSDC = 0, transmit. Change that last byte to 01 and the identical three-byte body is a purge. No confirmation step, no second field to disagree with the first. Forty minutes of history rides on one byte that no reviewer ever looks at.

All three data messages drew the same shape of reply:

RX  00 00 00 0A  00 0B 02 00 00 00 00 00 02 02     S2F0
RX  00 00 00 0A  00 0B 02 00 00 00 00 00 02 03     S2F0
RX  00 00 00 0A  00 0B 06 00 00 00 00 00 02 04     S6F0

Header byte 3 is 00: Function 0, E5's abort transaction, returned with the originating SystemBytes and no body. This listener implements the HSMS control messages and a few data streams, and spooling is not among them — so do not read these as equipment behaviour. Read them as a reply your host has to handle. SxF0 means "this transaction is over, nothing is coming"; a host that only matches on the function it expected ignores it and then sits out the full T3, 45 s by default, waiting for an answer that already arrived.

The Linktest sent straight afterwards came back in the same millisecond:

TX  00 00 00 0A  00 0B 00 00 00 05  00 00 02 05     Linktest.req
RX  00 00 00 0A  00 0B 00 00 00 06  00 00 02 05     Linktest.rsp

That is the article in four lines. HSMS was healthy the whole time; the application layer had nothing to say. Spooling starts from that judgement, not from a cable.

Failures worth knowing about in advance

The report was defined with the CEID but not the variables needed to interpret it, so MES gets "process complete" with no lot ID.
Spool survives a controller reboot on one vendor and does not on another, in the same fab, on the same GEM version.
Alarm set is spoolable but alarm clear is not (or the reverse), leaving MES with alarms that never end.
A 3,000-message drain arriving in a few seconds pushes the host past its own T3 window, the host drops the link, and the tool goes straight back into SPOOL LOAD with a partially-drained queue. MaxSpoolTransmit is the constant that prevents this, and it is almost never configured.
Spooling is enabled in the equipment configuration screen but no S2F43 was ever sent, so the spoolable list is empty and the whole feature is decorative.

The last one is common enough that it is the first thing to check. Ask for a capture of the host's connect sequence and look for S2F43 between S1F13/S1F14 and the first S6F11. If it is not there, nothing else in this article matters yet.

Where to practise the part you can

The spool filling up with forty minutes of real lot history behind it only happens on a tool. Everything else on this page you can put on a socket yourself: the SECS/GEM simulator takes the same S2F43 and S6F23 bytes, and it will also swallow a Linktest.req on demand so you can watch your host decide the session is gone on T6 rather than T3. Do that once and "the link went down" stops being a single moment.