Every S6F11 Arrived and the Lot History Still Has Holes

A customer complaint lands. Shipment from three months ago, and they suspect one chamber. You pull the lots for that window out of MES and every PROC_END event is there — except roughly forty of them have an empty chamber id. The host log says all the S6F11 messages arrived and every one got an S6F12 back with ACKC6 = 0. The link never dropped once.

An event arriving and a history being recorded are two different things. This is where the time goes.

The event has to describe itself

The most common design mistake is leaving lot context out of the event. The reasoning is that MES already knows what is loaded, so the CEID alone is enough. In the normal flow that is true.

The ways it breaks are predictable:

• The event sits in the spool for twenty minutes. By the time it arrives, MES's "current lot" is the next one.
• A dual-port tool has two carriers docked at once. A bare CEID cannot say which.
• Someone moved a lot by hand during maintenance and MES dispatch state no longer matches the floor.

So each production event's report needs lot or unit id, equipment id with the chamber, lane or port, route step, recipe id and recipe version, carrier id, and the material lots consumed at that step. Add one more field for event quality — normal / manual / inferred / corrected / late / duplicate.

None of this means MES stops validating. It means that three months later, that one event line is enough to reconstruct what happened.

On a GEM interface this list is your RPTID design. You define reports with S2F33 and link them to a CEID with S2F35. Skip a step in that sequence and the CEID still fires — with an empty report attached. Why your GEM link is green and MES still can't reconstruct the run walks through how that failure gets built.

Decide what you are tracing first

Agree on what the site means by "lot" before anything is wired. It differs by industry, and the grain you choose lands on barcode scans, PLC data blocks, MES tables and operator screens all at once.

Object	Example	What bites you
Lot or batch	Chemical batch, food batch, pharma lot	Process conditions over a time window
Unit	Serial number, assembly id	Individual pass/fail and rework history
Carrier	Cassette, tray, pallet, rack	The carrier stays; the contents change
Panel or strip	PCB panel, lead frame strip	One parent later splits into many units
Roll or coil	Film roll, steel coil, cable reel	Useless without the length position
Container	Tote, tank, silo	Material identity changes after cleaning or blending

If the tool only sends LotStart and the requirement is serial-level history, the model is already too thin before a single wire is run. MES cannot invent that later.

Event time and receive time are different values

Bad timestamps make the process history lie. Two minutes of NTP drift is enough to show a lot leaving the dryer before it entered.

Store at least two times separately. Event time is when the state actually changed on the tool; receive time is when MES stored it. Normally they are nearly identical. They diverge by tens of minutes the moment the network drops and recovers, and that is the only moment the distinction matters — which is exactly when you need it.

In GEM, event time does not arrive for free. It comes only if a clock variable is in the report. Leave it out and the S6F11 carries no time at all, so all MES has is receive time — and forty events dumped out of the spool at once get recorded as if they all happened in the same second. The format is controlled by the TIMEFORMAT ECV. Use the 16-character YYYYMMDDhhmmsscc form. The 12-character YYMMDDhhmmss has no century and no sub-second field, so it cannot order a start and a complete that land in the same second.

The equipment clock is set by the host with S2F31 Date and Time Set Request, and the tool answers S2F32 with TIACK. Setting it once at hookup is not enough — a few months of drift is enough to reorder lots. Monitoring that drift is a traceability commissioning item, not an IT chore.

A snapshot tag is not a history

People try to build history by polling a tag like CurrentLotId — on a GEM interface, S1F3 against an SVID on a timer. Fine for a display. Useless as a record. Anything shorter than the poll interval disappears completely, and if the value changes twice between polls the middle value is recorded nowhere.

What you need is transitions: lot loaded or carrier docked, process started, recipe selected and verified, material consumed, process completed, lot unloaded — plus hold, abort, rework, skip and manual release. Those last five are the ones investigations actually need, and the ones most often left out.

Attach a sequence number to every transition. After a reconnect it is the only way to tell what was lost from what was replayed twice.

Store genealogy as structure

Split, merge and transform are the whole point of traceability. One roll is slit into several child rolls. Two ingredient lots blend into one batch. A PCB panel is later separated into individual boards. A cassette holds 25 wafers and only 8 go into the chamber. Rework consumes the same unit and produces a revised state, not a new serial.

Write that in a free-text comment field and it cannot be queried when the complaint arrives three months later. Consumed material, produced material, carrier membership, split quantity, merge quantity and effective time each need to be their own column. The only real test of the design: when one supplier lot is suspected, can you pull every affected product in seconds?

On GEM300 tools much of this is already specified. E87 defines carrier state, E90 substrate tracking, E40 process jobs and E94 control jobs, each with standard transition events. If the vendor claims compliance, map to those before inventing custom CEIDs — the homemade events are the ones that get renumbered at the next software release.

Spool replay does not preserve order

While the host is down the tool spools events. GEM gives you that much. What MES gets wrong is what comes after.

Only the streams and functions the host allowed with S2F43 are spoolable at all. The drain does not start until the host sends S6F23, and one RSDC value separates transmit from purge. During the drain the tool keeps producing live events too — so MES sees a twenty-minute-old PROC_START and a just-now PROC_END land seconds apart. Sort by receive time with no event time available and the history inverts.

What SECS/GEM spooling actually protects when the host is gone for an hour covers the tool-side failures in detail. The MES-side conclusion is one line: sort on event time, and keep receive time for audit only.

Filter at the boundary, but never drop

The integration layer should not store an obviously broken event as normal production history. A required event with no lot id, an equipment / chamber / route step / recipe id that is not in the master, a timestamp in the future, the same event id with a different payload, a complete with no matching start.

Rejecting is right; dropping is not. Keep the raw payload and the reject reason in an exception queue. A discarded event becomes missing time in the history, and missing time is something nobody ever goes looking for. One screen showing daily reject counts catches most of these quiet collapses early.

Corrections append, they do not overwrite

Missed scans, late tool events, duplicates after a restart, lots moved by hand during maintenance. That is what a real line does. A design that assumes clean input fails silently.

For anything used in quality or customer response, I don't think append-only correction is one option among several — it's the only defensible one. Keep the original event, add a correction event, show the effective value in reports with the correction history one click away. Silent overwrite cannot be defended in an audit.

What actually has to be decided is authority. Who may correct lot id, time, route step, material consumption? Can a corrected record support a release decision? When MES state and equipment state disagree, which one stops the line and who is allowed to release it? Settle this at commissioning or you will settle it during an incident.

It shows up on the operator screen

The operator needs to see which lot the system believes is loaded, and on what evidence — a scan, a PLC value, or MES dispatch. The critical one is whether a disagreement between the barcode scan, the PLC lot id and MES dispatch is visible on screen at all.

A screen that only says MES error cannot get a line back. Separate unknown lot, wrong route, duplicate event, missing material and communication loss.

The failure modes that keep coming back

History breaks on small exceptions, not on architecture.

• The tool sends ProcessComplete after unload, so the lot context has already rolled to the next lot.
• The carrier id is right but the slot map inside it is stale. An 8-wafer partial lot exposes this immediately.
• Recipe name stored, recipe version not. Six months later "that recipe" cannot be reproduced.
• Manual rework moves product outside the route model entirely.
• The barcode has leading zeros, the PLC stores it as an integer and strips them. 0012345 and 12345 become different lots.
• The historian has process values, MES has lot records, and there is no shared key to join them.
• A software update shifted the VID order inside a report by one. A host parsing by position stores every field misaligned, and nothing errors.

Provoke these during commissioning. One good lot passing through proves nothing. Put a hold on, pull the network cable, send the same event twice, and run it with the clock deliberately skewed.

Before you rely on it

1. Every event carries lot context, equipment context, route step, event time and a quality flag.
2. Reports include a clock variable, and event time genuinely survives a spool drain.
3. Equipment, SCADA and MES clocks are synchronized and the drift is monitored.
4. Start, complete, abort, hold, rework and manual release paths have all been tested.
5. Split, merge, consume and produce are stored as genealogy.
6. Duplicate, late, missing and out-of-order events are handled visibly.
7. Historian process data joins to MES lots on an explicit key.
8. Correction history is auditable and never erases the original.

Number 2 is the one most often missing, and it is always discovered after the complaint arrives.

To see what a tool actually puts on the wire before you commit to a parser, the SECS/GEM Simulator will let you point your host-side parsing at a live interface and try it.