A Manual Override You Can't Audit Is a Fault You Can't Find

Six weeks of very good numbers from a dead flow meter

The magmeter on FT-110 lost its coil driver on a Tuesday night. The instrument tech did the sensible short-term thing: opened the faceplate, typed 35.0 m³/h into the manual value field, and told the shift supervisor he'd swap the transmitter on Thursday. Thursday he was on another site.

The substituted value sat there for six weeks. Nobody caught it, because it wasn't behaving badly — it was behaving perfectly. Trends were flat, the daily totaliser was reproducible to three decimals, the high-flow alarm never chattered. It surfaced when a monthly mass balance came out about 4% light and someone pulled the raw trend and saw a line you could set a ruler against.

Reconstructing who did it took most of a day. The faceplate logged the tag write but not the mode change, the station was signed in as a shared OPER1 account, and the historian had stored the manual number with the same Good quality as every real sample before it. There was no single dramatic failure in that chain. There were five small design decisions nobody made.

"Manual" is at least five different states

The most common root cause is a single generic Manual flag doing all the work. An operator looking at it can't tell whether the equipment is being driven by hand, the number on screen is fabricated, or an interlock has been stepped around. Those carry completely different risk and completely different recovery steps.

Override type	What the display should say	Why it's its own state
Command override	V-204 — HMI manual command	The sequence no longer owns the actuator.
Value substitution	FT-110 — manual value 35.0 m³/h	Trends, totals and calculations are no longer field data.
Interlock bypass	P-201 — low pressure permissive bypassed	Highest risk, must be time-limited and visible plant-wide.
Simulation	M-330 feedback simulated (commissioning)	Must be impossible to confuse with production.
Data correction	Downtime reason edited — supervisor	MES and reporting need a before/after, not a replacement.

Value substitution is the one that reaches furthest, because it escapes the HMI. If the historian receives only the final number, that number is indistinguishable from an instrument reading forever. Both OPC generations give you somewhere to put this and most integrations ignore it: OPC Classic DA has a Good — Local Override quality (sub-status 0xD8) meaning exactly "a manually entered value has been forced," and OPC UA has the Good_LocalOverride status code. NAMUR NE 107 covers the same ground from the device side with its Function check category. Use one of them, make sure your historian actually stores the quality byte rather than flattening everything to Good, and make sure the trend client renders it differently. A dashed segment or a hatched band on the trend would have caught FT-110 in a day.

What has to be on the faceplate

Override indication belongs next to the object it affects. Buried on a diagnostics page it may as well not exist — nobody opens that screen during production, which is exactly when it matters.

For an equipment faceplate: current control owner (auto sequence / local panel / HMI manual / maintenance / simulation), the reason text if one was captured, the user or role who enabled it, start time and elapsed duration, expiry if it's time-limited, and the conditions that must be true before auto can take it back.

Don't carry the state on colour alone. ANSI/ISA-101.01 is blunt about this — colour is a redundant cue, not the primary one, and roughly 8% of men have some form of colour vision deficiency. A small amber outline reads as "slightly different"; the word MANUAL in the state field reads as manual. Elapsed time matters more than people expect, too: "in manual" is neutral information, "in manual 41 h" is a work order.

Permission, and confirmation text that names the thing

Match authority to consequence rather than giving every override the same dialog.

A manual jog on a conveyor is fine at normal operator level. An interlock bypass should need a higher role, a reason, and an explicit duration. Safety-instrumented function bypasses generally shouldn't live in ordinary HMI logic at all — IEC 61511-1 expects bypasses of a SIF to be under a written operational procedure with the bypass indicated and time-bounded, which usually means the safety system owns the mechanism and the HMI only displays it. Value substitution should demand a reason and show the entered number with its engineering units in the confirmation, because a value typed into the wrong field is a class of error that no permission model catches.

Then write the confirmation dialog properly. "Are you sure?" trains people to click Yes. Bypass low pressure permissive on Pump P-201 for 30 minutes? gives them something to actually disagree with. The object, the action and the duration all in the sentence.

Two more that get missed: disable the control entirely in modes where the override is meaningless, rather than accepting the click and silently rejecting the write. And block remote override when the local panel holds ownership — an operator who thinks they have control and doesn't is worse off than one who knows they don't.

The audit record, and the half of it everyone skips

The test for an audit trail is whether someone can reconstruct the sequence without opening the PLC program. That needs the timestamp against a synchronised clock, the user and role and workstation, the object and tag path, the override type, old and new state, the manual value with units, the reason, the expiry, and the result — accepted, rejected, write failed, or auto-expired.

IEC 62443-3-3 puts this on a requirement footing: SR 2.8 calls for auditable events covering access control and configuration changes, and SR 2.11 calls for timestamps traceable to a common source. That second one is why the shared OPER1 login is a real finding and not just untidiness — an audit record that can't name a person satisfies neither.

The half that gets skipped is disable events. Sites log enables and stop there, and then can't distinguish an operator clearing an override from a timeout expiring it, a mode change dropping it, or a controller restart wiping it. Those four have very different implications for whether your override mechanism works. Log the disable with its cause.

Overrides that outlive the reason for them

Most override incidents are quiet ones — a night shift substitution that nobody hands over, a bypass that stays after the maintenance window closes, a simulation bit still true when the line goes back to production.

The conditions worth flagging: active longer than the configured duration; active while the line is in production mode; active with no open work order; still active after a controller restart; still active after the operator who set it logged out; one unit in manual while everything upstream and downstream runs auto.

Alarm systems solved this problem years ago and the pattern transfers directly. ANSI/ISA-18.2 defines shelving as an explicitly time-limited state with automatic un-shelving at expiry and a mandatory review of what's currently shelved. Do the same with overrides: default expiry rather than infinite, and a live "what's overridden right now" list that a supervisor is expected to look at each shift. If your system can't auto-expire, a daily report of everything active over 24 hours is a poor substitute that still beats nothing.

Going back to auto is the dangerous half

Clearing an override is a control transfer, and equipment moves during control transfers. This is where people get hurt.

For a command override, know in advance which sequence step will take control, where the actuator currently is, whether permissives are healthy, whether a pending command from before the override is still queued, and whether feedback timeout timers reset cleanly. The bad case is well documented: the sequence retained state from before the override, regains ownership, decides the actuator is in the wrong position, and drives it there immediately.

For a value substitution, confirm the real instrument reads Good again, the substituted number is out of every calculated tag that consumed it, the historian carries quality or an annotation across the manual interval, and any alarm that was suppressed because the input was manual is live again. That last one bites — an alarm suppressed by a value override will often stay suppressed until the alarm's own hold logic re-evaluates, which may not be on the same event.

A faceplate that shows the return path — "auto will resume at step 40; V-204 will close" — costs an afternoon of screen work and removes an entire category of surprise.

Symptom to first check

What you see	Look here first
Equipment ignores the auto sequence	Command override still set, local/remote ownership mismatch, stale manual mode
Trend is flat and suspiciously clean	Value substitution, simulation mode, or a calculation masking bad quality
Alarm didn't fire during a test	Alarm input overridden, bypass active, suppression with no indication
Override vanishes after a restart	State held in HMI memory only, not in the controller or a persistent store
Can't tell who changed it	Missing user field, shared login, unsynchronised clocks
Equipment moves the moment override is cleared	Pending command or retained step state in the sequence

The test that finds most of this

At handover, for each override class, write down who may enable it, whether a reason is mandatory, the maximum duration, where it appears on faceplates and overviews and reports, which table or event stream holds the history, and how historian quality is marked.

Then run the one test almost nobody runs: set an override, restart the SCADA server, and log back in. If the override is gone, it lived in HMI memory and your audit trail has a hole every time the server bounces. If it's still there but the indication is gone, that's worse — the plant is now running with a bypass that nothing on screen mentions.