Why Your HMI Trend Draws a Flat Line Through a Ten-Minute Outage

A pump discharge pressure pen sat at 4.18 bar, perfectly flat, for eleven minutes. The operator saw a stable process. What actually happened was a managed switch rebooting after a firmware push, and during those eleven minutes the SCADA server saw nothing at all. The pressure had swung to relief in the middle of it.

Nobody caught it until the next day, when someone pulled the raw samples and found eleven minutes with no rows.

That flat line is the single most expensive trend defect I run into, and it is almost never the historian's fault. It comes from one of three places: the HMI holding the last value instead of breaking the pen, the historian storing a value but discarding the quality code that came with it, or backfilled samples getting stamped with the time they arrived rather than the time they were measured. Each one has a different fix, so it's worth knowing which you have before you file a ticket with the vendor.

First: did quality survive the trip?

Every layer in the chain has a quality concept and every layer is a chance to lose it. OPC DA carries a quality byte where 192 is good, 24 means bad/comm failure and 28 means bad/last known value. OPC UA replaced that with a 32-bit StatusCode — Bad_NoCommunication, Bad_OutOfService, Uncertain_LastUsableValue — and IEC 62541-8 is explicit that a value delivered with an Uncertain status is still a value the client must not treat as measured.

The problem is that quality is optional almost everywhere downstream. Plenty of historian interfaces are configured to store the value and drop the status. Plenty of trend controls are configured to plot value and never read the status column. If either is true on your system, no amount of correct behaviour upstream will show up on the screen.

Test it directly: force a bad quality on one tag — pull the cable, or set the OPC item to out-of-service — then query the historian raw values for that window and look at what came back. If you get rows with plausible numbers and no status field, you have found your bug and it is a configuration one, not a display one.

Held values are a display choice, and usually the wrong one

Holding the last known value is defensible on an overview screen where a blank field looks broken. On a diagnostic trend it is indefensible. A held pen and a real steady process are pixel-identical, and the person reading the trend is usually reading it precisely because something went wrong.

My rule: any trend that an engineer opens during a post-event review breaks the pen on bad quality. No dotted line, no lighter shade — a gap. Dotted lines get lost when someone screenshots the trend into an email at 60% scale. A gap survives a screenshot.

Stale is a separate case from bad. If the HMI is showing the last value because the subscription hasn't updated in 30 seconds, that is honest as long as the screen says so — a small "stale" tag next to the value, or the age in seconds. What you cannot do is hold silently.

Late samples belong at their source timestamp

This is the failure that survives commissioning because it only shows up after an outage. The edge gateway buffers during the drop, the link comes back, the buffer flushes, and the HMI draws the entire hour of recovered data as a near-vertical spike at the current time.

The mechanism is usually that the write path uses the server's receive time instead of the timestamp travelling with the sample. OPC UA gives you two — SourceTimestamp and ServerTimestamp — and history writes should be keyed on SourceTimestamp. Sparkplug B carries a per-metric timestamp in the payload for the same reason, and store-and-forward on the gateway is worthless if the receiver ignores it.

Then there's the repaint problem, which is separate and just as common. The HMI queried the window while data was still missing, cached the result, and never re-queried. The gap is now correct in the historian and wrong on screen until someone changes the time range and changes it back. If your trend control has no way to invalidate a cached window, put a visible refresh button on the screen and tell operators when to use it.

Know how far back you can actually recover

Backfill only works if the samples still exist somewhere, and the buffer depths in a typical plant vary by three orders of magnitude. A PLC data block usually holds the current value and nothing else. An edge gateway might buffer an hour, or might buffer until the disk fills. A PI or FactoryTalk Historian interface node with buffering enabled will hold for days if the disk allows. Nobody knows these numbers off the top of their head, which is why they belong in the network drawing next to the link they protect.

Write down, per data path: where samples buffer, the ceiling in time or record count, whether it drops oldest or refuses newest on overflow, and whether it preserves source timestamps and quality on flush. That last one is the one people assume.

The payoff is at maintenance planning. If the switch replacement is scheduled for four hours and the gateway buffers one, the three-hour hole in the trend is a known consequence you can decide to accept, not a surprise finding in next week's production report.

Two things that quietly corrupt the recovered window

Compression. Historians using swinging-door or exception/deviation compression decide what to keep based on the shape of the incoming stream. A burst of backfilled samples arriving out of order can be evaluated differently than the same samples arriving live, and the transitions you most want — the excursion during the outage — are exactly the points a deviation filter is inclined to throw away. If a tag matters for incident review, verify what compression does to a replayed burst before you trust it.

Time zones and DST. Store UTC, convert at display. Every system that stores local time will produce a one-hour offset on backfilled data sooner or later, and the autumn DST rollback produces an hour that genuinely occurs twice — which reads on a trend as overlapping duplicate samples. If your PLC or gateway has no time zone concept and stamps in local time, the conversion has to happen at a defined point in the chain and be documented, or two engineers will apply it twice.

Also check that everything in the chain is on the same clock. NTP to a common source is the minimum; if you're doing sequence-of-events analysis across devices, IEEE 1588 (PTP) is the answer, and IEC 61850-5 sets the expectations for accuracy classes if you're on the substation side.

Break it on purpose during commissioning

The first real outage should not be the first test. This takes twenty minutes:

1. Pick three tags: a fast analog, a slow analog, and a discrete state.
2. Record a baseline trend with everything healthy.
3. Pull the link for a known duration — long enough to exceed one poll cycle by a wide margin, short enough to stay inside the edge buffer.
4. If it's safe, change the process value during the outage. A backfill test where nothing moved proves very little.
5. Restore the link. Confirm live values resume.
6. Confirm the missing window is either backfilled at the correct timestamps or shown as a gap. Anything else is a bug.
7. Export raw samples with timestamp, value and quality, and diff against the source.

Then run it again with an outage longer than the buffer, so you see what an unrecoverable gap looks like on the screen. Operators should recognise that shape.

If production reports pull from the same historian, run one test across a shift boundary too. A gap that straddles 06:00 is where you find out the shift totalizer silently substitutes zero.

What operators need on the screen

Not the mechanism — just whether the window they're looking at can be trusted. A visible gap, a marker on the outage period, a stale label, a refresh action, and a link to the comms health page covers it.

The one thing worth adding beyond that is a pending-backfill indicator. Without it, an operator who checks the trend two minutes after recovery sees a gap, concludes the data is lost forever, and stops looking. Ten minutes later the historian filled it in and nobody went back.