Why Your Historian Shows the Pressure Dropping After the Trip

A compressor trips. The PLC stamps the trip event 09:58:12. Discharge pressure collapses, and SCADA stamps that drop 10:02:40 on receive. Now the incident report says the pressure fell four and a half minutes after the machine tripped — which makes no physical sense, and sends the reliability engineer looking for a second fault that never existed. The two clocks were four minutes apart. Nothing was wrong with the process; the process story was wrong.

A historian value with a bad timestamp isn't a slightly-off measurement. It's a false measurement that still draws a clean trend line, which is worse, because it looks trustworthy. Reports, batch reviews, first-out alarm analysis, and downtime accounting all inherit the error.

The trap is treating the historian as if it owns the clock. It doesn't. On the way from sensor to trend, a value can be stamped by the PLC, an edge gateway, an OPC UA server, an MQTT publisher, the SCADA driver, or the historian's own insert time — and any of those can be running on the wrong clock.

Who stamped the value?

Before you touch a single historian setting, figure out where time comes from on each data path. This is the question that decides everything downstream.

Timestamp source	Where it fits	What bites you
PLC / device time	Sequence events, batch markers, fast trips	If the PLC clock drifts, every event drifts with it
Edge gateway time	MQTT / protocol conversion	Gateway restamps on arrival and hides the device's clock error
OPC UA SourceTimestamp	Values from an OPC UA server	Only as good as the server; ServerTimestamp is arrival, not process time
SCADA receive time	Simple polling drivers	Poll interval and network latency get baked into the timestamp
Historian insert time	Last-resort fallback	Fine for "when did it land", useless for "when did it happen"

There's no single right answer for the whole plant. A protection trip needs source time from the device that saw it. A tank level polled every 10 seconds is fine on SCADA receive time — the 200 ms of polling jitter is noise next to a level that moves over minutes. What matters is that the choice is deliberate and written down, not whatever the driver defaulted to.

One specific thing to check on OPC UA: subscriptions return both SourceTimestamp and ServerTimestamp, and historians vary in which one they store. You usually want SourceTimestamp. If your collector quietly stores ServerTimestamp, you've turned every value into receive time without realizing it.

Pick one time architecture and make it visible

Small systems tend to grow their time sync by accident — this server points at the domain controller, that one at pool.ntp.org, the PLCs run whatever the factory shipped. It survives commissioning and then quietly rots: a firewall rule blocks UDP 123, a domain controller gets replaced, an air-gapped machine drifts unwatched for eight months.

A design you can actually maintain has a clear chain:

1. One plant-approved reference — GPS with a PPS output, the domain hierarchy, or a controlled NTP appliance. Pick one and make it the root.
2. SCADA, historian, and broker hosts sync to that reference.
3. PLCs, gateways, and field devices sync where they support it.
4. Monitoring that alarms on offset, not just on "sync failed."

That fourth point is the one people skip, and it's the one that saves you. A device whose NTP daemon is happily running but sitting 900 ms off the reference reports healthy. Alarm when the offset crosses a threshold — I use 100 ms on a SCADA LAN, tighter if the site does any sequence-of-events work — not only when sync drops entirely.

On the sync mechanism itself: plain NTP over a decent LAN holds tens of milliseconds, which is enough for the vast majority of historian trends. Windows w32time is the usual disappointment here — the default domain config is built for Kerberos, not process data, and can sit hundreds of milliseconds out; on servers that matter, point them at a real NTP source and verify the offset instead of trusting the domain default. When you genuinely need sub-millisecond alignment — SOE across bays, line-shaft motion, IEC 61850 sampled values — that's PTP (IEEE 1588), and it needs switch support (boundary or transparent clocks) to deliver it. Don't spec PTP because it sounds rigorous; spec it because a requirement forced you there.

Drift writes fiction, slowly

The compressor example at the top is the whole failure mode: a clock that's a few minutes off doesn't crash anything, it just rewrites cause and effect. And because nothing alarms, you find out during a report dispute, a year in, when a controller has been running four minutes fast the entire time.

Catch it at commissioning, cheaply. During the SAT, compare PLC time against SCADA server time directly and record the offset as a numbered line item — not buried under "comms OK." Re-check after every power cycle and firmware update, because a device with no battery-backed RTC or a dead one comes back at its epoch or resyncs from whatever it can reach. If the PLC runs local time, find out exactly how it handles the DST jump before you trust its stamps; if the platform lets you keep UTC internally, do that and never think about DST at the device again.

Some protocols hand you the drift check for free. DNP3 outstations raise IIN1.4 (NEED_TIME) to tell the master their clock wants setting, and the master answers with a time-sync write — if you see that bit stuck on, sync is broken and the outstation knows it. Watch for it.

Time zones: store UTC, display local, test the transition

Storage and display should each be explicit about zone, and the safe pattern is boring: store UTC, convert to site-local only at the display and report layer. The mess starts when that discipline slips —

• local time saved with no offset, so 01:30 during a fall-back transition is genuinely ambiguous;
• a table with a mix of UTC and local rows and no column saying which;
• a reporting layer applying a second zone conversion on top of one already done;
• alarm logs on one time base and trends on another, compared side by side;
• a CSV export with no zone stated, mailed to a customer who assumes their own.

If the product forces local storage on you, fine — but document it and actually run a value through the spring-forward and fall-back transitions before you believe it works. "It should handle DST" is not a test result.

Late and out-of-order data

Store-and-forward gateways, MQTT collectors, and flaky radio links all deliver data late, and historians disagree sharply about what to do with it. Before production, get concrete answers to:

• Will it accept an older timestamp after newer data for that tag already exists?
• Same tag, same time, new value — overwrite, reject, or duplicate?
• Do compression and deadband apply to backfilled data the same way?
• Do reports recompute when a backfill changes an earlier range, or serve stale results?
• Is there a maximum accepted age, and what happens to samples past it?

The scenario that exposes all of this: a remote pump station buffers four hours across a dropped link, then dumps it when the link returns. That has to land as four hours of historical data at its original timestamps — not four hours of "now" arriving in one burst, which is what a naive collector does, and which puts a wall of samples at one instant.

Alarms and first-out are pure timestamp problems

Alarm analysis is where a second of skew stops being cosmetic. On a slow trend, one second is nothing. On a trip, one second reorders the first-out alarm — and first-out is the whole point of the review. Before you rely on any of it, know where the alarm time is generated (PLC, SCADA alarm engine, or historian), whether acknowledge and return-to-normal times share a clock, how redundant alarm servers reconcile conflicting event times, whether an SOE module has its own sync path (many use IRIG-B or PTP precisely because SCADA-grade NTP isn't tight enough), and whether operator workstation time ever leaks into the audit trail.

When a root-cause review actually matters, use a defined first-out strategy from a synchronized source. Inferring the first cause from a pile of unsynchronized logs is guessing with extra steps.

A time test you can run in an afternoon

Don't assume sync works — prove it once, during commissioning, and keep the result with the project file:

1. Record current time on the historian, SCADA server, engineering workstation, PLC, gateway, and broker host — all at once.
2. Check NTP/PTP status and the current offset wherever the device exposes it.
3. Force a test tag to change and confirm the trend timestamp matches the source you expect.
4. Pull the network on a store-and-forward device, restore it, and inspect how the late data lands.
5. Export a trend range to CSV and confirm the time zone is unambiguous.
6. Line up an alarm event, its process value, and the PLC sequence marker and check the ordering.
7. Reboot one device and confirm it comes back on the right clock before it publishes production data.

For any historian path, you need three answers: who stamped the value, what clock that system runs, and what happens when data shows up late. If you can't answer all three, the trend will still draw its line — you just won't know whether the line is telling the truth.