How to Keep MQTT Store-and-Forward From Punching Holes in Your Historian

The publish worked. The history still has a hole.

You reconnect the WAN after a two-hour outage, the edge gateway goes green, and the live dashboard fills right in. A week later someone pulls a shift report and finds a flat line from 02:00 to 04:00 — the exact window a compressor tripped. The publish call never failed loudly. The broker was simply unreachable, the gateway had nowhere to put the samples, and nobody had written down what should happen in that gap.

Store-and-forward is the part of an MQTT pipeline that decides whether that hole exists. It is a data contract, not something you get for free from the broker. MQTT's own session persistence doesn't save you here: a persistent session with QoS 1/2 only queues messages a broker is holding for a subscriber that will reconnect. When the publisher's broker is unreachable, there is no session to fall back on — the edge gateway has to buffer to its own storage. Clean Start and Session Expiry (MQTT 5.0), or the clean-session flag in 3.1.1, govern the subscriber side, not the publisher's outage.

So a useful edge design answers these questions before the first outage:

• Which messages are buffered?
• How long can the buffer hold normal traffic?
• Are timestamps captured at the device, edge gateway, or broker?
• How does the receiver detect gaps and duplicates?
• What happens when the link returns and old data arrives quickly?
• Which data is allowed to be dropped first when storage is full?

If these rules are not written, the site may have a nice live dashboard and a broken historical record.

Decide which data deserves buffering

Not every MQTT message needs the same persistence. Treat data classes differently.

Data class	Buffer rule	Notes
Process values for historian	Buffer with source timestamp and quality.	Usually the highest volume. Compression may happen before publish or at the historian.
Alarm and event messages	Buffer with event timestamp, state, and acknowledgement context.	Do not collapse active and return-to-normal into one final state.
Equipment state changes	Buffer in order with sequence number.	MES and downtime systems often depend on transitions.
Heartbeat or availability	Usually not buffered for long.	Old heartbeat messages can confuse recovery logic.
HMI commands	Normally not store-and-forward.	A delayed command can be dangerous unless explicitly designed as a queued job.
Configuration changes	Buffer or journal carefully.	Include user, time, old value, new value, and result.

For SCADA, the dangerous mistake is buffering everything equally. A week of heartbeat messages can crowd out one hour of production measurements if the queue is not prioritized.

Use source timestamps, not reconnect timestamps

When a gateway reconnects, it may publish thousands of old samples in a short burst. The receiving historian must store those samples at the time they were measured, not at the time they were delivered.

Each buffered message should include:

• Source timestamp.
• Gateway receive timestamp if different.
• Quality or reason code.
• Sequence number or monotonic counter per topic, asset, or stream.
• Payload schema version.
• Optional publish attempt count for diagnostics.

Avoid using only broker arrival time for historical data. Broker time is useful for transport diagnostics, but it is not the process time.

Add sequence numbers for gap detection

Timestamps alone are not enough. Two samples can share a timestamp, clocks can move, and some devices report only coarse time. A sequence number lets the receiver see missing or duplicated messages.

A practical sequence design:

• Keep a separate counter per logical stream, such as one equipment unit or one tag group.
• Increment for every published data message in that stream.
• Persist the counter across gateway restart if possible.
• Include a boot id or session id if the counter resets on restart.
• Let the receiver tolerate duplicates but flag gaps.

Assume duplicates. QoS 1 (at-least-once) redelivers whenever an ACK is uncertain, so the same sample lands twice after a flaky link; only QoS 2 is exactly-once, and most brokers and Sparkplug deployments run QoS 1 for throughput. The receiver needs an idempotency key — asset + stream + seq works fine — not an assumption that each message arrives once.

If you already run Sparkplug B, most of this is built in. Every payload carries a seq (0–255, rolling, reset to 0 on the NBIRTH that follows each connect), and the NBIRTH/NDEATH pair carries a bdSeq so the host can match a death certificate — delivered as the MQTT Last Will and Testament — back to the birth that opened the session. That is exactly the per-stream counter plus boot id described above, only standardized. Rolling your own JSON is fine too; just copy the pattern instead of trusting timestamps alone.

Example diagnostic fields:

{
  "asset": "compressor-01",
  "stream": "process-values",
  "seq": 1842331,
  "bootId": "gw-a-20260618-0915",
  "sourceTs": "2026-06-18T09:24:12.350Z",
  "quality": "good"
}

The exact field names are less important than making the rule consistent and testable.

Size the buffer from real traffic

Do not size the buffer from tag count alone. Use message size, publish rate, compression behavior, and outage expectation.

A simple estimate is:

buffer size = average payload bytes × messages per second × outage seconds × safety factor

Then add overhead for queue metadata, indexes, filesystem block size, and retained diagnostic records. If TLS certificates, broker addresses, or store files live on the same small disk, leave space for operating system logs too.

Field checks:

• Measure payload size with real JSON, Sparkplug, or binary payloads.
• Test the worst scan class, not only average production load.
• Confirm what happens when the gateway restarts with a non-empty queue.
• Confirm queue write performance on the actual industrial PC or embedded gateway.
• Watch disk wear if the gateway uses flash storage.

Control the replay rate

When the network returns, an edge gateway may flood the broker. That can overload the broker, historian connector, rules engine, or database.

Use a recovery policy:

• Limit replay messages per second.
• Send critical events before low-value samples if priorities are supported.
• Preserve per-stream order.
• Keep live data visible while old buffered data drains.
• Expose queue depth, oldest sample age, and replay status as SCADA diagnostics.

Operators should be able to tell the difference between "the link is back and the buffer is draining" and "the plant is producing old values right now."

Common failure modes

• Buffered messages are stored with reconnect time, creating a false production spike in the historian.
• Alarm active and clear messages are collapsed, so nobody sees the trip that happened during the outage.
• The queue is FIFO only, and low-value high-rate tags delay important state changes.
• Gateway restart deletes the buffer because the queue was kept only in memory.
• Duplicate messages create duplicate MES events because the receiver has no idempotency key.
• The gateway fills its disk and then loses both buffered data and current diagnostics.
• The broker accepts the burst, but the downstream historian connector drops records silently.

Commissioning tests

Run store-and-forward tests before the system is handed to operations.

Test	Expected result
Broker stopped for a short outage	Gateway buffers selected streams and reports queue depth.
Broker restored	Data replays with original source timestamps and in stream order.
Gateway restarted during outage	Persistent queue survives or the documented loss alarm is raised.
Queue reaches warning threshold	SCADA diagnostic alarm appears before data is lost.
Queue reaches full threshold	Drop policy follows the documented priority rule.
Duplicate replay simulated	Receiver ignores or marks duplicates without double-counting events.
Clock shifted on gateway	Receiver flags bad time or uses quality rules instead of corrupting history.

Capture broker logs, gateway logs, and historian records from the same test window. The useful evidence is not only that messages arrived, but that they arrived with the right time, order, quality, and count.

Practical project rule

For every MQTT edge pipeline, publish a diagnostic topic or tag set for buffer health. At minimum include connection state, queue depth, oldest queued timestamp, dropped message count, replay active flag, and last successful publish time. If the SCADA screen cannot show buffer health, the site will discover store-and-forward problems only after someone opens the historian and finds missing data.