Use MQTT Shared Subscriptions to Scale Consumers — but Keep the Historian Out of the Pool

The report was wrong by exactly half. A line 3 gateway was publishing ~1,800 telemetry messages a second, the historian dashboard looked alive, and yet the shift totals came out at roughly 50% of the counter reading on the PLC. The cause was two lines in two different config files: someone had subscribed a new report-preprocessing service to $share/line3/plant/+/telemetry, and the historian collector — added a year earlier by a different integrator — was subscribed to the same shared group. The broker did exactly what MQTT 5.0 says it should: it load-balanced the stream across both clients. Each got about half the messages. Neither logged an error.

That is the whole problem with shared subscriptions in one incident. They are a load-balancing primitive, not a data-model feature, and pointing one at a consumer that needs the complete stream fails quietly.

What the spec actually gives you

A plain subscription is fan-out: every client subscribed to plant/+/telemetry gets its own copy. That is what you want for an HMI gateway, a historian collector, and a diagnostics recorder running side by side.

A shared subscription is fan-in-then-distribute. Clients join a named group with the topic form $share/{ShareName}/{TopicFilter} and the broker hands each matching message to exactly one member of the group. This was standardized in MQTT 5.0 (OASIS MQTT Version 5.0, §4.8.2); before that it was a broker extension — Mosquitto added it in 1.6, and EMQX/HiveMQ shipped their own $share/ handling earlier — so if you are still on a 3.1.1 broker, confirm the syntax it expects rather than assuming.

The dangerous word in the spec is "exactly one." That is the load sharing you asked for, and it is also the fan-out you accidentally lost.

The one design decision: who needs a full copy?

Before touching replica counts, split your consumers into two lists.

Needs its own complete stream (never share-grouped together):

• historian ingestion — it must see every configured tag;
• alarm/event evaluation that holds state in memory;
• audit logging;
• the HMI gateway feeding operator screens;
• a commissioning recorder while you are still debugging.

Safe to run as a worker pool behind one shared group:

• stateless payload/schema validation;
• enrichment writing to a DB or queue with idempotent keys;
• report preprocessing where per-message order doesn't matter;
• per-equipment calculation, if every equipment's messages land on the same worker.

The rule that would have saved the line 3 report: a shared group name is part of the interface contract. Prefix it with the application and scope — hist-ingest-line3, report-prep-area1 — and treat a collision the way you'd treat two devices claiming the same Modbus unit ID.

Round-robin will scramble your trends

Say you get the grouping right and put three stateless workers behind $share/report-prep-area1/.... Now the ordering question shows up. For SCADA data, order almost never needs to be global — it needs to hold per device, per tag, per lot, or per alarm source. A shared subscription does not give you that for free.

It depends entirely on the broker's dispatch strategy. Mosquitto round-robins across the group, full stop — consecutive messages from one flow meter go to different workers, and if worker B is 40 ms behind worker A on a database write, a later sample can land in the current-value table before an earlier one. EMQX exposes this as a setting (shared_subscription_strategy) with round_robin, random, sticky, hash_clientid, and hash_topic. The hash strategies are what you actually want for order-sensitive work: hash_topic pins everything on plant/fic101/telemetry to one worker as long as the topic carries the device identity.

If your broker only round-robins, you have three honest options:

• keep order-sensitive work off the shared group entirely;
• partition at the topic level — one shared group per area, or fixed equipment ranges assigned to fixed workers — so the broker never has to make the ordering decision;
• make the consumer reordering-tolerant: carry source, sampleTime, and a monotonic seq in the payload, and reject any update whose sampleTime/seq is older than the current value. Historian inserts key off source time, never broker receive time.

Scaling replicas without one of these isn't scaling. It's moving the bottleneck out of CPU and into your data-quality layer, where it's much harder to see.

QoS decides when you lose data, so decide on purpose

With QoS 1 or 2 the broker holds a message against a client until it's acknowledged, bounded by that client's Receive Maximum (MQTT 5.0 flow control; defaults to 65535 if the CONNECT omits it, though most brokers cap the practical in-flight window far lower). Where you put the PUBACK is the whole reliability story:

• ACK before the DB write is durable → a worker crash silently drops those samples.
• ACK after the write → correct, but a slow historian backs the broker-side queue up fast, and you'll watch queue depth climb during any storage hiccup.
• ACK after handing off to a durable internal queue → the usual compromise for historian ingestion.

Pick the failure mode deliberately and write down where the reliable handoff is. Historian data acks after durability. Dashboard-only enrichment can ack early if a missing derived value is acceptable. Alarm/event processing gets tested with the exact QoS and clean-start/session-expiry settings you'll run in production — reconnect behavior is where clean-session surprises live. Skip this and every future outage post-mortem turns into an argument about whether the broker, the worker, or the database ate the sample.

Watching the group in production

One live client makes a shared group look healthy while it's quietly falling behind. Treat the group as a monitored production component, not a connection status light. The values worth alarming on:

• shared-subscription queue depth, and the age of the oldest queued message (the real lag signal — a group that never drains to zero is undersized or blocked downstream);
• per-client message rate and disconnect count;
• redelivery count after reconnect, and rejected stale/duplicate counts;
• sequence gaps per equipment or tag;
• DB/historian write latency sitting behind the consumers.

During commissioning, fire a controlled burst from a simulator and confirm lag rises and then returns to zero. While you're there, kill one worker mid-burst on purpose: a worker whose DB connection is wedged but whose MQTT session stays alive will hold its share of messages and make no progress. The fix is an application-level health check that disconnects the MQTT client when the worker can't write downstream — only then does the broker redistribute its load.

One last trap worth naming because it's subtle: two workers independently evaluating related alarm transitions will disagree, because neither sees the other's state changes. Alarm state stays on a single active processor, or in a store with atomic updates keyed by alarm source. There's no shared-subscription trick that makes concurrent alarm state machines correct — that one you have to design around.