How to Size an MQTT Wildcard Subscription Before It Floods Your Historian
Estimating the real load behind site/#, and the MQTT 5.0 features that actually contain it: Retain Handling, shared subscriptions, Receive Maximum, SUBACK reason codes.
The historian collector was running eight minutes behind. Broker CPU sat at 12%. The edge gateways were fine. The problem was somewhere else: during commissioning two months earlier, someone had pointed a dashboard service at site/# and walked away, and that one client was now receiving more messages than the historian.
Nothing was wrong with the broker. The filter was just too wide.
What # actually matches
MQTT Version 5.0 (OASIS Standard, 2019) defines the wildcard rules in §4.7.1. Two of them are routinely misremembered.
One: # also matches its own parent level. Per §4.7.1.2, sport/tennis/player1/# matches sport/tennis/player1 itself. So subscribing to site/a/line/pack01/telemetry/# delivers anything published to .../telemetry as well as everything beneath it. If you wrote a parser that assumes it only ever sees child topics, this is where it breaks.
Two: wildcards never match topics beginning with $. This is stated in §4.7.2. Neither site/# nor a bare # will deliver $SYS/.... To read broker statistics you must subscribe to $SYS/# explicitly. Worth noting that the $SYS tree itself is not part of the MQTT standard at all — it is a convention popularised by Mosquitto and others, and the paths differ between brokers.
Also from §4.7.1.2: # must be the last character of the filter and must be preceded by a topic level separator. site/line1# is an invalid filter. Send it and the broker either drops the connection or returns SUBACK reason code 0x8F (Topic Filter invalid).
The arithmetic you do before approving a filter
Counting topics tells you nothing. One slow-changing tank level and one vibration payload published every 100 ms have completely different impact.
Three multiplications settle it:
40 gateways x 60 tags x 1 Hz = 2,400 msg/s
2,400 msg/s x 280-byte JSON payload = 672 kB/s ~= 5.4 Mbit/s
On top of that: the MQTT fixed header is at least 2 bytes, the topic string is carried in every PUBLISH, and QoS 1 adds a PUBACK round trip. With a 60-character topic name, some tags cost more in topic than in payload.
What to measure:
- number of publishers under the tree, and publish interval per payload type;
- burst conditions — reconnect, store-and-forward flush, shift start;
- real payload size after JSON, Sparkplug B, or binary encoding;
- QoS level and whether retained messages are in use;
- downstream cost per message, such as historian writes or alarm evaluation.
Do not size this from a quiet production minute. The real test is the moment a WAN link comes back after twenty minutes down.
Match the subscription to the consumer
Giving every client the same broad filter is fast to configure and slow to debug.
| Consumer | Typical topic scope | Risk of a broad wildcard |
|---|---|---|
| HMI live display | One area, skid, or equipment group | Receives data it never shows, slows down under change bursts |
| Historian collector | Approved telemetry namespace | Stores debug, command, or test topics by accident |
| Alarm service | Alarm/event topics only | Treats status chatter as alarm input |
| Engineering tool | Temporary site or line wildcard | Becomes an untracked production load if left running |
| Dashboard | Aggregated metrics | Pulls high-frequency raw data when hourly totals were enough |
The safe default is the smallest topic tree that answers the consumer's question. If a collector needs several areas, use several explicit filters rather than one site-wide catch-all — broker metrics will then show you which filter is the expensive one.
Kill the retained burst in the SUBSCRIBE options byte
This is the part most people miss. The retained-message flood is not a broker setting. It is a value the subscriber chooses in its own SUBSCRIBE packet.
MQTT 5.0 §3.8.3.1 defines Subscription Options as one byte per topic filter:
| Bits | Name | Values |
|---|---|---|
| 0–1 | QoS | 0, 1, 2 |
| 2 | No Local | 1 = do not send me messages I published myself |
| 3 | Retain As Published (RAP) | 1 = forward the original RETAIN flag unchanged |
| 4–5 | Retain Handling | 0 = send retained at subscribe, 1 = only if the subscription is new, 2 = never |
If your historian collector re-receives the whole retained snapshot on every reconnect, set Retain Handling = 2. Retained messages published afterwards still arrive normally; the only thing suppressed is the bulk delivery at subscribe time. For a collector that keeps its session, 1 is enough.
RAP is not cosmetic either. With RAP = 0 the broker clears the RETAIN bit on messages it forwards, so the subscriber cannot tell a live value from one that was retained earlier. That is exactly how an operator screen ends up showing a dead device's last value in green. Set RAP = 1 and make the display logic read the RETAIN bit.
One caveat: retained messages delivered in the bulk at subscribe time carry RETAIN = 1 regardless of RAP. The ones worth distinguishing are the messages that arrive afterwards.
When the filter genuinely has to be wide, split the delivery
Sometimes you cannot narrow it. A historian is supposed to receive all telemetry. The answer is not a bigger single client.
Use a shared subscription, MQTT 5.0 §4.8.2:
$share/historian/site/a/+/+/telemetry/#
Three collector instances joining the same $share/historian/ group cause the broker to distribute messages between them. Distribution, not duplication. The filter stays wide and only the throughput grows.
Check support in the CONNACK properties. Property 0x2A (Shared Subscription Available) set to 0 means unsupported, and subscribing with $share/ anyway returns SUBACK reason code 0x9E. The same pattern applies to 0x28 (Wildcard Subscription Available) and 0x29 (Subscription Identifier Available). A broker configured to forbid wildcards returns SUBACK reason code 0xA2.
A SUBACK is not proof the subscription worked. Each byte of the SUBACK payload is the result for one filter: 0x00/0x01/0x02 are granted QoS, 0x87 is Not authorized, 0x8F is Topic Filter invalid. Plenty of client libraries throw these bytes away silently. That is why an ACL rejection gets investigated for hours as "the broker isn't publishing".
Client-side backpressure: Receive Maximum
When a collector falls behind, you want the backlog sitting in the broker, not on the client heap.
That is what the MQTT 5.0 CONNECT property 0x21 (Receive Maximum) does. It caps how many QoS 1 and QoS 2 PUBLISH packets the client will have in flight at once. Omit it and the default is 65535 — effectively unlimited. Set it to a few dozen on a collector and the broker stops pushing beyond that, holding the rest in its own queue where the backlog is visible in broker metrics. Far better than discovering the problem as an out-of-memory kill.
QoS 0 gets none of this protection. A broker is free to discard QoS 0 messages destined for a slow consumer, and it disappears without a trace. That is the reason not to run a historian path at QoS 0.
Keep command and telemetry trees separate
A broad read subscription is much less dangerous when topic direction is explicit.
site/a/line/pack01/telemetry/...
site/a/line/pack01/event/...
site/a/line/pack01/alarm/...
site/a/line/pack01/command/...
site/a/line/pack01/config/...
With this shape a historian can subscribe to site/a/+/+/telemetry/# and never see a command payload. An HMI command service gets publish rights on command/# and does not have to receive every vibration sample.
Mix them and wildcard ACLs and wildcard subscriptions stop being reviewable. The technical problem turns into an operations problem: no one can prove who is allowed to see what, or write what.
Common failure modes
The test client is left connected
The story at the top. An engineering laptop subscribes to site/# during commissioning and stays connected over a remote-access tunnel for weeks.
Field check: review the broker client list for unknown client IDs and broad filters before handover. On Mosquitto, start with $SYS/broker/clients/connected.
A dashboard subscribes to raw telemetry
The dashboard needs ten hourly KPIs. It subscribes to every second-by-second telemetry topic and aggregates in the browser. Fine in a demo, not fine with the line fully loaded.
Field check: push aggregation upstream, or publish a dedicated summary topic tree.
Store-and-forward replay floods the consumer
After an outage, edge gateways flush their buffers at once. The broker absorbs it; the historian collector does not, and writes late samples for hours.
Field check: test replay with a realistic buffer size and measure time to catch up. A low Receive Maximum keeps that backlog visible on the broker side.
Sparkplug birth payloads were never counted
In Sparkplug B, a reconnecting edge node publishes NBIRTH plus a DBIRTH per device, and a birth payload carries that node's full metric set. When 40 gateways recover together, the problem is not messages per second — it is metrics arriving in one burst.
Field check: restart one gateway, measure the birth payload size, multiply by 40.
Commissioning checks
- Start the client with a clean session; record initial message count and bytes received.
- Check the Retain Handling value. Confirm you are not sitting at 0 and re-pulling the whole snapshot on every reconnect.
- Restart one edge gateway and measure the birth/retained burst.
- Drop the WAN link, restore it, and measure flush rate and catch-up time.
- Confirm SUBACK reason codes are logged. Check you are not swallowing
0x87. - Snapshot broker metrics before and after adding any permanent wildcard subscriber.
If a dashboard client receives more traffic than the historian, the filter is too broad. There is almost never another explanation.
Broad wildcards are not the enemy. For discovery, diagnostics, and short fault investigations nothing beats them. The trouble starts when one becomes a permanent integration contract nobody remembers agreeing to. If it stays in production, write down three things: owner, purpose, and measured message rate.