Rotating an MQTT Broker Certificate Without Dropping Your SCADA Clients

The certificate expired at midnight and nobody had a clock

The usual way you learn a broker certificate expired is that every gateway drops at the same instant, the historian goes flat, and the alarm summary fills with connection failed — one line per client, no hint which layer broke. It looks like the switch died. It didn't. An X.509 notAfter timestamp rolled past, and because RFC 5280 defines those in UTC (Z time), it fired nine hours before the wall clock said the cert was due to expire for a plant on UTC+9.

That is the whole problem with treating MQTT-over-TLS rotation as an IT ticket. Renewing a web server cert is a page reload. Renewing the cert that fronts a dozen edge gateways, HMIs, a historian, an MES connector, and a couple of cloud bridges is a commissioning activity with process impact: gateways stop publishing, subscribers hold stale values behind a green icon, store-and-forward queues fill, and Sparkplug nodes replay their whole birth sequence. Plan it like a cutover, not a certbot renew.

Know every place a TLS handshake happens before you touch one

The broker cert is never the only endpoint, and the one that bites you is always the one nobody inventoried. Walk the whole path first and write down where each service loads its certificate from — a file path, an OS trust store, a container secret, a Kubernetes secret, or some vendor runtime folder that only the vendor knows about.

Item	What to record	Why it bites
Broker server cert	CN/SAN, issuer, notAfter, key location	Clients validate the broker's identity against the names they dial
Broker CA chain	Root + every intermediate	A missing intermediate fails validation on strict clients, passes on lax ones — inconsistent by design
Client certs (mTLS)	Gateway/app name, issuer, expiry	Each is a separate expiry clock you now have to track
Client trust stores	Which CA trusts the broker	New CA must land here before the broker cutover, not during
Bridge credentials	Upstream + downstream files	Broker-to-broker links usually carry their own cert pair
Test tooling	Laptop certs and trust store	A test that passes from your laptop and fails from the service is a trust-store mismatch, nothing more

If you skip the "loads from where" column you will replace a file, restart the service, and watch it keep presenting the old cert because the real one lived in a store you didn't know about.

Almost every failure is names, time, or chain — not crypto

I have never once seen a rotation fail because the cipher was wrong. It fails on identity and validity, so that is where the pre-window checks go.

Before the maintenance window, prove out:

• The broker cert's SAN list contains every name clients actually dial. In segmented plant/DMZ networks one gateway connects to mqtt-broker.local and another connects by raw IP — both have to be in the SAN, or those clients get reconfigured. CN-only certs are dead; modern clients ignore CN entirely and check SAN.
• Client and broker clocks agree. A gateway drifted 20 minutes fast will reject a fresh cert as not yet valid.
• The full chain is assembled, intermediates included, in the order the runtime wants.
• keyUsage/extendedKeyUsage actually say serverAuth (or clientAuth for the client certs). A cert issued for the wrong purpose validates in a browser and gets refused by a strict MQTT stack.
• The service account can read the private key and nothing wider can.

The one command that catches most of this before you touch production:

openssl s_client -connect broker:8883 -servername broker.host -showcerts </dev/null

Read what the broker actually presents — the chain it sends, the dates, the SAN — from the network, not from the file you think you deployed. openssl x509 -noout -dates -ext subjectAltName on the file confirms the rest.

Add trust first, change identity second — never both at once

The one rule that makes this safe: a client can trust two CAs at the same time, so widen trust before you narrow identity.

1. Push the new CA (or new broker cert, if self-signed) into client trust stores while the old broker cert is still live. Clients don't care that they now trust a CA they aren't using yet.
2. Reload one pilot client and confirm it still connects on the old cert.
3. Swap the broker cert.
4. Confirm the pilot reconnects and publishes a real payload — not just a handshake.
5. Roll through the rest.
6. Remove the old CA only after a defined overlap — I give it a week, so a gateway that was powered off during the window still trusts the broker when it wakes up.

For mutual TLS, rotate the client certs the same staged way: teach the broker to trust the new client CA first, then swap one client cert and re-prove authentication, topic authorization, and retained/birth behavior on that one before touching the rest.

The pattern to avoid is the big-bang — broker identity and every trust store changing in one shot. When that fails, and it does, nothing tells you which side rejected the handshake, and you're rolling back blind with the plant down.

Do the buffer math before you start the clock

Rotation must not quietly drop process data. Before the window, get four numbers off the edge buffer: current queue depth, max queue size, publish rate, and free disk. Then divide.

A gateway pushing 500 msg/s into a buffer that holds 20 minutes gives you a hard ceiling — a one-hour window loses data no matter how clean the cutover is. When the math doesn't fit, you have four levers: shrink the window's scope, grow the buffer, throttle non-critical publishing for the duration, or split into shorter cutovers. Pick one before you start, not at minute 22 when the queue alarm trips.

Two things people forget on the replay side. Check whether QoS and session settings actually preserve messages across the reconnect (a clean-session client with QoS 0 keeps nothing). And confirm the buffer replays with the original source timestamp, not reconnect time — otherwise an hour of held data lands on the historian stamped as one spike at cutover, and your trends and production records lie.

Verify MQTT behavior, not a green handshake

A successful TLS handshake means the transport is up. It says nothing about whether the client can publish, whether authorization survived, or whether its identity is intact. Test at the protocol level, per representative client type:

mosquitto_pub -h broker -p 8883 --cafile new-ca.pem \
  --cert client.pem --key client.key \
  -i real-client-id -t plant/line1/test -m rotation-check

Then subscribe and confirm payload, timestamp, and quality fields come back clean; confirm an unauthorized topic is still refused; restart the client and watch the birth message; force a disconnect and confirm the last-will fires. Read the broker log for the cert subject, client ID, and authorization result on each — that log line is where an "authenticated but not authorized" failure hides, and it looks nothing like a cert problem from the client side.

If you run Sparkplug B, a cert change must not mint a new logical device. Confirm the NBIRTH sequence and metric aliases match what the node published before rotation. A gateway that comes back with a fresh bdSeq and re-registered aliases has effectively become a new device to your MES — same wire, new identity, broken mapping.

Give operators a diagnostic that points at a layer

A single red comms icon tells maintenance nothing, so they treat a five-minute cert reject and a cut fiber the same way — slowly. Surface enough to separate the layers without leaking secrets:

• Broker reachable but TLS rejected → cert problem, not network.
• Cert expired / not yet valid → check clocks and dates.
• Name mismatch → SAN doesn't cover the name this client dials.
• Authenticated but topic authorization failed → ACL, not TLS.
• Queue depth + oldest queued timestamp + last successful publish time → how much is buffered and how far behind.

None of that exposes a key. It just aims the person holding the radio at the right layer instead of the switch.

The failure modes that repeat every rotation

Only some gateways fail after the broker swap. The survivors dial a name in the SAN; the casualties dial one that isn't — usually a raw IP or an alternate DNS zone.

Works from the laptop, fails from the service. Different trust store. Always test from the runtime account on the runtime host; the laptop is lying to you.

Fails right after you add the intermediate. Chain order or file format is wrong for that runtime. Some want leaf → intermediate → root concatenated; some want only CA certs in the trust store and reject a bundle with a leaf in it.

Data doubles after reconnect. Store-and-forward replay and QoS retry both fired. Confirm message IDs, source timestamps, and idempotent handling on the subscriber.

Everything drops at a clean hour, not at the expiry you had on the calendar. notBefore/notAfter are UTC. Your calendar wasn't. See the first paragraph.

Leave evidence, not "renewed certs"

The next rotation is half-done if this one leaves a record: which certs and serial numbers, old and new issuers, expiry dates, the client list you tested, the broker listener, queue depth before and after, any client that needed a manual restart, and the date the old trust material gets pulled. "cert renewed" in a ticket comment guarantees the next person repeats every mistake you just made — including the one about UTC.