Your SCADA Backup Isn't a Backup Until It Restores on a Clean Machine
Running a restore drill: what falls out of the backup set, the OPC UA certificate traps, RPO and retention per artefact, and a measured RTO against the target.
The backup zip was definitely there
2 a.m., the SCADA server disk is dead. The backup zip is on the NAS. You install the SCADA software on a spare PC, unpack the project, and the editor opens. So far so good.
Then the runtime won't start. The license was bound to the MAC address of the machine that just died, and vendor support opens on Monday.
Having a backup file and being able to recover are different capabilities, and the gap always shows up at the worst hour. The standards don't run the two together either. IEC 62443-3-3 splits them — SR 7.3 covers backup, SR 7.4 covers recovery and reconstitution, and SR 7.3 RE 1 separately requires that the backups themselves be verified. IEC 62443-2-1 puts the same material on the management-system side rather than the technical one: there has to be a procedure and someone who owns it. If you quote a clause number from it, quote the edition with it — the 2010 and 2024 editions are structured differently, and an internal document carrying only the number is unfindable two or three years later. NIST SP 800-82 Rev 3 (2023) tailors the SP 800-53 contingency planning family for OT, and there too CP-9 (System Backup) and CP-10 (System Recovery and Reconstitution) are separate controls, with backup testing for reliability and integrity broken out again as the CP-9(1) enhancement. ISO 22301:2019 uses RTO and RPO as defined terms and asks for more than a written plan: the capability has to be validated through an exercise programme.
Three documents, all of them splitting "we took a copy" from "we brought it back". A drill is the only thing that fills in the second column.
What actually goes missing from a backup set
The graphical project is one piece of a running system. Most restores stall on something outside it.
| Item | What happens without it |
|---|---|
| Runtime deployment package | Many products build runtime files separate from the editor project. The editor opens; you still can't run anything. |
| Tag database export | No partial recovery, no version comparison. It becomes all-or-nothing. |
| Alarm database and rationalization records | Priorities, limits, delays, and messages vanish. This is the master alarm database ISA-18.2 talks about, and recreating it is weeks of work. |
| Driver and channel configuration | PLC paths, device IDs, scan classes, timeouts. Rebuild it by hand and you will get one wrong. |
| Historian connector settings | Data source names, compression settings, buffering paths, credentials. |
| OPC UA application certificate and private key | Its own section below. This is where most restores stop. |
| License files and activation notes | This is where the 2 a.m. story above comes from. |
| Scripts, reports, SQL jobs, scheduled tasks | They live outside the HMI editor, so they quietly fall out of the backup scope. |
| Network and hostname notes | Some products bind to hostnames or service names. |
Knowing the artefact by its file name saves an argument at 2 a.m. An Ignition gateway backup is a single .gwbk. A TIA Portal project archive is a .zapNN file whose number tracks the Portal version — .zap17, .zap18 — and it is not the working project folder built around the matching .apNN file; archive the folder as-is and you get something the next Portal version may refuse to migrate. An AVEVA System Platform (Wonderware) object export is an .aaPKG. Write which of these is in the zip on the manifest, because "we have the project" and "we have something deployable" are usually different files and sites routinely keep only one.
On virtual machines, capture the VM configuration and not just the guest files: vCPU count, NIC order and MAC addresses, disk paths, time sync source. I have watched a driver come back talking to the wrong subnet because NIC order changed on the replacement VM.
Measure the sizes too. The gateway backups I deal with run 40-120 MB; the full system image of the same server is around 60 GB. A 1 Gb/s NAS link makes that an 8-minute copy on paper, and it has never once finished on paper. If you don't know whether that single line is 30 minutes or two hours, the RTO you wrote was wrong before the drill started.
Restoring onto the machine that made the backup finds nothing
This is the whole point of the drill, and it is the step sites skip. Restore to the original machine and of course it works — the drivers, registry keys, ODBC DSNs, and old .NET runtime are all still sitting there.
The target has to be clean: a spare industrial PC, a test VM on the same OS family, a vendor recovery image, a lab workstation isolated from the process network. Start from a clean OS. Install the SCADA software, apply patches, restore the backup, and write down every manual step someone had to perform.
If the restore only worked because something happened to be left over on the old machine, the backup set is incomplete. That leftover will not be on the spare PC at 2 a.m.
Certificates and accounts block more restores than files do
Modern systems fail after a restore because identity changed, not because data is missing. The project restores fine, but the hostname, the certificate, and the service account SIDs are all different.
OPC UA reports this precisely, in status codes. Restore a server under a new hostname and clients reject it with BadCertificateHostNameInvalid — the DNS name in the certificate's SubjectAltName no longer matches the endpoint host they connected to. A mismatched ApplicationUri gives BadCertificateUriInvalid; a trust store that didn't come along gives BadCertificateUntrusted. OPC UA Part 4's certificate validation rules treat those as three distinct failures. Being able to tell them apart in the log cuts hours off a recovery.
Open one of these certificates and it's obvious why they're distinct. An OPC UA application instance certificate carries two kinds of SubjectAltName entry: DNS and IP entries holding the host, and a URI entry holding the ApplicationUri verbatim. Rename the restore target and the first kind breaks; if the product builds its ApplicationUri out of the hostname, the second breaks with it. That is how one hostname change produces two different status codes.
Key length bites here too. The Basic256Sha256 security policy requires RSA keys of 2048 bits or more, up to 4096 bits. Restore an old project's 1024 bits certificate untouched and it never reaches the hostname check — the policy rejects it first. If the restored node has to face current clients, reissue is the only way out.
During the drill, confirm:
- Where the application certificate and its private key actually live. If the key sits in an OS key store rather than a file, a file-level backup misses the private key entirely — the quietest failure in this list.
- Whether you copy the trust store or re-trust from scratch. Write down the procedure for both.
- Whether the certificate's SubjectAltName includes the restore target's hostname or IP. If not, you need a reissue — and a reissue drags client-side trust work along with it.
- Which service account runs each of the drivers, historian, and web gateway. Local accounts get brand-new SIDs when the OS is reinstalled, so file ACLs and DCOM permissions still referencing the old SID fail even though the account name matches.
- Whether license activation is tied to hardware, VM UUID, MAC address, or a dongle, and whether that can be moved on a weekend. Whether an emergency license exists, and how long it lasts, is per-vendor contract language — check yours, there is no general number here.
Don't put plaintext passwords in a general backup archive. Keep credential recovery on the site's approved secret management path. The account side is covered in rotating a SCADA service account password, and reissuing certificates hits the same problems described in the OPC UA certificate renewal checklist and rotating an MQTT broker certificate.
An editor that opens is not a recovery
The pass criterion is that the runtime proves its interfaces, and "checked" is not a verdict. Pin a number to each path and nobody argues about whether the drill finished.
| Check | Pass criterion |
|---|---|
| PLC/controller communication | 10 representative tags read quality good and match the field (or simulator) values |
| OPC UA endpoint | A client connects on Basic256Sha256 with no extra trust work |
| Alarms | Raise, acknowledge, and history all work; an unprivileged role is refused the acknowledgement |
| Historian write path | One test tag logged for 10 minutes and read back with zero missing samples |
| Reports | One report from last month generates end to end against the database |
| Login and roles | Two roles minimum, and each role's forbidden screens are actually blocked |
| Redundancy | One forced switchover, with the tag update gap inside the limit you set beforehand — same ground as testing a redundant SCADA server switchover |
| Time sync | Restored node within 1 s of the historian clock, same timezone. Get this wrong and historian data drifts silently |
Add one counting step to that list. Record the tag count, alarm count, and screen count in the manifest at backup time, pull the same three numbers out of the restored runtime, and compare. A partial restore never fails the "editor opens" test; it fails the count. Archives need the same treatment — a passing CRC says the zip is intact, not that the contents are right — so write the SHA-256 next to each archive when you make it and check it before starting the base software install. Finding out an archive is truncated after a 1 h 40 min install ends the drill for the day.
Safety is a separate concern. A restored test node must not write commands to live equipment. Use a test network, a simulator, a read-only connection, or a planned maintenance window. No exceptions unless the procedure explicitly allows it.
Set RPO per artefact, not per server
Plenty of sites have one line of policy: full image every Sunday. That makes the system's RPO seven days. The trouble is that a SCADA project doesn't change at a steady rate. During two weeks of commissioning an alarm limit changes twenty times a day; for the six months after that nobody touches it. A seven-day RPO is generous in the quiet half-year and a disaster in the commissioning weeks.
Splitting the target by artefact is cheaper than tightening it everywhere.
| Artefact | Backup trigger | Copies kept | Effective RPO |
|---|---|---|---|
| Project source, tag/alarm exports | every change (git commit) | full history | last commit |
| Runtime deployment package | every deployment | last 10 | last deployment |
| Full system image | weekly, plus before and after patching | 4 weekly + 12 monthly | 7 days |
| Certificates, licenses, manifest | on change | full history | last change |
Leave the retention column blank and what survives is usually one copy — the one that faithfully contains yesterday's bad configuration. Four weeklies buy you a month, twelve monthlies buy you a year, and finding a configuration mistake two weeks late is not a rare event.
A policy that has only the third row is a policy missing the first, and missing the first means rebuilding the alarm rationalization by hand. How to capture those records in the first place is the alarm rationalization workshop checklist. The RTO side stays a wish until you measure it the way below.
Record the time with a stopwatch, not with hope
The output of a drill is not "passed" — it is measured time per stage. A real one looks roughly like this:
Find the correct backup 0:25 (5 files; confirming the approved one took most of it)
Base software + patches 1:40
Restore files/configuration 0:35
Rebuild certificates, trust 1:10 ← the stage with the widest spread
Reapply license 0:50 (includes vendor portal wait)
Verify comms and alarms 1:05
Hand back to operations 0:20
----------------------------------
Total 6:05
The approved target on this site was an 8-hour RTO and a 24-hour RPO. A 6:05 lab measurement leaves 1 h 55 min of headroom, and one bad night call, one hardware hunt, or one vendor portal queue eats it. Treat the table as a floor, not an estimate: the same work at 2 a.m. while answering the phone runs longer.
When the target and the measurement sit that close, one of them has to move. Before I stretch the target I go after the 1:40 of base software installation — a golden image with patches already applied collapses that stage to a few minutes of restore, and it is the largest single drop available for one piece of work. Writing a one-hour RTO against a six-hour lab result gets the document approved and then fails identically during the real event.
Naming feeds straight into that first line. final_new_2.zip is the name that turns finding a backup into 25 minutes. Site, system, date, version, and source should be readable from the filename.
WTP_SCADA-SRV1_RuntimeBackup_2026-06-19_app-4.8.2.zip
WTP_SCADA-SRV1_ProjectSource_2026-06-19_git-a1b2c3d.zip
WTP_OPCUA_TrustStore_2026-06-19.zip
Keep a short manifest next to the backup, plain text or PDF: contents, who made it, software versions, license status, tag and alarm counts, the archive SHA-256, and the date and result of the last restore test. It has to be readable without the SCADA engineering tool — the person doing the recovery doesn't have that tool installed yet.
Failures seen in the field
- The project source is backed up; the runtime deployment package is not.
- Alarm database changes were made directly in production and never exported.
- The OPC UA private key lived in the OS key store, so the file backup took the certificate and left the key behind.
- A scheduled report depends on an ODBC DSN that existed only on the old server.
- Historian buffering used a local path nobody recreated on the new machine.
- License activation cannot be moved on a weekend.
- The backup sat on the same disk that failed.
- Five backup files exist and nobody knows which one is approved.
Once a drill is done, the next thing to check is already decided: find every step somebody had to do by hand, and get it into the backup set. A manual step is a step the next engineer doesn't know about.