SCADA Service Account Password Rotation: Trigger It on Events, Not a Calendar

A domain policy expired DOMAIN\svc_scada at midnight. Nobody was in the plant. At 06:40 the day shift found the HMI up and green, the historian missing six hours, and the night batch report never generated.

The HMI was green because the driver service had authenticated hours earlier and was sitting on an established session. What died at midnight was only the part that had to authenticate again: the historian's SQL connector reconnecting after its idle timeout, the 02:00 report task, the file-share write. That is the shape of a service-account failure. Partial, silent, and blamed on the wrong subsystem for the first hour.

Runtime identity is control system configuration, not IT background detail. IEC 62443-3-3 splits people and software into separate requirements. SR 1.1 covers human user identification and authentication; SR 1.2 covers software process and device identification and authentication. A service account belongs in the SR 1.2 column. How that credential is issued, revoked and changed falls under SR 1.5 (authenticator management). SR 1.7 (strength of password-based authentication) carries the length and composition rules, and its focus is human users — which is half of the argument below. Binding all of it into organisational policy and procedure is IEC 62443-2-1's job. Neither one fixes a rotation interval as a number.

Events force a rotation, calendars don't

A change-everything-quarterly policy was built around passwords a human keeps in their head. That premise was overturned in NIST SP 800-63B §5.1.1.2: verifiers SHOULD NOT require memorized secrets to be changed periodically, and SHALL force a change when there is evidence of compromise. The reasoning is plain — a forced cycle produces Plant2026! followed by Plant2027!.

Quoting that at a service account and concluding "so we never rotate" overreaches. The clause is about secrets a person memorizes, and nobody memorizes a service account password; it lives in a config file, the SCM and a vault. But the difference cuts against periodic rotation, not for it. The risks periodic rotation reduces — guessing, reuse, shoulder-surfing — barely exist for a random string no human has ever seen, while the risk periodic rotation creates — a half-applied credential and a six-hour hole at midnight — stays exactly where it was.

Under NERC CIP the auditor opens CIP-007-6 R5, and the 15-calendar-month interval is Part 5.6. That part scopes itself to "password-only authentication for interactive user access". A service account that never sits at a login prompt is outside that wording. Auditors and entities read this differently, so write your interpretation down with its reasoning; explain it verbally and you will explain it from scratch every time.

What actually forces a rotation is an event.

Someone who knew the credential left: an integrator's engineer, a departing employee, a contract that ended.
Exposure is suspected: a config file found in plaintext, a screenshot attached to a ticket, a handover document that went around by email.
Vendor handover. If the commissioning account went straight into production, handover day is rotation day.
A lockout with no known source, or 4625 events that will not settle. That is both an investigation and a reason to rotate.
A project archive or backup medium left your control.

"The quarter ended" is not on that list.

Find every place the credential lives before you touch it

The dangerous sentence is "it's only used by SCADA." On any system older than about two years, I have never found that to be true.

Where it hides	How to actually look
Windows services	`sc.exe qc <service>` for each, or filter `Win32_Service` on `StartName`
Scheduled tasks	`schtasks /query /fo LIST /v` — read "Run As User" and "Logon Mode"
IIS app pools	`appcmd list apppool /@t:*` — ProcessModel identity, plus cert store access
ODBC	User DSNs live in `HKCU\Software\ODBC\ODBC.INI` under the old profile
OPC Classic (DA)	`dcomcnfg` → server → Identity tab; also OPCEnum launch/activation ACLs
OPC UA clients	Application instance certificate, its private key ACL, endpoint user credentials
SQL Server	Logins, SQL Agent proxies per job step, linked servers, `CREATE CREDENTIAL` objects
Connection strings	Vendor config files, `web.config`, historian collector configs
File shares	Share ACL and NTFS ACL are two different lists — check both

Grepping the SCADA project directory and the vendor config tree for the account name finds more usage sites than any inventory document I've been handed.

A password stored inside configuration is a separate case. A SQL login saved in an HMI project file, a stored credential in a historian collector's config, a DCOM Identity for OPC DA — changing the account password in AD does not finish any of them. The project has to be opened, corrected, rebuilt and redeployed to every runtime node and client before the change takes effect. Miss three clients on a twelve-client site and those three will keep locking the account out. The old password also survives inside project backups and version history, so retirement is not complete until you have decided who can read those.

Commission with the account you will run with

Least privilege is the right direction, and over-tightening produces a recognizable symptom. The HMI opens, tags read fine, and one thing at the edge is broken: trend export, alarm-history insert, recipe download. Read paths and write paths in a SCADA stack rarely share an owner.

What I grant, concretely:

SeServiceLogonRight ("Log on as a service") and nothing else from the interactive rights set. No local logon unless the vendor documents why.
EXECUTE on a schema of stored procedures instead of db_owner. A historian collector almost never needs DDL after commissioning.
Modify on the report output folder only. Read-only on recipe templates when approval lives in another system.
Read on the private key of the client certificate. On Windows that is an ACL on the key container under C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys, not a permission you can see in the certificate store view. This one costs people an afternoon regularly. For the renewal side of it, see the OPC UA certificate renewal checklist.

Then commission with that account. Testing as a domain admin means you discover the missing ACL on startup day with the customer standing behind you.

The better answer is to delete the rotation work. A group managed service account has AD rotate the password itself — ManagedPasswordIntervalInDays defaults to 30 days, cannot be changed after creation, and no human ever sees the value. It needs a KDS root key in the forest and a service that supports gMSA. SQL Server and IIS do. A fair number of SCADA services now do. Anything that stores the password in its own config file does not, and never will — those carry the redeployment problem above, so plan a manual rotation for them and get on with it.

Rotation is a change window, not a ticket

A password change restarts services, drops database sessions, tears down OPC UA secure channels, and interrupts scheduled reports. Book it like any other control system change:

Confirm the usage inventory above, and who is allowed to restart what.
Verify the backups are recent — SCADA project, vendor config, certificate stores. Whether they actually restore is the subject of the backup and restore drill.
Have a rollback. If policy allows briefly re-enabling the old password, know the command before you need it.
Apply the new credential everywhere at once. Half-applied is worse than not started.
Restart bottom-up: database reachable → local license service → PLC drivers, OPC UA clients, gateways → historian interface and buffer → alarm/event processing → HMI runtime and web gateway → reporting and export tasks.
Test behavior, then read the security log.

Step 5 matters more than it looks. Restarting the HMI while a driver service is still failing to read its certificate only generates the log you will spend 20 minutes reading later. And don't rotate 5 minutes before shift change unless the crew has rehearsed the procedure.

Profile-scoped state is where rotations actually break

Changing DOMAIN\svc_scada_old to DOMAIN\svc_scada_new moves %APPDATA%, %LOCALAPPDATA%, HKCU, and the per-user certificate store. Everything an application quietly wrote there is, from the service's point of view, gone.

The OPC UA client regenerates its application instance certificate under the new user store. The server rejects it — correctly; it is an untrusted certificate with a new thumbprint.
Worse, if the ApplicationUri moved with it, trusting the new certificate is not enough. IEC 62541-6 requires the ApplicationUri to match a uniformResourceIdentifier entry in the certificate's SubjectAltName; a mismatch comes back as BadCertificateUriInvalid, which reads nothing like a permissions problem.
The certificate copies to the new store, the private key ACL doesn't follow, and the handshake fails in a way that looks like a network fault.
User DSNs vanish. Convert them to System DSNs before the change.
Mapped drives are per-logon-session state. A service never reliably sees them. UNC paths, always.
Vendor license caches under %APPDATA%: some products silently re-activate, some refuse to start.

A password change on the same account avoids all of this. An account change does not. They are two different change windows and combining them is how a two-hour job becomes an overnight one.

After the restart, check behavior — not service state

A green icon in the service list proves the SCM started a process. It proves nothing about authentication.

Area	What proves it
Live data	Representative PLC tags show good quality and changing values
Write path	A defined safe tag or simulation point accepts a value and reports feedback
Alarms	New alarm raises, acknowledges, and lands in alarm history
Historian	Fresh samples arrive with correct timestamp and quality, and the buffer drains
Reports	A manual run queries data and writes output to the real folder
File shares	Recipe import/export, backup target, batch report drop folder
OPC UA	Secure endpoint connects with no new trust prompt and no certificate error
Security log	No repeating 4625 / 4771 after the window closes

Write tests go to a safe tag or a simulation point. Credential work is not the time to improvise a valve stroke.

The ones that keep coming back

HMI green, historian gapped. An established session survives; a reconnect doesn't. Anything with an idle timeout or a scheduled reconnect is the first casualty and the last thing anyone checks.

Reports fail only at night. Task Scheduler keeps its own copy of the password in the Credential Manager vault. Changing the service logon password does not update it. Re-enter it per task.

Works from my laptop, fails in runtime. The laptop test ran as the engineer. Re-run the vendor diagnostic under the service identity — runas /user:DOMAIN\svc_scada or the vendor's own "test connection as service account" button if it has one.

Authentication failures with no obvious source. Event 4625 on the member server gives you the process name. 4771 or 4776 on the domain controller gives you the calling computer. Chase 4625 alone and you will miss a forgotten scheduled task on another node quietly locking the account out every 15 minutes.

The morning after, don't start in the SCADA logs — check the account's lockout state and badPwdCount on the DC. One missed usage site shows up there as a clean periodic failure long before it shows up as an outage, and it will pick shift change to finally trip the lockout threshold.