Skip to content

Secret and Certificate Rotation

Status: Procedures below match the behaviour of the current Helm chart and gateway code. None of them have been exercised end-to-end on a real production cluster yet -- rotation is the most common cause of self-inflicted outages in cloud-native systems, so drill each runbook on a test cluster (and record the drill date in the "Last drilled" column) before relying on it under pressure.

Step-by-step runbooks for rotating every long-lived credential and certificate in a Cloud-Native DCS deployment. Each runbook is written as a graceful (zero or near-zero downtime) path, with an emergency branch when a credential is suspected compromised.

Commands assume the default install from Deploy Your Own (helm install dcs … --namespace dcs-system), which names chart-managed resources with the dcs-cloud-native-dcs prefix (written <release>-… where this doc speaks generically). Substitute your own release namespace and fullname prefix if they differ.

What Needs Rotation

Credential How it is issued Default lifetime Graceful path Last drilled
mTLS component certs (gateway, operators, io-probe, historian) cert-manager, signed by the in-cluster CA ClusterIssuer mtls.certManager.duration = 24h, renewed at 8h remaining Automatic (cert-manager; pkg/tlsutil hot-reloads in place, no pod restart) 2026-04-21
mTLS runtime cert (per site-<name> namespace) cert-manager Certificate created by the Site reconciler in each site namespace, signed by the CA ClusterIssuer 24h, renewed at 8h remaining Automatic (cert-manager rotates the Secret; Unit reconciler watches it and recreates the runtime Pod when resourceVersion changes — see Runtime cert drill) 2026-04-21 (partial — no runtime Pod on drill cluster)
mTLS CA certificate cert-manager self-signed Certificate 8760h / 1 year, renewed at 720h remaining Manual re-issue + cmctl renew --all (every renewal changes CA identity under cert-manager 1.18+ defaults) 2026-04-21
MQTT broker TLS cert cert-manager, signed by the mTLS CA ~90d (the mqtt.tls.certManager default; component certs use 24h) Automatic (cert-manager) + broker rolling restart (mosquitto does not hot-reload) 2026-04-21
MQTT client passwords (gateway, operator, runtime, historian, omf-egress) Helm-generated Opaque Secret <release>-mqtt-auth Indefinite Credential slots (#1579, ADR 0061): three helm upgrades — open the standby account, flip activeSlot, retire the account you left. No client is ever locked out, and the broker never restarts. Unit runtimes are the exception: their pods are recreated by hand, at a moment of your choosing. See MQTT Client Passwords 2026-04-21 (non-graceful); 2026-08-14 (#221 window drilled, found broken, #1579); 2026-08-15 (slot window drilled end to end, #1579)
Unit runtime bearer token Helm-generated Opaque Secret <release>-runtime-auth Indefinite helm upgrade + manual kubectl delete pod of runtime pods (bare Pods, so rollout restart deploy does not touch them) 2026-04-21 (partial — no runtime Pod on drill cluster)
HMAC signing key (e-signatures / audit) Versioned Secret dcs-signing-key in kube-system (auto-generated v1 on first startup) Annual kubectl patch to add new key version + gateway restart. Historical signatures remain verifiable via key-ID chaining. See HMAC procedure 2026-04-21
LDAP bind password (gateway.auth.ldap.bindPasswordSecret) External directory Indefinite helm upgrade + gateway pod restart not yet
OMF endpoint credential (omfEgress.auth) The customer's PI System, or whoever administers the OMF endpoint Indefinite Update the Secret + restart the egress pod. No dual-credential window, and the overlap would have to exist in the PI Server's own account store — see OMF Endpoint Credential not yet
Historian DB password CNPG operator-managed Secret Indefinite CNPG kubectl cnpg restart + password rotation not yet
Cluster node join credentials Deployment layer (Talos machine config, k3s token, …) Distro-defined Deployment-layer concern (ADR 0004), out of product scope n/a

Lifetime note: The runbook tables that used to claim 90-day mTLS lifetimes were wrong. With the shipped defaults, component certs are short-lived (24h) precisely so rotation exercises its own code path every day. A silent cert-manager failure surfaces within hours instead of quarters.

Rotation Schedule

Event Action
Every 24h (automatic) cert-manager renews component mTLS certs at 8h remaining. Release-namespace Deployments (gateway, operators, io-probe, historian) hot-reload the cert files in place via pkg/tlsutil. Per-site runtime Pods are bare Pods, so the Unit reconciler recreates them when it sees the Secret resourceVersion change (dcs.io/runtime-mtls-rv annotation). Nothing to do if monitoring is green.
Annually (automatic, 30 days before CA expiry) cert-manager renews the mTLS CA cert. Every renewal changes the CA identity (new private key under cert-manager 1.18+ defaults), so this is not hands-off: run cmctl renew -n dcs-system --all immediately after any CA renewal so every component's trust store picks up the new CA. See mTLS CA Certificate.
Quarterly Drill one credential from the "Graceful path" column above that isn't on cert-manager autopilot. Rotate MQTT passwords and the runtime bearer token at least once per quarter. MQTT passwords no longer need a quiet window for the release-namespace clients (#1579), but the unit-runtime step still does: the runtime pods are recreated by hand, and a runtime restart puts a Running batch on Hold. Plan that step for a unit that is idle.
Annually Rotate the gateway HMAC signing key (see caveats) and review the LDAP bind password if in LDAP mode.
Immediately on suspected compromise Any credential: follow the emergency branch of the relevant runbook.

Detection and Monitoring

The rotation runbooks are only useful if someone notices a certificate is drifting towards expiry before it expires. The chart ships with DCS-authored PrometheusRule alerts in deploy/helm/cloud-native-dcs/templates/prometheusrule.yaml that cover the cert-expiry signal end-to-end:

Alert Expression Threshold Severity Intended routing
DCSCertExpiryWarning dcs_cert_expiry_seconds - time() in [8h, 48h) for 15m < 48h remaining warning Ticket — cert-manager still has ~8h before it normally renews; investigate next business day unless the 8h renewal also fails.
DCSCertExpiryCritical dcs_cert_expiry_seconds - time() in [0, 8h) for 5m < 8h remaining critical Page — cert-manager's renewal window has passed without renewal; mTLS will break at NotAfter.

Both alerts fire off the DCS-emitted dcs_cert_expiry_seconds gauge (pkg/tlsutil/certexpiry.go). The gauge reads the actual cert file mounted in the container, which can disagree with what cert-manager reported to the API server. That disagreement is the 2026-04-19 failure mode (cert rotated in etcd, but kubelet's cached Secret volume was stale), and a cert-manager-native alert would have missed it entirely. Complementary upstream signals (certmanager_certificate_ready_status, certmanager_certificate_expiration_timestamp_seconds) are still worth forwarding if cert-manager itself is in the same monitoring stack. They alert on cert-manager health, while the DCS alerts above alert on workload impact.

Grafana dashboards: the shipped dashboards in grafana/dashboards/ are batch- and audit-focused (no cert-expiry panels as of this writing). Operators who want an at-a-glance cert-expiry view should add a custom panel using dcs_cert_expiry_seconds - time() grouped by {component, host, file}. File an issue if the out-of-the-box dashboards should grow a cert-health row. It is a small addition against the existing metric.

Log-side signal: a failed renewal usually shows up as x509: certificate has expired or is not yet valid on the client side before the server-side alert fires. Forward these log lines to your SIEM and alert on any occurrence from the DCS namespaces.

Verification After Rotation

Every runbook ends with this checklist. Do not mark a rotation complete until every box is ticked.

  • [ ] All pods using the rotated credential are Running and Ready.
  • [ ] No TLS handshake or auth errors in gateway, operator, or runtime logs since the rotation started.
  • [ ] dcs health reports all components healthy.
  • [ ] A canary batch runs to completion against a simulated unit.
  • [ ] The AuditRecord trail for the rotation window is present and signatures verify (see HMAC caveat).
  • [ ] The old credential is fully decommissioned -- Secret data removed, broker ACL entry deleted, or identity provider client deleted.
  • [ ] Rotation drill logged with date, operator, and outcome.

mTLS Component Certificate (Graceful)

Applies to release-namespace Deployments: gateway, the operators, io-probe, and historian. Runtime pods follow a different path, covered in Runtime cert drill below.

cert-manager renews each component certificate automatically at renewBefore of its expiry (mtls.certManager.renewBefore, default 8h). The human-driven drill exists to prove that the rotation path works. The rotation itself already happens on its own.

# 1. Verify cert-manager is healthy.
kubectl get pods -n cert-manager
kubectl get certificate -n dcs-system \
  -l app.kubernetes.io/part-of=cloud-native-dcs

# 2. Pick one component and force a renewal.
cmctl renew -n dcs-system dcs-cloud-native-dcs-gateway-mtls

# 3. Watch the Secret `dcs-cloud-native-dcs-gateway-mtls` get rewritten and the
#    gateway pod pick up the new cert on its next poll.
kubectl get secret -n dcs-system dcs-cloud-native-dcs-gateway-mtls \
  -o jsonpath='{.metadata.resourceVersion}'

# 4. Confirm no TLS errors and that active sessions still work.
kubectl logs -n dcs-system deploy/dcs-cloud-native-dcs-gateway --since=5m | grep -i tls
dcs health

Deployments in the release namespace pick up the new cert without a pod restart because pkg/tlsutil hot-reloads the mounted Secret files. If you see x509 errors on any of these clients, something has gone wrong. Fall through to the emergency branch.

mTLS Runtime Certificate (Per Site)

Runtime pods run in per-site namespaces (site-<name>), outside the release namespace. Each site has its own cert-manager.io/Certificate, created by the Site reconciler (internal/controller/physical/site_controller.go) and signed by the same CA ClusterIssuer as the release-namespace components. cert-manager owns rotation natively per site. There is no copy-from-dcs-system step.

Runtime pods are bare Pods (not Deployments), so neither Stakater Reloader nor the operator's existing spec-drift check picks up a Secret rotation. Instead, the Unit reconciler stamps the Secret's resourceVersion on each pod as the dcs.io/runtime-mtls-rv annotation at create time, and Watches() the Secret. On change, the reconciler deletes and recreates the pod so the fresh cert is mounted.

Drill path:

# 1. Pick a site and confirm its Certificate is Ready.
SITE=site-plant-01   # replace with any existing site-<name>
kubectl get certificate -n "$SITE"
kubectl get secret -n "$SITE" dcs-cloud-native-dcs-runtime-mtls \
  -o jsonpath='{.metadata.resourceVersion}'

# 2. Capture the current annotation on a runtime pod in that site.
POD=$(kubectl get pod -n "$SITE" -l dcs.io/component=unit-runtime \
  -o jsonpath='{.items[0].metadata.name}')
kubectl get pod -n "$SITE" "$POD" \
  -o jsonpath='{.metadata.annotations.dcs\.io/runtime-mtls-rv}'

# 3. Force cert-manager to rotate the site's runtime cert.
cmctl renew -n "$SITE" dcs-cloud-native-dcs-runtime-mtls

# 4. Within seconds, the Unit reconciler should delete and recreate the
#    pod. Watch for the new pod to reach Ready, and confirm its annotation
#    matches the new Secret's resourceVersion.
kubectl get pod -n "$SITE" -l dcs.io/component=unit-runtime -w
kubectl get secret -n "$SITE" dcs-cloud-native-dcs-runtime-mtls \
  -o jsonpath='{.metadata.resourceVersion}'
kubectl get pod -n "$SITE" -l dcs.io/component=unit-runtime \
  -o jsonpath='{.items[0].metadata.annotations.dcs\.io/runtime-mtls-rv}'

# 5. Confirm no x509 errors from the operators talking to the new pod.
kubectl logs -n dcs-system deploy/dcs-cloud-native-dcs-procedural-operator --since=5m \
  | grep -iE 'x509|tls'
dcs health

What to watch for:

  • In-flight batches should tolerate the short pod-recreate window (the runtime replays persisted FB state from hostPath on startup). If a batch faults or goes to Hold during the drill, file a bug. The recreate flow is supposed to be transparent.
  • Pre-existing pods without the dcs.io/runtime-mtls-rv annotation are deliberately left alone by the reconciler so that operator upgrades don't thrash every runtime pod at once. In a real rotation, those pods need a one-time manual kubectl delete pod to opt into the watcher. Note this in the drill report if any pods predate the annotation.

mTLS Component Certificate (Emergency)

Use this when cert-manager itself has failed or a key is suspected compromised.

# 1. Find which cert-manager component is unhealthy.
kubectl get pods -n cert-manager
kubectl logs -n cert-manager deploy/cert-manager --since=15m

# 2. If cert-manager is recoverable, force renewal and let it heal.
cmctl renew -n dcs-system --all

# 3. If cert-manager is down and a cert is expired, manually issue from
#    the CA Secret. (This is a break-glass path; prefer fixing cert-manager.)
kubectl get secret -n dcs-system dcs-cloud-native-dcs-mtls-ca \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ca.crt
kubectl get secret -n dcs-system dcs-cloud-native-dcs-mtls-ca \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ca.key
# Use openssl or cfssl to re-issue the component cert against ca.crt/ca.key
# with the same dnsNames from the Certificate CR.

# 4. Patch the component Secret with the new cert+key.
kubectl create secret tls dcs-cloud-native-dcs-gateway-mtls \
  --cert=new.crt --key=new.key --dry-run=client -o yaml \
  | kubectl apply -n dcs-system -f -

# 5. Restart the component to force a clean reload.
kubectl rollout restart deploy/dcs-cloud-native-dcs-gateway -n dcs-system
kubectl rollout status deploy/dcs-cloud-native-dcs-gateway -n dcs-system

After recovery: file an incident ticket on the cert-manager failure mode. Do not leave the cluster running on break-glass certs. Re-issue via cert-manager as soon as it is healthy.

mTLS CA Certificate

The in-cluster CA certificate has a 1-year lifetime. cert-manager renews it 30 days before expiry.

Important — CA identity changes on every renewal. Under cert-manager 1.18+ defaults (privateKey.rotationPolicy: Always), every CA renewal generates a new private key and therefore a new CA identity (fingerprint + serial change) even though the Subject CN stays the same. Verified on 2026-04-21: pre-renewal fingerprint DA:5A:41:BB… → post-renewal 63:E1:A7:42…. The earlier version of this runbook claimed "the CA identity stays the same"; that is not true with current cert-manager defaults.

This matters because each component's mTLS Secret carries a copy of the CA cert under ca.crt (used as the trust store). The Secret is not automatically refreshed when the CA rotates — it is only rewritten when cert-manager next issues that component's own certificate. So there is a window (up to 24h with default component duration=24h, renewBefore=8h) where some components trust only the old CA while others present certs signed by the new CA — a silent cross-component mTLS break.

After any CA renewal (automatic or manual) you must force every component cert to be re-issued so every trust store updates:

cmctl renew -n dcs-system --all

Wait ~30 seconds. Every component's Secret ca.crt will match the new CA. Verify:

LIVE=$(kubectl -n dcs-system get secret dcs-cloud-native-dcs-mtls-ca \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)
for c in gateway physical-operator procedural-operator control-operator io-probe historian; do
  FP=$(kubectl -n dcs-system get secret dcs-cloud-native-dcs-$c-mtls \
    -o jsonpath='{.data.ca\.crt}' | base64 -d | \
    openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)
  [ "$FP" = "$LIVE" ] && echo "$c: in-sync" || echo "$c: STALE"
done

If you need to replace the CA with a different identity (e.g. compromise or a switch to an organisational PKI):

  1. Create the new ClusterIssuer or Issuer you want to use.
  2. helm upgrade --set mtls.certManager.issuerRef.name=<new-issuer> \ --set mtls.certManager.issuerRef.kind=ClusterIssuer.
  3. Force renewal of every component certificate: cmctl renew -n dcs-system --all (same step as above).
  4. Roll every deployment that has cached the old CA bundle: kubectl rollout restart deploy,ds,sts -n dcs-system.
  5. Delete the old auto-created Issuer and CA Secret only after you have confirmed no component is still presenting the old CA.

This is effectively a full re-keying -- exercise it on a test cluster before running it in production.

MQTT Broker TLS Certificate

The broker cert is cert-manager managed (mqtt.tls.certManager.enabled, issued by the mTLS CA by default). The rotation path is identical to the mTLS component flow above -- cmctl renew -n dcs-system dcs-cloud-native-dcs-mqtt-tls, then restart Mosquitto so it reloads the cert:

kubectl rollout restart deploy/dcs-cloud-native-dcs-mqtt -n dcs-system   # HA: sts/dcs-cloud-native-dcs-mqtt

Clients reconnect automatically. The gateway, operators, runtimes, and historian use persistent MQTT connections with auto-reconnect.

MQTT Client Passwords

The Helm chart gives each MQTT client identity a broker account in the Secret <release>-mqtt-auth, keyed by account name. Mosquitto loads them from the rendered passwd file at startup. The broker pod runs a passwd-reloader sidecar that detects Secret changes, rewrites the hashed passwd file, and SIGHUPs mosquitto in place. A password change therefore never restarts the broker.

Each identity has up to two accounts, its credential slots (ADR 0061). Slot a is the base name, slot b is that name with a -b suffix, and both carry the same ACL rules. A release with no rotation in flight has slot a only.

Identity (values key) Slot a account Slot b account
gateway dcs-gateway dcs-gateway-b
operator dcs-operator dcs-operator-b
runtime dcs-runtime dcs-runtime-b
historian dcs-historian dcs-historian-b
omfEgress dcs-omf-egress dcs-omf-egress-b

Mosquitto's builtin backend keys credentials on the username, so two accounts are what makes a window. Two passwords for one username is what the chart used to write, and it either swapped the accepted credential or killed the broker. That is #1579, and mqtt.auth.users.*.rotationPassword is refused at render time.

Rotation is three helm upgrades. They can be minutes or days apart. What matters is that step 3 happens, because the old credential works until it does.

# Step 1 — OPEN the window. The dcs-gateway-b account appears with the same ACL
# rules as dcs-gateway; the reloader sidecar rehashes and SIGHUPs the broker in
# place. Nothing restarts, and no client changes credential.
NEW_GW=$(openssl rand -base64 24 | tr -d '=+/')
helm upgrade dcs deploy/helm/cloud-native-dcs --reuse-values \
  --set-string mqtt.auth.users.gateway.passwordB="$NEW_GW"

# Confirm the reload landed and the account is there:
kubectl logs -n dcs-system deploy/dcs-cloud-native-dcs-mqtt -c passwd-reloader | tail -1
#   passwd-reloader: mqtt-auth changed, rehashing and SIGHUP mosquitto
kubectl exec -n dcs-system deploy/dcs-cloud-native-dcs-mqtt -c mosquitto -- \
  cut -d: -f1 /mosquitto/auth/passwd
#   dcs-gateway
#   dcs-gateway-b
#   …

Wait for the account to appear before flipping. The raw credentials reach the broker through a mounted Secret, so kubelet's sync period (up to a minute) and then the reloader's 10s poll sit between the helm upgrade and the account existing. A flip that overtakes them rolls the clients onto an account the broker has not loaded yet, and they are refused until it catches up. The cut -d: -f1 check above is the gate, and it is the whole reason step 1 and step 2 are separate upgrades.

# Step 2 — FLIP. Only the gateway's pods roll, and they roll onto dcs-gateway-b.
# Both accounts are served throughout, so a pod that has not rolled yet keeps
# working.
helm upgrade dcs deploy/helm/cloud-native-dcs --reuse-values \
  --set-string mqtt.auth.users.gateway.activeSlot=b

kubectl rollout status -n dcs-system deploy/dcs-cloud-native-dcs-gateway
kubectl logs -n dcs-system deploy/dcs-cloud-native-dcs-gateway | grep -i mqtt | tail -3
#   MQTT connection established   (and no "bad user name or password")

# Step 3 — RETIRE. The old account stops being served. Until this runs, the
# credential you are rotating away from still authenticates.
helm upgrade dcs deploy/helm/cloud-native-dcs --reuse-values \
  --set-string mqtt.auth.users.gateway.retireSlot=a

The next rotation runs the same three steps in the other direction: set password (slot a), flip activeSlot back to a, then retireSlot=b. Clear retireSlot in the step that flips, since the chart refuses a retireSlot that names the active slot.

Retiring is not optional, and the chart cannot enforce it. A slot you leave standing keeps the password it was serving. Flipping back onto it later therefore re-adopts a credential that was already in use, and nothing is rotated. Render-time values say nothing about which slot was active before, so this is a hazard the procedure carries and not one helm can catch.

Unit runtimes are the one manual step. Their pods are bare Pods written by the physical operator, so rollout restart does not touch them and the operator deliberately leaves running ones alone: recreating a runtime pod puts any batch that pod is executing on Hold. Flip the runtime slot, then delete the pods when each unit is idle.

# Step 2 for the runtime identity: the flip reaches new pods only.
helm upgrade dcs deploy/helm/cloud-native-dcs --reuse-values \
  --set-string mqtt.auth.users.runtime.passwordB="$(openssl rand -base64 24 | tr -d '=+/')" \
  --set-string mqtt.auth.users.runtime.activeSlot=b

# Then, per site, at a moment the units are idle:
for ns in $(kubectl get ns -l dcs.io/site -o name); do
  kubectl delete pod -n "${ns#namespace/}" -l dcs.io/component=unit-runtime
done
# Only once every runtime pod has been recreated:
helm upgrade dcs deploy/helm/cloud-native-dcs --reuse-values \
  --set-string mqtt.auth.users.runtime.retireSlot=a

Emergency (compromise suspected): the compromised credential has to stop working now, so the window is the wrong shape. Set the identity's active-slot password directly and accept the disconnect.

helm upgrade dcs deploy/helm/cloud-native-dcs --reuse-values \
  --set-string mqtt.auth.users.gateway.password="$(openssl rand -base64 24 | tr -d '=+/')"

That rolls the identity's pods on the new credential with no overlap, which is roughly 40 seconds of MQTT disconnect for that client. If the runtime identity is the compromised one, any batch in Running goes to Hold on the runtime side and an operator has to Resume it.

With mqtt.auth.existingSecret, the Secret is yours to key by account name. The chart still reads activeSlot to decide which key each client mounts, and it cannot check that the key or its ACL rules are there. Add the -b key and a matching ACL block before flipping.

Unit Runtime Bearer Token

The runtime-facing REST API (port 61152) authenticates with a bearer token in the <release>-runtime-auth Secret (key token). Every unit-runtime pod mounts the same token.

# 1. Generate a new token and push it through helm.
NEW_TOKEN=$(openssl rand -hex 32)
helm upgrade dcs deploy/helm/cloud-native-dcs \
  --reuse-values \
  --set-string unitRuntime.auth.token="$NEW_TOKEN"

# 2. Delete every unit-runtime pod so the Unit reconciler recreates it
#    with the new token (bare Pods — rollout restart does not touch them,
#    see the note below).
for ns in $(kubectl get ns -l dcs.io/site -o name); do
  kubectl delete pod -n "${ns#namespace/}" -l dcs.io/component=unit-runtime
done

The operator clients (physical, procedural, control) read the same <release>-runtime-auth Secret. They pick up the new token on their next reconcile without a restart. If they don't, restart them:

kubectl rollout restart deploy/dcs-cloud-native-dcs-physical-operator \
  deploy/dcs-cloud-native-dcs-procedural-operator deploy/dcs-cloud-native-dcs-control-operator \
  -n dcs-system

Runtime pods are bare Pods, not Deployments. The kubectl rollout restart deploy -n site-<name> command that used to be in this section does not restart the runtime pods themselves — they are created directly by the Unit reconciler as standalone Pods, and Kubernetes rollout only operates on workload controllers (Deployment / StatefulSet / DaemonSet). Unlike the runtime mTLS rotation (where the Unit reconciler watches the Secret and recreates Pods via the dcs.io/runtime-mtls-rv annotation — see #211), the runtime-auth Secret has no such watcher. To deliver the new bearer token to a runtime Pod you must manually delete it:

for ns in $(kubectl get ns -l dcs.io/site -o name); do
  kubectl -n "${ns#namespace/}" delete pod \
    -l dcs.io/component=unit-runtime
done

The Unit reconciler will immediately recreate the deleted Pods with the new token injected as an env var. Surfaced during the #168 drill on 2026-04-21.

HMAC Signing Key

The gateway stores the HMAC-SHA256 signing key(s) for e-signatures and audit records in a versioned Kubernetes Secret: dcs-signing-key in kube-system. Each key version is stored under key.<id> (e.g. key.v1, key.v2), and a current field names the active signing key. The gateway auto-generates the first key (v1) on startup.

Key-ID chaining ensures historical signatures remain verifiable after rotation: each ElectronicSignature and BatchRecordFinalization carries a keyID field that identifies which key version was used. The verifier selects the matching key from the store. Legacy signatures (created before key-ID chaining) have an empty keyID, and the verifier falls back to trying every key in the store for them.

Session tokens are chained the same way. The gateway mints a session JWT under the current key and stamps that version into the token's kid header, then verifies each token against the version it names. A rotation therefore leaves signed-in users signed in: their tokens were issued under a key the store still holds, and they expire on their own schedule. Removing an old key version does end the sessions it signed, which is one more reason step 1 below says to retain old keys.

Rotation procedure

The rotation runbook below, performed end to end: a v1-signed approval, the new key version patched in alongside the old one, the gateway reloading the store, the supervisor's session carrying across the restart because its token names a key version the store still holds, a v2-signed release, and every signature's key version still resolving.
  1. Generate a new key and add it to the Secret alongside the existing key(s). Do not remove old keys. 
    # Generate a 32-byte random key.
NEW_KEY=$(openssl rand 32 | base64)

# Patch the Secret to add the new key version and set it as current.
kubectl patch secret dcs-signing-key -n kube-system --type merge \
  -p "{\"data\":{\"key.v2\":\"$NEW_KEY\",\"current\":\"$(echo -n v2 | base64)\"}}"
  2. Restart the gateway to pick up the new key store: 
    kubectl rollout restart deploy/dcs-cloud-native-dcs-gateway -n dcs-system

Signed-in users stay signed in, as long as step 1 was followed and the old key versions are still in the Secret. Tokens issued under the previous version name it in their kid header, the restarted gateway still holds that key, and those sessions run to their normal expiry. Remove an old key version and the sessions it signed end at the next restart, with browser sessions bouncing to the login page and CLI sessions needing dcs login again.

  1. Verify that every historical signature still resolves to a key: 
    dcs audit verify
    All records should show PASS or WARN (legacy, no keyID). If any show FAIL, the Secret is misconfigured. Check that the old key data was not modified.

What this command proves. For active records it checks that each signature's keyID is present in the key store. It does not recompute the digest, because the AuditRecord does not carry the payload that was signed. That is exactly the check this step needs — it catches the one mistake rotation can make, which is dropping a key version that existing records still reference — but it is not a cryptographic re-verification of the signatures themselves. The archived path (dcs audit verify --archived) does recompute digests, because the archive stores the covered records. 4. Test a new signature: approve a draft recipe or finalize a batch record and confirm the AuditRecord carries the new keyID.

Compromised key procedure

If a specific key version is suspected compromised:

  1. Add a new key version (as above) and set it as current.
  2. Do not remove the compromised key from the Secret — it is still needed to verify historical records signed before the compromise.
  3. Document the compromise window in a compliance note (AuditRecord).
  4. Run dcs audit verify to confirm all signatures remain structurally valid.

Secret format reference

apiVersion: v1
kind: Secret
metadata:
  name: dcs-signing-key
  namespace: kube-system
data:
  current: djI=          # base64("v2") — active signing key ID
  key.v1:  <base64>      # original key (32 bytes)
  key.v2:  <base64>      # rotated key (32 bytes)

Legacy format (single key field) is auto-detected and treated as keyID="v1". No migration is required.

Gateway Setup Token (retired)

gateway.auth.setupToken was the pre-shared secret that unlocked /api/v1/setup* before OIDC was live. The setup endpoints and the token mechanism were removed in June 2026 together with the in-product k3s join brokering (ADR 0004, #487). There is no setup-token credential to rotate on current versions. The 2026-04-21 drill entry below is retained as a historical record.

LDAP Bind Password (LDAP Mode Only)

When the gateway is in auth.mode: ldap, the bind password lives in the Secret named by gateway.auth.ldap.bindPasswordSecret, key password.

# 1. Rotate the password in your directory.
# 2. Update the Secret.
kubectl create secret generic <secret-name> \
  --from-literal=password="<new-password>" \
  --dry-run=client -o yaml \
  | kubectl apply -n dcs-system -f -

# 3. Restart the gateway so the new password is read on startup.
kubectl rollout restart deploy/dcs-cloud-native-dcs-gateway -n dcs-system

# 4. Verify a user can still log in.
The four steps below performed against a live directory, including the window between step 1 and step 3 when no one can sign in.

The bind password is read once, from the LDAP_BIND_PASSWORD environment variable, when the gateway process starts. Step 3 is therefore load-bearing: updating the Secret alone changes nothing until the pod restarts.

Between step 1 and the end of step 3 the gateway is binding with a password the directory no longer accepts, so no one can sign in. Keep the old password valid in the directory until the restart completes if your directory allows two, or plan the window.

Logins during that gap fail in a way an operator can act on. A failed service-account bind means the directory is unreachable, and it proves nothing about anyone's credential. The gateway therefore answers 503 with authentication backend unavailable, where a wrong user password draws 401, and the attempt does not count toward the SR 1.11 consecutive-failure lockout. An operator who sees that message should stop retrying their own password and check this runbook's step 3.

OMF Endpoint Credential

The northbound OMF egress authenticates to a host somebody else administers, usually a PI Web API server. The credential is HTTP Basic by default and lives in the Secret named by omfEgress.auth.existingSecret, keys username and password. A deployment that set omfEgress.auth.username and omfEgress.auth.password inline instead is holding the credential in the Helm release. For that deployment, step 2 below is a helm upgrade.

The old and new values cannot be made to overlap on our side, because the overlap would have to exist in the PI Server's own account store. Whether it can is the PI administrator's answer to give. The MQTT client passwords used to have the same problem, and that one was ours: it is fixed by the credential slots above (#1579).

# 1. Rotate the credential in the PI System (or wherever the OMF endpoint
#    authenticates). Keep the old one valid until step 3 if that is possible.

# 2. Update the Secret the egress reads.
kubectl create secret generic pi-omf-credential \
  --from-literal=username="<user>" \
  --from-literal=password="<new-password>" \
  --dry-run=client -o yaml \
  | kubectl apply -n dcs-system -f -

# 3. Restart the egress so the new credential is read on startup.
kubectl rollout restart deploy/dcs-cloud-native-dcs-omf-egress -n dcs-system

# 4. Confirm the endpoint is accepting again. The readout that matters is
#    the metric, not the pod — a pod whose credential is refused is Ready.
kubectl port-forward -n dcs-system deploy/dcs-cloud-native-dcs-omf-egress 8096:8096 &
curl -s localhost:8096/metrics | grep dcs_omf_egress
#   dcs_omf_egress_endpoint_up      back to 1
#   dcs_omf_egress_queue_depth      falling
#   dcs_omf_egress_dropped_total    not advancing

The credential is read once from the environment when the process starts, so step 3 is load-bearing. Updating the Secret alone changes nothing.

Two properties of this component make the rotation gentler than it looks, and one makes it sharper.

The gentle part is that the gap between step 1 and step 3 is an endpoint outage, which is the case the egress is built to ride out. It buffers, retries with backoff, and posts in order when the endpoint accepts it again. A rotation done inside the buffer window therefore loses nothing.

The sharp part is that the buffer is bounded and in memory. A rotation left half-finished past omfEgress.buffer.queueLimit records starts dropping the oldest, and the pod restart in step 3 empties whatever is still queued. Do the restart promptly, and do not restart the pod between steps 1 and 2. That throws away the buffer without having fixed the credential.

Read dcs_omf_egress_dropped_total after any rotation on this component. It is the only readout that distinguishes a rotation that cost nothing from one that lost a window of plant history, and it does not recover on its own.

For auth.mode: producerToken, the key is producerToken and everything else above is unchanged.

Historian DB Password

With historian.database.cnpg.enabled, the CloudNativePG operator manages the PostgreSQL superuser and application passwords. Rotate through CNPG. Never edit the Secret directly.

# 1. Check the Cluster name and managed secrets.
kubectl get cluster -n dcs-system

# 2. Rotate the app user password (CNPG-native).
kubectl cnpg psql <cluster-name> -- \
  -c "ALTER USER app PASSWORD '<new-password>';"

# Update the Kubernetes Secret the historian reads.
kubectl edit secret -n dcs-system <cluster-name>-app

# 3. Restart the historian so it reconnects with the new password.
kubectl rollout restart deploy/dcs-cloud-native-dcs-historian -n dcs-system

# 4. Verify WAL shipping is uninterrupted.
kubectl cnpg status <cluster-name>

With an external PostgreSQL (historian.database.externalURL), the password lives wherever the external DB admin stored it. Update the external URL Secret and restart the historian.

k3s Node Join Token

The k3s join token is install-time and is not normally rotated. It only matters when adding a new node. Treat a compromised join token as a cluster-rebuild event (it implies the attacker can provision new controllers). That event is out of scope for this runbook. File a platform issue.

Drill History and Notes

Observations recorded during the #168 drill campaign. Each paragraph is dated. Add new ones at the top so the most recent run stays in view.

2026-08-15 — MQTT credential slots (the window that works, #1579)

Drill target: the three-step rotation documented above, run as three helm upgrade --reuse-values against a live release on a kind cluster. The gateway identity was rotated end to end, then rotated back.

  • The window is real. Step 1 put dcs-gateway-b in the auth Secret. The broker loaded it 100 seconds later without restarting, and from that point both credentials authenticated at the same time, from one passwd file. Slot a's password was byte-identical before and after the upgrade, which is the lookup preservation the whole procedure rests on.
  • The flip cost nothing. Step 2 moved the gateway to dcs-gateway-b. Slot a stayed accepted throughout, and the gateway logged zero authentication failures across the roll.
  • Retiring works, and did not before this drill. Step 3 removed the key from the Secret immediately and the broker dropped the account 50 seconds later. The old credential was refused from then on, with no effect on the running gateway.
  • The next rotation is a real rotation. Flipping back to slot a generated a fresh password. It did not re-adopt the retired one, and that credential stayed refused. This is what step 3 buys.
  • The propagation delay is the thing to plan around. Between a helm upgrade and the broker serving the change sit kubelet's Secret sync and the reloader's 10-second poll: 50 to 100 seconds in this drill. Step 2 must not overtake step 1, which is why the procedure gates on reading the account out of the broker's own passwd file.
  • Found here, fixed here. The first attempt failed step 3. The Secret used stringData, which Kubernetes merges into data and then clears. A key the chart stops rendering is therefore a key Helm's three-way merge has nothing to diff against. retireSlot rendered a manifest with the account gone and left the credential authenticating three upgrades later. Every render test agreed the account was gone, because in the manifest it was. The Secret carries data now, and the chart test asserts that it never goes back.
  • Method note. The first attempt drove the docs-shots capture stack. That stack applies helm template output and holds no Helm release, and lookup returns nothing outside helm install/upgrade. Every apply therefore regenerated every password, and no window could stay open. A rotation drill has to be run against a real release.

2026-08-14 — MQTT graceful rotation window (found broken, #1579)

Drill target: the dual-credential window the 2026-04-21 entry below asked for. #221 shipped it on 2026-04-22, one day after that drill, and nothing drilled it in the four months since. The runbook and the drill log disagreed about whether the graceful path existed, which is what sent someone to check (#1552).

  • The window has never worked, on any Mosquitto 2.x. A passwd file holds one password per username, and the chart's helper wrote a second line for the same username whenever a rotationPassword was set.
  • Drilled by lifting the helper script out of _helpers.tpl and running it, unmodified, in the broker image the chart pins. On 2.0.18 and 2.0.21 the broker starts and accepts only the last entry, so the credential every running client holds is rejected the instant the window opens. On 2.1.2, which is what the pinned eclipse-mosquitto:2 resolves to today, the broker reports Duplicate user 'dcs-gateway' in password file and terminates.
  • Both of the chart's roads reach it. The init container builds the same file on a cold install, so a fresh install with a window configured never comes up. The passwd-reloader sidecar SIGHUPs a healthy broker into a crash loop on the upgrade that opens one.
  • Fixed here: the chart refuses a rotationPassword at render time, and the helper refuses to emit the duplicate entry so an operator's own mqtt.auth.existingSecret carrying rotation keys leaves the broker up on the primary credential. MQTT password rotation was the single-step path above until #1579 shipped a mechanism Mosquitto can serve, which is the credential slots of ADR 0061.
  • Why it survived four months: #221's acceptance was signed off by a chart test that renders the templates and greps the YAML for the Secret keys. That proves what the manifest says. The question the values raise is what the broker does with the file, and only a running broker answers it. pkg/mqtt now runs the chart's own script against one under make test-interop.

2026-04-21 — MQTT client passwords (non-graceful)

Drill target: helm upgrade --set-string mqtt.auth.users.*.password=… + parallel client rollout. Graceful (dual-credential window) path is blocked on a missing chart helper. Drilled the non-graceful path per the #168 decision.

  • Full rotation wall-clock: 125 seconds. Helm upgrade + rollout trigger: ~90s. Pod rollout itself: 39s.
  • MQTT broker was rolled automatically by helm (the Deployment spec changed via its Secret reference annotation). Broker pod creationTimestamp confirmed it restarted during the upgrade window. Mosquitto came back with the new bcrypted passwd file loaded from the refreshed mqtt-auth Secret mount.
  • Gateway log around reconnect: connecting to MQTT broker → MQTT connection established → subscribes resumed. No bad username/password or auth errors after reconnect.
  • Broker disconnect events during the window ("Protocol error") were kubelet probe connections hitting the TLS port without a TLS handshake, unrelated to the password rotation.
  • Gap: the chart still lacks a dual-credential helper so a graceful window where old + new passwords both work simultaneously is impossible without a hand-rolled ACL overlay. 125s of client disconnect is acceptable for annual rotation in a planned window but would be uncomfortable for anything more frequent. Follow-up issue to file for the chart helper (mqtt.auth.users.*.rotationPassword).
  • Closed on 2026-08-15, by a third route. #221 answered this gap the next day with the rotationPassword knob. The 2026-08-14 entry above found that the knob never opened a window. The gap this paragraph names was therefore continuously true for four months. #1579 closed it with credential slots, and the entry at the top of this log is the drill.

2026-04-21 — Unit runtime bearer token rotation (partial)

Drill target: helm upgrade --set-string unitRuntime.auth.token=… followed by operator + runtime-pod roll.

  • Secret dcs-cloud-native-dcs-runtime-auth rotated correctly: RV 2681 → 23867, token SHA-256 changed.
  • helm upgrade wall-clock: 273 seconds on this small cluster, longer than hoped because the chart also re-renders every component with the new token reference, triggering rolling restarts where pod templates changed.
  • Post-upgrade pod ages: physical-operator and control-operator rolled (pod creationTimestamp matches the upgrade window). procedural-operator did not roll (pod still 90+ min old). The procedural-operator likely uses the same Secret via envFrom but wasn't triggered because its Deployment spec didn't change during the upgrade (a subtle behavior worth calling out in production).
  • Finding: the original runbook suggested kubectl rollout restart deploy -n site-<name> to roll runtime pods. That command does nothing for runtime pods because they are bare Pods created directly by the Unit reconciler, with no Deployment behind them. Patched the runbook with the correct kubectl delete pod -l dcs.io/component=unit-runtime command scoped across all dcs.io/site-labeled namespaces.
  • Not exercised on this cluster: runtime Pod actually picking up the new token. Same limitation as drill #6: no IOModule chain.

2026-04-21 — Per-site runtime Certificate rotation (partial)

Drill target: the #213 per-site Certificate flow. site-drill-site namespace was created by the Site reconciler. A runtime Certificate dcs-cloud-native-dcs-runtime-mtls was auto-created by the reconciler and signed by the cluster-wide mTLS CA ClusterIssuer.

  • Pre-drill ClusterIssuer check: kubectl get certificate -n site-drill-site showed spec.issuerRef.kind=ClusterIssuer pointing at dcs-cloud-native-dcs-mtls-ca, the cluster-wide CA. No per-site Issuer is involved, which confirms the #213 design.
  • cmctl renew -n site-drill-site dcs-cloud-native-dcs-runtime-mtls rotated the Secret in 28 seconds. Fingerprint changed, issuer still the cluster CA.
  • Trust chain: per-site ca.crt fingerprint matched the cluster CA Secret fingerprint byte-for-byte. The ca-injector path is working.
  • Unit reconciler did fire on the Secret change (two reconcile log entries within seconds of the cert rotation), confirming the Watches() wiring from #211 is live. However, the Unit on this drill cluster has no IOModules (no full Controller/IOModule/ControlModule/ ControlModuleTemplate chain), so no runtime Pod was ever created. The pod-recreate observation (the dcs.io/runtime-mtls-rv annotation flipping and the bare Pod being deleted + recreated) was therefore not exercised in this drill. That code path is covered by the Unit reconciler's unit tests. Drilling end-to-end requires a cluster with a working reference-plant Unit setup and is a follow-up.

2026-04-21 — mTLS CA cert re-issue

Drill target: force CA rotation via cmctl renew and observe downstream impact. Cluster: drill cluster with 7 components deployed, mTLS enabled.

  • CA re-issue: 3 seconds. But: CA identity CHANGED (fingerprint DA:5A:41… → 63:E1:A7…, serial also changed). The prior version of the runbook claimed "CA identity stays the same". That claim was wrong for cert-manager 1.18+ default privateKey.rotationPolicy: Always. Updated the runbook to document the real behavior and the required cmctl renew --all follow-up.
  • Immediately after CA renewal, component Secrets' ca.crt stayed STALE (still held old CA) for every component except the one whose cert I force-renewed next. That one (gateway-mtls) picked up the new CA via cert-manager's secret rewrite. All 4 operators remained on old CA in their trust store.
  • Ran cmctl renew -n dcs-system --all (the runbook's documented full re-key step): 31 seconds wall-clock. Afterwards every component's ca.crt fingerprint matched the live CA.
  • No TLS errors logged on any component during the out-of-sync window. Can't rule out that the drill cluster is idle enough that no cross-component mTLS handshake happened during the stale-trust window. A production cluster with active batch traffic would likely have seen handshake failures.
  • Action taken: rewrote the mTLS CA section to (a) drop the "identity stays the same" claim, (b) make cmctl renew --all a mandatory post-renewal step, (c) ship a verify-sync snippet that diffs each component's ca.crt fingerprint against the live CA.

2026-04-21 — HMAC signing key rotation (v1 → v2, key-ID chaining)

Drill target: graceful HMAC key rotation per the procedure documented above. Cluster: drill cluster. Secret dcs-signing-key initially held only key.v1 (auto-generated on first gateway startup).

  • Full rotation wall-clock (patch + gateway rollout): 10 seconds.
  • Pre-rotation gateway log: loaded HMAC signing keys {currentKeyID: v1, totalKeys: 1}.
  • After patch + rollout: loaded HMAC signing keys {currentKeyID: v2, totalKeys: 2}, with both keys loaded and current pointing to v2. Key-ID chaining is functional.
  • dcs audit verify ran cleanly against the gateway (returned 0 records since the drill cluster is fresh, with no signed AuditRecords yet). The verify path works end-to-end. Verification across a real corpus of v1-signed records requires a cluster with batch history. Re-drill on a cluster with pre-rotation signed records is a nice-to-have follow-up.
  • No gotchas. The procedure as documented produced the expected state.

2026-04-21 — Gateway setup token revocation

Drill target: helm upgrade --set gateway.auth.setupToken="" + gateway rolling restart. Cluster: drill cluster running gateway.auth.mode: none (no OIDC configured on the drill box).

  • Baseline (pre-revocation, with auth.mode: none): GET /api/v1/setup returned HTTP 200 with no token, with wrong token, and with the valid token. The setup token was not gating anything because mode: none already exposes the admin endpoints publicly.
  • Revocation (setupToken="" + gateway rollout): 29 seconds end-to-end.
  • Post-revocation: same GET still returns HTTP 200 with any or no token. No setup-marker ConfigMap was created (none/revoked doesn't trigger the setupTokenInvalidatedAt write path).
  • Finding: the runbook's "revoke the setup token" procedure is a no-op in mode: none. Noted inline in the Gateway Setup Token section. A production cluster running mode: oidc or mode: ldap is the only configuration where the drill is meaningful. The setup token is the pre-OIDC bootstrap admin, and once OIDC is live, revocation closes the bootstrap door. Re-drilling on an OIDC-configured cluster is a follow-up, filed separately.

2026-04-21 — MQTT broker TLS (dcs-cloud-native-dcs-mqtt-tls)

Drill target: broker cert renewal via cmctl + rolling restart of the mosquitto Deployment. Cluster: same as the mTLS drill.

  • Secret resourceVersion bumped after 14s (slower than the 4s for component-mTLS, likely because the mqtt-tls private key rotated too under cert-manager 1.18's default privateKey.rotationPolicy: Always).
  • Broker rolling restart: 31s (single-replica on drill cluster — an HA StatefulSet with multiple instances would be longer).
  • Gateway MQTT client logged MQTT connection lost, will reconnect followed by MQTT connection established within the same second. No messages queued beyond the broker outage window.
  • Gotcha surfaced: the MQTT broker cert lifetime is ~90 days (cert observed notAfter = Jul 20 16:33:39 2026 from a renewal on 2026-04-21), where component mTLS uses the 24h default. The runbook's original "Same as components (24h)" row was wrong. The table now shows ~90d. Drill exposed this inconsistency.

2026-04-21 — mTLS component cert (gateway-mtls)

Drill target: release-namespace Deployment cert renewal via the graceful (automatic cert-manager) path. Cluster: fresh DO k3s droplet, chart at defaults, mTLS with ClusterIssuer mode. Method: cmctl renew -n dcs-system dcs-cloud-native-dcs-gateway-mtls, observe.

  • Secret resourceVersion bumped in 4 seconds after the cmctl call (1605 → 13179). New cert notAfter advanced by ~1h to reflect a fresh 24h lifetime.
  • Gateway pod restartCount stayed at 0 — pkg/tlsutil hot-reload works as documented, no rolling restart needed.
  • No x509 or TLS errors in gateway logs in the 2-minute window around renewal.
  • Gotcha surfaced during drill setup (not part of this rotation path but worth recording): chart template bug in templates/mqtt-certificate.yaml and templates/webhook-certificate.yaml hardcoded $issuerKind = "Issuer" in the mtls-CA fallback branch, which breaks mqtt-tls and webhook-tls on any install using the default mtls.certManager.clusterIssuer: true (i.e. every default install post-#213). Fixed in the same working session by using ternary "ClusterIssuer" "Issuer" .Values.mtls.certManager.clusterIssuer. Without this fix, drill #8 (MQTT broker TLS) would have failed on renewal with Referenced "Issuer" not found.