Disaster Recovery Runbook

Status: Partial. Scenarios are marked drillable today, blocked on infrastructure, or blocked on feature work — see the per-section banners. 21 CFR Part 11 §11.10(c) requires tested restore procedures, not just documented ones; the compliance row stays at Implemented (archival) / Untested (restore) until every drillable scenario has a measured drill date. Tracker: #166.

Recovery procedures for Cloud-Native DCS disaster scenarios. This doc complements Backup and Recovery, which focuses on what is backed up. This doc focuses on how to restore, with RTO/RPO targets and verification.

How to read this runbook

Every scenario section carries one of three banners:

• Drillable today — the procedure can be exercised on the available infrastructure (single-node k3s is sufficient). Run the drill, fill the measured RTO/RPO in the table below, and stamp Last drill with today's date.
• Blocked on infrastructure — the procedure is written but cannot be exercised on the current clusters (needs a multi-node HA cluster, a second edge device, or re-enabled CNPG S3 backups). Keep the procedure current, and mark the drill pending when the infra lands.
• Blocked on feature work — the scenario depends on platform capabilities that are not yet implemented (tamper-evident audit archival, Object Lock storage). Do not drill, since the outcome would be misleading. Resolve the linked feature issue first.

Do not promote a row in the compliance traceability matrix based on a tabletop walkthrough alone. A drill means an actual restore on an actual cluster, with wall-clock RTO measured and a signed Target.Kind=Qualification AuditRecord attached as evidence (see Restore Validation below).

RTO / RPO Targets

Tentative targets below are aspirational until drilled. Rows pinned to n/a — pending drill in Measured RTO / Last drill have not been exercised end-to-end on a real cluster.

Component	RPO target	RTO target	Measured RTO	Last drill	Drill status
CRDs (recipes, equipment, procedures)	Last snapshot	Minutes	95 s (sqlite/kine proxy drill)	2026-04-22 (sqlite proxy)	Rolled back all post-snapshot CRs on the DR-drill k3s; markers disappeared and count returned to the pre-snapshot baseline. See Full Etcd Loss.
etcd (control plane)	Last snapshot	15 min	95 s (sqlite/kine proxy drill) / etcd primary-path: pending	2026-04-22 (sqlite proxy)	Sqlite/kine equivalent of the restore flow drilled on the DR-drill k3s (which ships without --cluster-init). Embedded-etcd primary path still needs a drill on a --cluster-init-provisioned cluster. See Full Etcd Loss.
Historian (tag data, S3 + WAL)	Continuous WAL	30 min	91 s (correct-path) / 632 s (incl. 2 learning iterations)	2026-04-22	Drilled on two-droplet k3s with DO Spaces. See Historian Database Loss — Procedure (primary path).
Historian (tag data, pg_dump fallback)	Last pg_dump	1 hour	388 s (6.5 min)	2026-04-22	Drilled on two-droplet k3s (ctrl + edge). See Historian Database Loss — Procedure (fallback path).
Audit records (hot, in etcd)	Last etcd snapshot	Minutes	95 s (sqlite/kine proxy drill)	2026-04-22 (sqlite proxy)	Pre-snapshot AuditRecords (e.g., qual-20260422-182643) survived the rollback intact; post-snapshot command audit records rolled back as expected. See Full Etcd Loss.
Audit records (archived, in historian)	Continuous	1 hour	<1 s detection / 124 s full recovery	2026-04-22	Drilled on two-droplet k3s with immutable.enabled=true pointed at in-cluster minio with Object Lock (COMPLIANCE, 365-day per-object retention). One manifest covering 3 records mutated via UPDATE audit_records; dcs audit verify --archived flagged digest_mismatch; bundle pulled from minio via mc cp; transactional BEGIN/UPDATE×3/COMMIT restored canonical message field; re-verify clean. See Audit Archive Corruption. Signed qualification: qual-20260422-192940-443c0887.
Unit runtime state (edge device)	N/A (derived)	Minutes	41 s (power-on → pod Running) / 132 s wall-clock	2026-04-22	Drilled on two-droplet k3s (ctrl + edge). See Node Loss (Single Edge Device).
Control-plane quorum (3-node HA)	Last etcd snapshot	30 min	n/a — pending drill	n/a — pending drill	Blocked on infrastructure — no HA k3s cluster exists in the project

Scenario: Full Etcd Loss

Drillable today. Single-node k3s is sufficient as long as it was installed with --cluster-init (embedded etcd). A default k3s install uses kine/sqlite and has no etcd datastore — see the "Backend detection" subsection below before running this drill. Do not drill against a live or shared instance — spin up a throwaway k3s VM and install a minimal Cloud-Native DCS release seeded with a representative CR set.

Partial drill 2026-04-22 on the DR-drill two-droplet k3s (dcs-dr-ctrl + dcs-dr-edge). The control-plane was provisioned via scripts/dr-drill-infra.sh which did not pass --cluster-init, so the cluster ran kine/sqlite. Ran the functionally-equivalent sqlite snapshot + restart flow: 95 s wall-clock RTO, rollback of all post-snapshot CRs verified, AuditRecords preserved, dcs command safety interlocks enforced. The embedded-etcd primary path (k3s --cluster-reset) still needs a dedicated drill on an appropriately-provisioned cluster. Signed qualification: qual-20260422-183656-feb850f7.

An etcd loss is recoverable whenever a recent snapshot exists. The detailed procedure below was written and drilled against k3s (the DR-drill infrastructure). The Talos path for the reference deployments is summarized first.

Talos path (reference deployments)

Both reference deployments run Talos (since 2026-05-12, #363), which always uses real etcd, so no backend detection is needed. Snapshot and restore are talosctl operations (see the Talos v1.10 disaster-recovery guide):

# Snapshot (any healthy control-plane node):
$ talosctl -n <control-plane-ip> etcd snapshot db.snapshot

# Restore: wipe the ephemeral partition, then bootstrap from the snapshot
$ talosctl -n <control-plane-ip> reset --graceful=false --reboot \
    --system-labels-to-wipe=EPHEMERAL
$ talosctl -n <control-plane-ip> bootstrap --recover-from=./db.snapshot

After the control plane is back, resume at step 6 of the k3s procedure below (wait for nodes/controllers, then run dcs qualify dr). The validation and stuck-procedure triage steps are platform-independent. A measured Talos drill is still pending. The k3s drill results below do not transfer.

The guide is linked at a version-pinned URL because the Siderolabs documentation site serves no unversioned path for that page. It is pinned to v1.10 because the reference deployments run Talos v1.10.9. The recovery procedure changes between Talos minor versions, so a link to whatever release is current would describe a platform this project does not run. A Talos bump on the reference deployments therefore has to move this link and the matching one in Backup and Recovery. Upgrade and Rollback carries that step.

k3s path (DR-drill infrastructure)

On k3s, etcd snapshots come from the built-in schedule (every 12 hours by default), plus on-demand via scripts/etcd-snapshot.sh. The procedure below restores the full control plane from a single snapshot file.

Backend detection

Before running this procedure, confirm that k3s is actually running embedded etcd. Otherwise k3s etcd-snapshot save fails with etcd datastore disabled and the rest of the procedure does not apply.

# ls /var/lib/rancher/k3s/server/db/

• state.db (+ state.db-shm, state.db-wal) → kine/sqlite backend. The k3s etcd-snapshot / cluster-reset flow does not apply. Fall back to the sqlite-equivalent procedure at the end of this scenario.
• etcd/ subdirectory with member files → embedded etcd. Proceed with the full procedure below.

If a drill cluster needs to use embedded etcd, reprovision with INSTALL_K3S_EXEC='server --cluster-init …' (the --cluster-init flag is what switches the backend). scripts/dr-drill-infra.sh does not currently set this. Add it there if you plan to drill the embedded-etcd path on that infrastructure.

Preconditions

• Snapshot file produced by scripts/etcd-snapshot.sh (copies to /var/lib/dcs/backups/etcd/ with retention) or by k3s's automatic schedule (/var/lib/rancher/k3s/server/db/snapshots/).
• Root/sudo access on the k3s server node.
• A known-good baseline expected-state file (for example pkg/qualify/baselines/dr-default.yaml, or a customer-approved derivative).

Procedure

1. Capture a baseline of expected state.

On a healthy cluster, prior to the drill:

# kubectl get crd -o name | wc -l
# kubectl get batches.batch.dcs.io -A --no-headers | wc -l
# kubectl get procedures.procedural.dcs.io -A --no-headers | wc -l

Record the CRD count and the per-CR counts. The restore must match these: ±0 for CRDs, and pre-outage CR counts as the floor for everything else. In-flight work created during the outage window is lost RPO.

1. Stop the k3s server on the node that will run the restore.

# sudo systemctl stop k3s

Expected: systemctl status k3s shows inactive (dead).

1. Identify the most recent consistent snapshot.

# sudo ls -lht /var/lib/dcs/backups/etcd/ | head -5
# sudo ls -lht /var/lib/rancher/k3s/server/db/snapshots/ | head -5

Pick the newest snapshot older than the earliest known-bad timestamp. Verify the file size is non-zero and the .sha256 sidecar (if present) matches.

1. Restore from the snapshot.

# sudo k3s server \
    --cluster-reset \
    --cluster-reset-restore-path=/var/lib/dcs/backups/etcd/on-demand-NODE-TIMESTAMP.db

Expected tail (~30–60 s):

INFO[...] Managed etcd cluster membership has been reset, restart without --cluster-reset flag now

The command exits. k3s is not running yet, and that is expected.

1. Restart k3s normally.

# sudo systemctl start k3s
# sudo systemctl status k3s   # expect: active (running)

1. Wait for the API server and core controllers.

# sudo k3s kubectl wait --for=condition=Ready node --all --timeout=5m
# sudo k3s kubectl -n kube-system wait --for=condition=Ready pods --all --timeout=5m

1. Wait for Cloud-Native DCS controllers.

# sudo k3s kubectl -n dcs-system wait --for=condition=Available deploy --all --timeout=5m

Expected: all five deployments ready (cloud-native-dcs-physical-operator, cloud-native-dcs-procedural-operator, cloud-native-dcs-batch-operator, cloud-native-dcs-control-operator, cloud-native-dcs-gateway).

1. Run scripted restore validation. See Restore Validation for the single-command dcs qualify dr invocation. Stop the wall-clock here. This is the RTO boundary.

2. Triage stuck procedures. Any procedures or batches that were transitioning at the snapshot instant are now frozen in an intermediate state. See Scenario: Stuck Procedure After Restore.

3. Record measured RTO/RPO in the RTO/RPO table above and stamp Last drill with today's date. Attach the dcs qualify dr report's AuditRecord name to the drill log.

If this fails

• Snapshot missing or corrupt. Fall through to the next-oldest snapshot. You lose more RPO but avoid a longer outage. If no snapshot is usable, the cluster is un-recoverable from etcd alone. Rebuild from declarative manifests (Helm + applied CRs) and accept loss of all runtime/status fields since the last manifest snapshot. The CR half of that rebuild is dcs restore crds, and it targets a cluster with nothing executing: the gateway refuses a document whose target is running (ADR 0060), and the restore stops at the first refusal. Carrying on would write the rest of a plant model over a live plant. In this scenario the cluster is empty, so every document is a create and nothing can be refused. A refusal here means you are pointed at the wrong cluster. See Restore CRDs from Backup.

• k3s server --cluster-reset exits non-zero. Check journalctl -u k3s -n 200 --no-pager for the actual error. Common causes: ownership mismatch on /var/lib/rancher/k3s/server/db/ (chown to root:root), disk full, or a corrupt snapshot (try the next-oldest).

• Operators crashloop after restart. kubectl -n dcs-system logs -l app.kubernetes.io/name=cloud-native-dcs --tail=200 --prefix. Typical causes: cert-manager issuer not ready yet (wait), missing secrets that were not backed into the etcd snapshot (re-run helm upgrade with --reuse-values to re-materialize secrets from Helm values), or a CRD schema mismatch because the cluster was restored from a snapshot taken before a CRD migration (roll the chart forward to match the snapshot epoch, then roll again to current).

• dcs qualify dr reports FAIL. The report enumerates which checks failed. CRD-missing and pod-not-ready failures usually indicate the restart is not finished. Wait a minute and re-run. Helm-values-digest drift means someone edited values out of band. Reconcile against the approved values YAML before trusting the restore.

Fallback procedure (kine/sqlite backend)

If the cluster was installed without --cluster-init, k3s stores state in state.db (sqlite via kine) and the etcd CLI does not apply. The online-backup + file-swap flow below is the functional equivalent. This is what the 2026-04-22 DR drill exercised, at 95 s wall-clock RTO.

# On the k3s server node:

# Online snapshot while k3s is running (sqlite .backup API):
sqlite3 /var/lib/rancher/k3s/server/db/state.db \
    ".backup '/var/lib/dcs/backups/sqlite/state-$(date +%F-%H%M%S).db'"

# --- incident occurs ---

# Stop k3s, keep current state.db as evidence, restore:
systemctl stop k3s
cp /var/lib/rancher/k3s/server/db/state.db \
   /var/lib/dcs/backups/sqlite/state-at-incident.db
rm -f /var/lib/rancher/k3s/server/db/state.db-shm \
      /var/lib/rancher/k3s/server/db/state.db-wal
cp /var/lib/dcs/backups/sqlite/state-<SNAPSHOT_TS>.db \
   /var/lib/rancher/k3s/server/db/state.db
chown root:root /var/lib/rancher/k3s/server/db/state.db
chmod 600       /var/lib/rancher/k3s/server/db/state.db
systemctl start k3s

After k3s restarts, run dcs qualify dr --fail-on-deviation exactly as in step 8 of the primary procedure. The acceptance criterion is identical. Operators will rediscover the rolled-back CRs and reconcile. Expect a few restarts (RESTARTS=1-2) on the operator pods as they re-list. The sqlite WAL/SHM files must be removed before copying the snapshot in, or sqlite will refuse to open the database because the WAL ≠ main-file.

Scenario: Historian Database Loss

Blocked on infrastructure for the S3 + WAL path. The reference cluster has CNPG backup disabled since 2026-04-13, when a misconfigured WAL archiver filled the root disk (see the comment in cndcs-deploy-demo's flux/clusters/demo/apps/dcs-release.yaml). The S3 path is drillable only on a separate test cluster with backup re-enabled. The pg_dump fallback path is drillable today on any cluster.

CNPG stores base backups + WAL in S3-compatible object storage when enabled. The primary restore is point-in-time recovery from those artifacts. Without S3, the fallback is a pg_dump output loaded into a fresh CNPG Cluster: coarser RPO, no point-in-time, but functional.

Procedure (primary path: CNPG + S3 PITR)

Drilled 2026-04-22 on a two-droplet k3s cluster (dcs-dr-ctrl + dcs-dr-edge) with DO Spaces as the S3 endpoint. Wall-clock RTO was 91 s on the correct-spec path; the first two attempts each wasted ~5 min on misconfiguration pitfalls that are now called out in the template below. RPO in the drill was ~9.5 min because we deliberately induced loss 9.5 min after the last base backup; a production cluster with continuous WAL archiving sees RPO in seconds. Signed qualification: qual-20260422-182643-22ffd036.

Preconditions: historian.backup.enabled=true at install time with historian.backup.s3.{endpointURL,bucket,path,secretRef} populated, and at least one completed ScheduledBackup run visible in kubectl get backups.postgresql.cnpg.io -n dcs-system before the inducement. Without a base backup in the bucket there is no WAL chain anchor to replay from, and you are in the fallback path below.

Gotchas the 2026-04-22 drill surfaced:

• bootstrap.recovery.database and owner must mirror the original initdb.database/owner. Default app DB is app. If you omit these, CNPG creates an app DB that the recovered backup never populated, and the -db-app secret's uri still says …/app. The historian writer then connects to an empty schema-less DB and crashloops on migration 001.

• serverName in the recovery cluster's barmanObjectStore must be new. CNPG's pre-bootstrap check rejects a non-empty archive path (barman-cloud-check-wal-archive: Expected empty archive) to prevent timeline collisions. Rotate the suffix per attempt (-recovered-YYYYMMDD[-vN]) or leave backup: off the recovery cluster entirely if you don't need the recovered cluster to push new WAL.

• Detect and confirm data loss. Rule out transient query-planner regressions, blocked connections, and ingestion pauses before declaring loss. Restoring a working database is more disruptive than finishing a long query.

# kubectl -n dcs-system exec -it deploy/cloud-native-dcs-historian -- \
    dcs historian tags --tag <known-good-tag> --from "$(date -u -d '5 min ago' +%Y-%m-%dT%H:%M:%SZ)"

If this returns empty while MQTT ingestion is healthy, data loss is real. Record outage_start as the wall-clock timestamp of the first failing query.

1. Identify the most recent base backup and WAL chain.

# kubectl -n dcs-system get backups.postgresql.cnpg.io \
    -l cnpg.io/cluster=cloud-native-dcs-historian-db \
    --sort-by=.status.stoppedAt \
    -o custom-columns=NAME:.metadata.name,STARTED:.status.startedAt,PHASE:.status.phase | tail -5

Pick the newest completed backup. Its backupId becomes the bootstrap.recovery.backup.name in step 4.

1. Stop the historian writer so it cannot ingest into the doomed cluster while the restore is in flight.

# kubectl -n dcs-system scale deploy cloud-native-dcs-historian --replicas=0
# kubectl -n dcs-system wait --for=delete pod \
    -l app.kubernetes.io/component=historian --timeout=60s

1. Apply a recovery Cluster with bootstrap.recovery pointing at the backup objectstore. Use a distinct name (-restore) so the original Cluster object stays quarantined as evidence until the restore is verified green.

# cat <<'YAML' | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloud-native-dcs-historian-db-restore
  namespace: dcs-system
spec:
  instances: 1
  imageCatalogRef:
    apiGroup: postgresql.cnpg.io
    kind: ClusterImageCatalog
    name: cloud-native-dcs-timescaledb
    major: 16
  podSecurityContext: { runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000 }
  postgresql:
    shared_preload_libraries: [timescaledb]
  storage: { size: 20Gi }
  bootstrap:
    recovery:
      # Must match the original cluster's initdb.database/owner
      # (see "Gotchas" above). Without these, the CNPG-generated
      # app secret points at an empty default `app` DB.
      database: historian
      owner: historian
      source: cloud-native-dcs-historian-db
      recoveryTarget:
        targetTime: "TARGET_TIMESTAMP_UTC"   # ← outage_start - 1m
  externalClusters:
    - name: cloud-native-dcs-historian-db
      barmanObjectStore:
        destinationPath: "s3://BUCKET/historian"
        endpointURL: "SPACES_ENDPOINT"       # e.g., https://nyc3.digitaloceanspaces.com
        s3Credentials:
          accessKeyId:     { name: cloud-native-dcs-historian-backup-s3, key: ACCESS_KEY_ID }
          secretAccessKey: { name: cloud-native-dcs-historian-backup-s3, key: ACCESS_SECRET_KEY }
YAML

Substitute TARGET_TIMESTAMP_UTC = outage_start - 1m (RFC 3339, UTC). The -1m is the deliberate test: the query in step 6 asks for data that must exist in the restored database.

1. Wait for the recovery Cluster to finish replay.

# kubectl -n dcs-system wait cluster/cloud-native-dcs-historian-db-restore \
    --for=condition=Ready --timeout=15m
# kubectl -n dcs-system exec cloud-native-dcs-historian-db-restore-1 -- \
    psql -U postgres -d historian -c "SELECT pg_is_in_recovery();"

Expected: condition Ready=True and pg_is_in_recovery() = false (recovery mode exited, server is accepting writes). Stop the wall-clock here. This is the RTO boundary.

1. Verify tag continuity across the restore point.

# kubectl -n dcs-system patch deploy cloud-native-dcs-historian --type=merge -p \
    '{"spec":{"template":{"spec":{"containers":[{"name":"historian","env":[{"name":"HISTORIAN_DB_HOST","value":"cloud-native-dcs-historian-db-restore-rw"}]}]}}}}'
# kubectl -n dcs-system scale deploy cloud-native-dcs-historian --replicas=1
# dcs historian tags --tag <known-good-tag> \
    --from "$(date -u -d 'outage_start -1m' +%Y-%m-%dT%H:%M:%SZ)" \
    --to   "$(date -u -d 'outage_start -30s' +%Y-%m-%dT%H:%M:%SZ)"

Expected: at least one sample in the minute before the outage. If the window is empty the restore regressed past the WAL tail. Re-apply step 4 with a later targetTime that still predates the inducement.

1. Promote the restored cluster once verification passes. Rename by swapping the rw service endpoint, then delete the original Cluster. Keep the most recent backup of the original as evidence attached to the deviation report.

2. Record measured RTO/RPO in the RTO/RPO table and stamp Last drill with today's date. Expected RPO is the WAL archive lag: seconds for a healthy barman-cloud-wal-archive, and the 2026-04-13 incident saw it balloon to hours when the bucket filled.

Procedure (fallback path: pg_dump restore)

Use this path when either (a) no S3 backup is configured, or (b) the objectstore itself is the failure domain (e.g., the S3 endpoint is unreachable during the incident).

Drilled 2026-04-22 on a two-droplet k3s cluster (dcs-dr-ctrl + dcs-dr-edge). Steps 2–6 below are what was exercised; wall-clock RTO 388 s for a 50-row corpus. Restore itself was trivial (0 s stream-in); the dominant cost was the new CNPG primary's initdb + TimescaleDB image pull (~5.5 min). Signed qualification: qual-20260422-181008-3428ada6.

1. Locate the most recent pg_dump output and validate it before anything else is touched:

# dcs backup verify /var/lib/dcs/backups/historian/dcs-historian-2026-08-15.sql

Dumps are captured by dcs backup historian. That command runs pg_dump inside the primary Pod and writes the sidecar integrity manifest verify reads. It refuses a dump it cannot prove ran to completion. See backup-recovery.md "pg_dump Fallback (Administrator)". A dump captured with a shell redirection has no manifest, so verify cannot be run over it at all. Step 5 is then the first thing that would notice a truncated file. 2. Stop the historian writer by scaling cloud-native-dcs-historian to 0 replicas. 3. Re-create the CNPG Cluster (delete the old one if it still exists, since a helm upgrade reapplies the templated Cluster CR). Wait for status.phase = "Cluster in healthy state" and status.readyInstances = 1. On a cold node this is bounded by the TimescaleDB-HA image pull (~300 MB). 4. Re-scale historian to 0 again after the helm upgrade. It will otherwise crashloop against a freshly initdb'd DB that has no historian schema yet. The previous step's helm reapply restarts it at replicas=1. 5. Stream the dump back in: kubectl exec -i <primary> -c postgres -- psql -U postgres -d historian -v ON_ERROR_STOP=1 < dump.sql. The dump already contains the TimescaleDB CREATE EXTENSION and hypertable metadata, so the _hyper_*_chunk child tables get re-registered automatically. 6. Scale historian back to 1. Verify ingestion resumes by publishing a test MQTT sample and confirming it lands in tag_values. 7. Record the RPO as last_pre_outage_write_ts - last_dump_row_ts, typically on the order of hours (cadence of the dump cron).

If this fails

• Base backup present but WAL gap. CNPG PITR requires a continuous WAL chain. A gap (common when WAL archiving was failing unnoticed) forces recovery to the base-backup epoch only, a much coarser RPO. This is the failure mode that caused the 2026-04-13 incident. Monitor barman-cloud-wal-archive exit codes.
• Fresh CNPG cluster won't bind to the restored volume. Check that the new Cluster spec's storage.storageClass and claim size match the snapshot. CNPG refuses to start a restore into a smaller volume.

Scenario: Node Loss (Single Edge Device)

Drillable today. Exercised 2026-04-22 on a two-droplet k3s cluster (dcs-dr-ctrl + dcs-dr-edge, NYC1) spun up via scripts/dr-drill-infra.sh. Reproduce by standing up that harness, pinning a workload to dcs.io/device=edge-01 with 30 s unreachable/not-ready tolerations, then power-cycling the edge droplet. Do not drill against a live or shared instance.

Unit runtime uses hostPath for FB-network persistence, so losing the node means losing the persisted runtime state. The declarative state (Unit CR) still lives in etcd.

Measured RTO (2026-04-22 drill)

Event	Elapsed from power-off
Node Ready=Unknown (k3s detection)	~50 s (default node-monitor-grace-period)
Pod Pending (30 s toleration expires)	~80 s
Power-on issued	90 s
Node Ready=True again	~125 s
Pod Running on reattached node	132 s

Wall-clock RTO: 132 s. From the moment the edge droplet powers back on until the pinned workload is Running again: 41 s. RPO is 0 for declarative state (CRs live in ctrl-plane etcd which was untouched). The hostPath-persisted FB network survives a clean reboot, so runtime RPO is 0 as well.

Signed AuditRecord: qual-20260422-175914-536330b2 (retrieve with dcs audit export --target-kind Qualification).

Procedure

1. Confirm the unit-runtime pod is stuck, with transient restarts ruled out (kubectl -n <site> describe pod <runtime>, kubectl get node <nodeName>, NotReady > 5 min with no pending kubelet reconnect).
2. If hardware is unrecoverable, join a replacement device at the deployment layer and register it (see Device Enrollment).
3. Rebind the Unit CR's nodeSelector label to the new device (kubectl patch unit ... --type=merge -p '{"spec":{"nodeSelector":{"dcs.io/device":"<new-name>"}}}').
4. Verify the runtime replays the last-known FB network from its local hostPath. It will: each device keeps its own copy, so the new device starts "fresh" and the operator re-applies the declarative UnitSpec.
5. Verify I/O drivers reconnect to their remote I/O endpoints.

If this fails

• New device has no cached FB network. Expected: a fresh device starts with no /var/lib/dcs/fb-cache. The operator will reconcile the UnitSpec and produce a fresh network. I/O will be offline briefly while the new runtime starts.
• I/O addresses were device-local. Should not happen. Addresses are encoded in IOModule CRs, which live in etcd and survive. If this does happen it is a bug in the product, and no recovery procedure covers it.

Scenario: Node Loss (Control Plane, Multi-Node)

Blocked on infrastructure. No 3-node HA k3s cluster exists anywhere in the project. Provisioning one is a prerequisite for this drill. Open a separate infra issue before filling this section.

Quorum-loss recovery for an HA control plane follows upstream guidance for the platform: on Talos, talosctl bootstrap --recover-from=<snapshot> on one surviving member, then rejoin the others (see the Talos disaster-recovery guide). On k3s, restore the surviving node's etcd snapshot via --cluster-reset, then add fresh replacement members. Procedure is intentionally left as a pointer to upstream until we have a cluster to test against.

Scenario: Certificate Expiry

Drilled 2026-04-22 on the DR-drill two-droplet k3s (dcs-dr-ctrl + dcs-dr-edge). Walked the cert-manager-failed degenerate case end-to-end: scaled cert-manager to zero, corrupted the gateway-mtls Secret, killed the gateway pod so the failure surfaced (EOF on /api/v1/system/health, "tls: failed to find any PEM data" in pod logs), manually re-issued an ECDSA P-256 cert from the auto-created CA Secret using openssl, patched the Secret, rolled the gateway, confirmed health. Wall clock from cert-manager-down to gateway-healthy: 123 s. Signed qualification: qual-20260422-192458-ca6ef6e0.

Normal cert rotation is handled by cert-manager + the rotation runbook: cmctl renew (graceful) or the rotation runbook's "mTLS Component Certificate (Emergency)" path. The runbook's emergency path assumes cert-manager is recoverable. This scenario covers the unrecoverable case: cert-manager itself is down and one or more component certs are minutes from expiry. The fallback is openssl + the auto-created CA Secret.

Procedure (manual emergency re-issue)

The gateway is the canonical example because it is the most externally-visible component. The same procedure applies to any other component with a cert-manager.io/Certificate in dcs-system (the operators, io-probe, historian).

1. Confirm cert-manager is the failure mode. Healthy cert-manager would have already renewed. If it has not, something in the controller / webhook / cainjector trio is wedged.

kubectl -n cert-manager get pods
kubectl -n cert-manager logs deploy/cert-manager --since=15m | tail

If cert-manager is recoverable (e.g. crashlooping on a transient API outage, OOMKilled), fix it first and let cmctl renew --all handle the rotation. That is the rotation runbook's path. Only continue here if the cert window is shorter than the cert-manager recovery ETA.

1. Pull the CA cert + key from the in-cluster CA Secret. This is the break-glass material. The auto-created CA is in cloud-native-dcs-mtls-ca in the release namespace.

mkdir -p /tmp/cert-emergency && cd /tmp/cert-emergency
kubectl -n dcs-system get secret cloud-native-dcs-mtls-ca \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ca.crt
kubectl -n dcs-system get secret cloud-native-dcs-mtls-ca \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ca.key
openssl x509 -in ca.crt -noout -subject -dates

1. Mint a replacement cert with the original Certificate's dnsNames. The dnsNames are pinned in the cert-manager.io/Certificate CR. Do not free-hand them. Algorithm defaults are ECDSA P-256 (matches mtls.certManager defaults).

kubectl -n dcs-system get certificate cloud-native-dcs-gateway-mtls \
  -o jsonpath='{.spec.dnsNames}' ; echo
openssl ecparam -name prime256v1 -genkey -noout -out gateway.key
cat > csr.cnf <<'EOF'
[req]
prompt = no
distinguished_name = dn
req_extensions = req_ext
[dn]
CN = dcs-gateway
[req_ext]
subjectAltName = @alt_names
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
[alt_names]
DNS.1 = cloud-native-dcs-gateway
DNS.2 = cloud-native-dcs-gateway.dcs-system.svc
DNS.3 = cloud-native-dcs-gateway.dcs-system.svc.cluster.local
EOF
openssl req -new -key gateway.key -out gateway.csr -config csr.cnf
openssl x509 -req -in gateway.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 1 -out gateway.crt -extfile csr.cnf -extensions req_ext

-days 1 is deliberate. These break-glass certs must be replaced with cert-manager-issued certs as soon as cert-manager is healthy, and a tight expiry forces the followup.

1. Patch the component Secret + roll the deployment. Preserve ca.crt so peers still validate.

kubectl -n dcs-system create secret generic cloud-native-dcs-gateway-mtls \
  --from-file=tls.crt=gateway.crt \
  --from-file=tls.key=gateway.key \
  --from-file=ca.crt=ca.crt \
  --type=kubernetes.io/tls --dry-run=client -o yaml \
  | kubectl apply -n dcs-system -f -
kubectl -n dcs-system rollout restart deploy/cloud-native-dcs-gateway
kubectl -n dcs-system rollout status deploy/cloud-native-dcs-gateway --timeout=120s

1. Verify. The health endpoint comes back, dcs health is green, and pod logs no longer show TLS errors.

dcs health
kubectl -n dcs-system logs deploy/cloud-native-dcs-gateway --since=2m | grep -iE 'tls|x509'

1. Restore cert-manager and force a canonical re-issue. Once cert-manager is back, delete the manually-issued Secret so cert-manager re-creates it from its own Certificate CR, then roll the component once more.

kubectl -n cert-manager scale deploy --all --replicas=1
kubectl -n cert-manager rollout status deploy --timeout=120s
kubectl -n dcs-system delete secret cloud-native-dcs-gateway-mtls
# Wait ~10s for cert-manager to recreate the Secret, then:
kubectl -n dcs-system rollout restart deploy/cloud-native-dcs-gateway

1. File a deviation report AuditRecord. Capture cert-manager failure mode, openssl break-glass justification, and the re-issued cert's serial / fingerprint. The 1-day expiry on the manually-issued cert is itself the action item, and the followup ticket should be filed against the original incident.

What this drill measured

Step	Wall clock
cert-manager scaled to 0	0 s (T0)
Gateway pod failure surfaced (EOF on health)	~25 s
Manual cert minted (openssl) + Secret patched + rollout restart issued	~85 s
Gateway rollout complete and health green	123 s (T1)

The bulk of the wall clock is openssl invocation + kubectl round-trips. A templated re-issue script checked into ops would compress this further.

If this fails

• Step 2: the CA Secret itself is missing or unreadable. Then cert-manager hasn't merely failed. The ClusterIssuer's own material has been wiped. There is no break-glass CA to re-sign against, and the deviation has escalated to a CA-loss incident. The mTLS CA Certificate rotation procedure covers that case separately, and it assumes a healthy cert-manager. Restore the CA Secret from a Helm-state backup before proceeding, or rebuild the cluster. There is no in-cluster path.

• Step 4: the patched Secret is rejected. Confirm the Secret type is kubernetes.io/tls (the operator pkg pkg/tlsutil expects this) and that tls.crt + tls.key match. A mismatch typically surfaces as tls: private key does not match public key in the component's startup logs.

• Step 6: cert-manager re-issued the Secret but the gateway still trusts only the break-glass cert. Each component's Secret carries its own copy of ca.crt. After a CA rotation, run cmctl renew -n dcs-system --all to force every component's Secret to be rewritten with the current CA bundle (see the mTLS CA Certificate warning about CA fingerprint changes on renewal).

Cross-link: Secret and Certificate Rotation, the graceful and emergency cmctl-based paths.

Scenario: Audit Archive Corruption

Drilled 2026-04-22 on the DR-drill two-droplet k3s (dcs-dr-ctrl + dcs-dr-edge) with immutable.enabled=true pointed at an in-cluster minio bucket created --with-lock and default retention COMPLIANCE 30d (per-object retention 365d set by the archiver). One manifest covering 3 records corrupted via UPDATE audit_records, detected via dcs audit verify --archived (digest_mismatch), quarantined via AuditRecord (audit-quarantine-zc8pl), bundle pulled from minio with mc cp (proof: X-Amz-Object-Lock-Mode: COMPLIANCE, Retain-Until-Date: 2027-04-22T19:18:32Z), restored via transactional psql BEGIN/UPDATE×3/COMMIT, re-verify clean, deviation report filed (audit-deviation-lzcxh). Wall clock from corruption to clean re-verify: 124 s (detection itself was <1 s — single-manifest verify). Signed qualification: qual-20260422-192940-443c0887.

Drillable on any cluster with immutable.enabled=true. The three #216 primitives are all on main: per-batch HMAC manifests (Phase A), scheduled verification plus UI panel (Phase B), and the Object Lock mirror (Phase C). The drill is conditional because step 4 depends on historian.audit.archival.immutable.enabled=true having been in effect when the corrupted batch was archived — there is no way to retroactively produce an immutable copy of a batch that was never mirrored. Run the drill on a throwaway cluster with a pre-provisioned Object-Lock bucket; do not drill on a live instance or any cluster with live pharma batches.

The 2026-04-22 drill pulled the bundle by name against a bucket where nobody had issued a delete, so it did not exercise the delete-marker case step 4 now covers. A drill that wants to cover it issues mc rm "s3-alias/BUCKET/${OBJECT_KEY}" (no --vid) after step 1 and confirms the version-targeted read still recovers the bundle.

This scenario covers the detection-through-recovery loop for 21 CFR Part 11 §11.10(c). The archive is tamper-evident by construction: any mutation to an archived record, any deletion of a record covered by a manifest, or any attempt to forge a new manifest invalidates the HMAC signature and is surfaced by dcs audit verify --archived. The drill exercises detection, evidence preservation, and restoration from the immutable S3 copy.

Preconditions

• historian.audit.archival.immutable.enabled=true in the Helm values and at least one archival CronJob run has completed before the corruption is simulated (otherwise there is nothing in the bucket to restore from).
• Direct psql access to the historian database or a break-glass IAM role that can write to the audit_records table. Regular operators cannot produce the corruption we're detecting, which is the point.
• mc (or aws CLI) configured against the same bucket the archiver uses.

• A known-good baseline expected-state: the most recent dcs audit verify --archived run should have reported all manifests clean. Pull that report from the Archive Integrity panel's history or from an AuditRecord with Target.Kind=ArchiveIntegrityCheck created in the hour before the drill.

  If that baseline is not clean and the failures are missing_records on manifests older than your upgrade to 0.2.5, they are most likely orphans, with no tampering involved. Archivers before that release derived a random manifest ID per attempt, so every retry after a failed mirror upload committed a manifest that no record pointed at. The archiver now removes such rows on each run, but only when another manifest in the same namespace carries an identical digest and still holds its records. An empty manifest that is not a proven duplicate is left alone, because retention trimming can empty a legitimate manifest and that row is then the only surviving evidence those records existed. Run the CronJob once and re-verify before treating a stale missing_records as a finding.

Procedure

1. Simulate the corruption. Mutate one row in audit_records so its content no longer matches the canonical bytes the manifest digest was computed over. In a drill, a one-field rewrite is sufficient:

# psql "$DATABASE_URL" -c \
    "UPDATE audit_records SET message = 'tampered by drill'
     WHERE manifest_id = 'MANIFEST_UUID' AND uid = 'UID_UNDER_TEST';"

Pick MANIFEST_UUID + UID_UNDER_TEST from the manifest you want to break. Record the pre-mutation values of the row for the deviation report (the row you just rewrote is the evidence).

1. Detect the corruption. Run the verifier against the affected window:

# dcs audit verify --archived \
    --since "$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" \
    --output json

Expected: non-zero exit, and at least one results entry with reason: digest_mismatch naming MANIFEST_UUID. Stop the wall clock here for the detection-RTO target.

If the gateway.archiveIntegrity scheduler is enabled, the failure should also appear in the Archive Integrity panel's "Latest Verification" card within one scheduler interval (default 6h), and an AuditRecord with Result=Failure and Target.Kind=ArchiveIntegrityCheck will be written to the gateway Pod's namespace (default dcs-system). Capture that record's name. It is the primary audit-trail anchor for the deviation.

1. Isolate the affected manifest (never delete). Stamp an AuditRecord that marks the manifest as under quarantine. This is the 21 CFR Part 11 §11.10(e) trail of the quarantine decision:

# kubectl apply -f - <<'YAML'
apiVersion: audit.dcs.io/v1alpha1
kind: AuditRecord
metadata:
  generateName: audit-quarantine-
  namespace: dcs-system
  labels:
    audit.dcs.io/target: ArchiveIntegrityCheck
spec:
  timestamp: "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  action: Update
  result: Failure
  actor:
    component: dcs-operator
    userID: "OPERATOR_OIDC_SUBJECT"
  target:
    apiGroup: audit.dcs.io
    kind: ArchiveIntegrityCheck
    name: "MANIFEST_UUID"
  reason: "Quarantined after digest_mismatch detected during DR drill"
  message: "Manifest MANIFEST_UUID under quarantine pending restore from S3 mirror"
YAML

Do not delete or overwrite the tampered PG rows. The tampered row is evidence. Destroying it destroys the forensic record the deviation report needs.

1. Pull the authoritative copy from the immutable mirror. The object key is derived from the manifest's namespace and CreatedAt. Read the bundle by version. A name-addressed read is the one a delete marker fools. Object Lock protects the object version alone, so a DeleteObject call that names no version is accepted against a locked bundle. It appends a delete marker as the new latest version, and every read addressed by name then reports the object missing while the retained version sits intact underneath it.

# OBJECT_KEY="audit-archive/NAMESPACE/YYYY/MM/DD/MANIFEST_UUID.json"
#   (NAMESPACE and the date come from the manifest row's namespace
#    and created_at columns in audit_archive_manifest)

# mc ls --versions "s3-alias/BUCKET/${OBJECT_KEY}"
# VERSION_ID=$(mc ls --versions --json "s3-alias/BUCKET/${OBJECT_KEY}" \
    | jq -rs 'map(select(.isDeleteMarker != true)) | .[0].versionId')

# mc cp --vid "${VERSION_ID}" \
    "s3-alias/BUCKET/${OBJECT_KEY}" /tmp/archive-restore.json
# jq -r '.manifest.digest_sha256' /tmp/archive-restore.json \
    | base64 -d | xxd -p -c 32

mc ls --versions lists the retained PUT version even when a delete marker sits above it, and --vid reads straight through the marker. Listing is newest first, so .[0] of the non-marker versions is the live bundle. Use jq -s with no head -1 pipe. A key the archiver retried carries several PUT versions, and a head that closes the pipe early kills jq with EPIPE under a pipefail shell. Record VERSION_ID for the deviation report. Retention rides the version itself, so the version-targeted mc stat is what proves it:

# mc stat --vid "${VERSION_ID}" "s3-alias/BUCKET/${OBJECT_KEY}"
#   X-Amz-Object-Lock-Mode             : COMPLIANCE
#   X-Amz-Object-Lock-Retain-Until-Date: 2027-08-11T15:10:48.000Z

The bundle's manifest keys are snake_case (digest_sha256, signature_hex, key_id). The digest_sha256 value is base64, which is why the readback above decodes it before comparing. Sanity-check the bundle against the manifest row in PG. Those three fields should match the PG row even though the records array is known-good while the PG rows are tampered. That version was written under Compliance-mode Object Lock at archive time, so no party with credentials could have mutated it. The X-Amz-Object-Lock-Mode metadata above is the proof.

1. Restore the tampered records from the bundle. Reload the records array from the JSON bundle back into audit_records:

# jq -c '.records[]' /tmp/archive-restore.json | \
    while read -r rec; do
      UID=$(echo "$rec" | jq -r '.uid')
      NS=$(echo "$rec" | jq -r '.namespace')
      TS=$(echo "$rec" | jq -r '.ts')
      MSG=$(echo "$rec" | jq -r '.message')
      psql "$DATABASE_URL" -c \
        "UPDATE audit_records
           SET message = '$MSG'
           WHERE namespace = '$NS' AND uid = '$UID' AND ts = '$TS';"
    done

(In a real incident, write this as a transaction with explicit BEGIN; SELECT ... FOR UPDATE; UPDATE ...; COMMIT; and capture the psql session log for the deviation report. The one-liner above is the drill-simplified version.)

1. Re-verify. The same dcs audit verify --archived invocation from step 2 should now exit 0 and report every manifest clean. Stop the wall clock. This is the full-recovery-RTO target.

Steps 2, 5 and 6 of this procedure end to end: the clean baseline, the same dcs audit verify --archived naming digest_mismatch once a row is altered underneath it, and the archive proving itself again after the mirrored copy is restored. The corruption and the restore are injected off camera. What is filmed is the verification on either side of them.

1. File the deviation report. The report must include, at minimum:
2. Detection timestamp + the failing dcs audit verify --archived JSON body.
3. Quarantine AuditRecord name (the CR from step 3).
4. Object key, version ID, and mc stat --vid output for the S3 bundle used for restore (proof that the authoritative copy was immutable at time of access). If a delete marker was covering that version, its own version ID and creation time belong here too (see the first failure mode below).
5. Psql transaction log from step 5.
6. Post-restore green dcs audit verify --archived JSON body.
7. Root-cause analysis (for a drill, "simulated by DR runbook step 1", and for a real incident whatever the storage-level investigation turns up).

8. Quality-system signature per the site's SOP.

9. Record measured RTO/RPO in the RTO/RPO table and stamp Last drill with today's date. Two RTO values: detection (step 2) and full recovery (step 6). RPO for this scenario is effectively zero: the mirror captures every signed batch, so nothing is lost.

If this fails

• Step 4: the read reports "Object does not exist", or the prefix lists empty. Do not conclude that the archive is gone. An unversioned DeleteObject against a Compliance-locked bundle is accepted by the server: it appends a zero-byte delete marker as the new latest version and returns success, and a name-addressed mc cp/mc cat/mc ls then behaves exactly as it would if the object had never been written. List versions before drawing any conclusion:

# mc ls --versions "s3-alias/BUCKET/${OBJECT_KEY}"
#   [...]    0B ... d7423dab-... v2 DEL MANIFEST_UUID.json
#   [...] 1.0KiB ... 6d7960b3-... v1 PUT MANIFEST_UUID.json

A DEL row sitting above a PUT row means the bundle is intact and recoverable through step 4's --vid read. It also means somebody holding s3:DeleteObject on the bucket issued that delete, which is a second deviation in its own right and is very likely the same actor as the PG tampering. Record the marker's version ID and creation time, and treat the bucket credentials as compromised. Only if the key has no versions at all was the object never written. That is the next failure mode.

• Step 4: the S3 object was never written. The corrupted manifest was written by an archiver run that pre-dated immutable.enabled=true being flipped on. There is no restore source. The deviation report must record this as an unrecoverable loss of the affected records. The tampered PG rows stay in quarantine and the archive is flagged in the QMS as partial. The remediation is an organizational control: enable the feature mid-cluster only with a backfill, or accept the gap window in writing.

• Step 4: the S3 object's own digest does not match the PG manifest digest. Two legitimate causes. (a) The bundle was written by an older archiver whose canonical JSON format differed. Check schemaVersion in the bundle and re-canonicalize using the older format. (b) The signing key referenced by the manifest is no longer in dcs-signing-key, so dcs audit verify --archived correctly reports unknown_key. Re-mount the retired key version as key.<id> in the Secret and re-run the verify. Never modify the bundle. If neither (a) nor (b) applies, the bundle itself is compromised (Object Lock bypass or bucket-root abuse). Escalate to the QMS as a serious deviation.

• Step 5: the PG update transaction fails. The compressed TimescaleDB chunks older than 30 days cannot be updated in place without decompression. Decompress the affected chunk with SELECT decompress_chunk(show_chunks('audit_records', ...));, re-run the update, and optionally recompress when done. The archive retention policy (default 3 years) governs compression. A corrupted manifest older than the 30-day compression boundary will always need this step.

• Step 6: verify-archived still reports digest_mismatch. Confirm that the UPDATE affected every tampered row, beyond the one you first targeted. Re-run step 1's SELECT on a fresh window to spot any additional mutations. If the verify still fails with a clean set, the canonical bytes differ for a reason other than content. Check timezone drift in the ts / k8sCreationTimestamp columns. The CanonicalRecordBytes helper forces UTC on marshal, so this should be stable. A custom-imported archive might not be.

• Object Lock retention already expired. Objects pass out of lock after retentionDays. If a corruption is detected more than retentionDays after the batch was archived, the mirror no longer protects it. Bucket lifecycle rules should transition such objects to Glacier Deep Archive and never expire them, so the data is still retrievable (see backup-recovery.md). If the object was lifecycle-expired, this degrades to the same "unrecoverable loss" case as the first failure mode.

Scenario: Stuck Procedure After Restore

Drillable today as part of the etcd-loss drill (any etcd restore will leave in-flight procedures frozen in intermediate states).

Identification

After the etcd restore completes, enumerate procedures in intermediate states:

$ dcs get procedures
$ kubectl get procedures.procedural.dcs.io -A \
    -o jsonpath='{range .items[?(@.status.state=="Stopping")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'
$ kubectl get procedures.procedural.dcs.io -A \
    -o jsonpath='{range .items[?(@.status.state=="Holding")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'

Do the same for unitprocedures, operations, phases, and batches. ISA-88 intermediate states that indicate a transition was interrupted: Starting, Stopping, Holding, Unholding, Restarting, Resetting, Aborting, Pausing, Resuming, Clearing.

Decision: force state, or roll the batch?

Force the next stable state using dcs command when all of:

• The equipment is known to be safe in the target stable state (confirmed by the operator at the physical cell).
• The intermediate-state duration is short (< 15 min) and there is no material safety reason to treat the in-flight work as compromised.
• The target stable state matches the one the procedure was transitioning toward at snapshot time (e.g., Stopping → force Stop, Holding → force Hold).

$ dcs command Batch <batch-name> Stop --reason "post-dr-restore force-state, equipment confirmed safe"
$ dcs command Batch <batch-name> Hold --reason "post-dr-restore force-state, awaiting operator review"

Commands target the owning Batch (the only commandable kind) and propagate down to the in-flight procedure.

dcs command already requires --reason for irreversible commands (Stop/Hold/Abort/Reset), so the justification is captured in the resulting AuditRecord. Make the reason explicit. The audit trail is the regulatory record that the transition was a conscious recovery action, and an unexplained command reads as a normal one.

The force-state command on a held lot: the CLI refusing to send it without a justification, the batch leaving Held, and the reason read back out of the audit trail under the batch's own correlation ID. No disaster precedes this run and it is not a timed drill.

Roll the batch (abort, then start a new one from a checkpoint) when any of:

• Physical-cell state is unknown or unsafe (for example, a valve was mid-stroke and cannot be confirmed without manual inspection).
• The procedure had been in the intermediate state for > 15 min at the snapshot instant, meaning the transition itself may have been stuck, with no transient explanation left.
• Material integrity is in question (contamination, out-of-spec conditions during the outage).
• Quality deviation rules require a CAPA for any post-DR-restore batch regardless of state.

To roll:

$ dcs command Batch <batch-name> Abort --reason "post-dr-restore, quality requires roll"
# wait until state is Aborted
$ dcs get batches                                  # confirm batch is in terminal state
$ dcs create batch ...                             # new batch from appropriate checkpoint

Evidence preservation

Do not delete or edit stuck-procedure CRs. The intermediate-state AuditRecords from before the outage are the legal record of what the system was doing at the snapshot instant. Archive them via the normal audit-archival flow. Any post-restore force-state AuditRecord will carry the same CorrelationID as the originating batch, so the full recovery story is reconstructible from dcs audit trace <batchID>.

Before You Restore: is the backup whole?

A restore can only rebuild what the export captured, and an export captures only the kinds the gateway was permitted to read. Run the pre-restore check first. It answers integrity and completeness:

$ dcs backup verify /backups/dcs-crds-2026-03-27.yaml

PARTIAL BACKUP means the file is intact but a kind was never in it, almost always over a missing rule in the gateway ClusterRole. Do not start the drill clock on that file: take a fresh backup once the gateway can read the kind. If the outage means no better backup exists, restore it with --allow-partial and record the named kinds as a known gap in the drill evidence, because nothing downstream will report them again (dcs restore crds reports 0 failed over kinds that were never in the file). Background and the footer format: Backup and Recovery (#1299).

Restore Validation

Restore validation runs a scripted DR qualification baseline against the restored cluster and persists the report as a signed Target.Kind=Qualification AuditRecord, the same retention path as IQ/OQ/PQ. The drill therefore produces regulatory evidence, with no ad-hoc markdown log to lose.

The qualification machinery dcs qualify dr drives after a restore, on camera in its IQ/OQ/PQ form: dcs qualify iq comparing live cluster state to a quality-approved baseline, its PASS report persisting as a signed Qualification AuditRecord in the adjacent trail.

$ dcs qualify dr --fail-on-deviation

This command:

1. Fetches the cluster snapshot via the gateway's qualification endpoints (CRDs, operator pods, Helm release, ClusterRole), the same data path as dcs qualify iq.
2. Evaluates against the embedded dr-default baseline (pkg/qualify/baselines/dr-default.yaml): all DCS CRDs present, the four DCS operators and the gateway Ready, Helm release in deployed status. The DR baseline is stricter than the IQ default. It pins the full CRD set, beyond the execution-critical subset, and treats every listed CRD as FAIL-if-missing.
3. Emits a markdown report to stdout and POSTs a signed AuditRecord with Target.Kind=Qualification, Target.Name=dr-<runID>.
4. Exits non-zero if any check fails, suitable for CI-style use or for gating the end-of-drill wall-clock timer.

For formal pharma validation, pass an explicit approved baseline:

$ dcs qualify dr --baseline /path/to/approved-dr-baseline.yaml \
                 --fail-on-deviation

Retrieve DR drill evidence later via:

$ dcs audit trace <runID>

Beyond the automated baseline

The dcs qualify dr check covers the structural cluster invariants (CRDs, operators, Helm) but does not yet exercise data-plane continuity or detect stuck procedures automatically. Add the following manual checks to each drill until they can be automated:

• [ ] No procedures, operations, phases, or batches are stuck in an ISA-88 intermediate state (Stopping, Holding, Starting, etc.). See Stuck Procedure After Restore.
• [ ] Historian returns a tag value from the minute immediately before the outage (dcs historian tags --tag <name> --from <T-1m>).
• [ ] Audit records for the outage window are present and their electronic-signature digests verify (dcs audit verify).
• [ ] A canary batch runs start-to-finish on restored equipment.
• [ ] Alarms configured before the outage still fire on the same conditions.
• [ ] OIDC login still works and RBAC bindings are intact (run dcs commands as each role).
• [ ] mTLS between operators and runtimes still verifies (dcs health).

Automating the data-plane checks (canary batch, historian continuity, alarm verification, stuck-procedure detection) is tracked as a follow-up to the DR baseline work.

Drill Schedule

• Quarterly: tabletop walkthrough of each scenario (reading this document as a team, confirming command sequences still match the deployed code).
• Annually: full restore to an isolated cluster, run dcs qualify dr, verify the manual checks above, and update the RTO/RPO table.
• After every upgrade that changes a CRD schema, audit-archiver behaviour, or the Kubernetes platform version (Talos or k3s): re-run the etcd-loss drill on a disposable cluster built from the new images.

Related Documentation

• Backup and Recovery — what gets backed up and how
• High Availability — failure modes DR responds to
• Upgrade and Rollback — rollback uses DR primitives
• 21 CFR Part 11 Traceability — regulatory requirements for tested restore
• Rotation Runbook — credential rotation (cert-expiry scenario depends on this)