Backup and Recovery¶

Procedures for backing up and recovering all stateful components of the DCS. Designed for R&D and pilot-scale deployments. Application-level backups (dcs backup) are platform-neutral. Cluster-state (etcd) snapshots are a platform concern and depend on your Kubernetes distribution. Both Talos (the reference deployments) and k3s paths are covered below.

What Is Backed Up¶

Component	Method	RPO	RTO
CRDs (recipes, equipment, batches, procedures, control programs, alarms)	`dcs backup crds`	Last backup	Minutes (apply from YAML)
Audit records (21 CFR Part 11)	`dcs backup audit` + archival CronJob	Last backup (hot) / continuous (archived)	Minutes (apply) / DB restore
Historian database (tag values, alarm events, state transitions, archived audit)	CNPG ScheduledBackup to S3, or `dcs backup historian` where there is no S3	Continuous WAL shipping (S3) / last dump (`dcs backup historian`)	~10 min (CNPG restore)
etcd (full cluster state)	Platform snapshot tooling (`talosctl etcd snapshot` on Talos, `k3s etcd-snapshot` on k3s)	Last snapshot	~5–15 min (platform restore)

What Is NOT Backed Up¶

Secrets and signing keys (mTLS keys, MQTT credentials, OIDC client secret, e-signature HMAC keys, session signing key): excluded by design (IEC 62443-4-2 11.5.2 cautions against keys in backup sets, #973). dcs backup exports custom resources only; on recovery, secrets are re-provisioned or rotated per the rotation runbook. A restored backup never reintroduces old key material.
MQTT broker state (retained messages, client sessions): ephemeral, and retained messages repopulate immediately when publishers reconnect.
Unit runtime FB network caches (/var/lib/dcs/runtime/*/last-network.json): derived from ControlProgram CRDs, and the runtime re-fetches from the API server on startup.
Unit runtime MQTT queue (queue.jsonl): transient store-and-forward buffer, replayed automatically when MQTT connectivity is restored.

CRD Backup¶

Export all DCS custom resources as portable, re-appliable YAML:

# Back up all CRDs across all sites
dcs backup crds -o /backups/dcs-crds-$(date +%F).yaml

# Back up a specific site only
dcs backup crds -s newark-plant -o /backups/newark-plant-crds-$(date +%F).yaml

Every export also writes a sidecar integrity manifest (<file>.manifest.json: SHA-256, byte count, document count, source, timestamp). dcs backup verify <file> validates it, and that validation is mandatory before any restore (IEC 62443 SR/CR 7.3 RE(1), #973).

Scheduling¶

Add to the server's crontab for daily automated backup:

# Daily CRD backup at 1 AM
0 1 * * * /usr/local/bin/dcs backup crds -o /var/lib/dcs/backups/crds/dcs-crds-$(date +\%F).yaml

What Is Exported¶

The backup includes all DCS resource types across all API groups: physical model (Enterprise, Site, Area, ProcessCell, Unit, ControlModule, Controller, IOModule, SimulationPreset), recipes (MasterRecipe, ControlRecipe, and library resources), procedures and templates, control programs, function block types, batches, alarms, and alarm definitions.

Exported manifests are cleaned of runtime metadata (status, resourceVersion, managedFields, internal labels/annotations) so they can be directly re-applied.

Completeness: complete, partial, and unverified exports¶

An export can only contain the kinds the gateway was able to read. Before #1299 a kind the gateway could not read was skipped in silence, so a backup taken against a gateway ClusterRole that was missing one rule looked exactly like a backup of a cluster that had no objects of that kind. That is what happened to CoordinationBlock before #1222 added its rule.

Every export now ends with a completeness footer, written as YAML comments so the file stays restorable:

#
# dcs-export-complete: false
# dcs-export-skipped: [{"group":"control.dcs.io","kind":"CoordinationBlock","namespace":"site-newark-plant","class":"forbidden","message":"coordinationblocks.control.dcs.io is forbidden"}]
# PARTIAL EXPORT: 1 kind(s) could not be read. This file is NOT a complete
# record of the cluster and must not be treated as one. See docs/backup-recovery.md.
#   control.dcs.io/CoordinationBlock in site-newark-plant [forbidden]: coordinationblocks.control.dcs.io is forbidden

Read it as follows:

Footer	Meaning	`dcs backup` exit
`dcs-export-complete: true`, `dcs-export-skipped: []`	Every kind was read.	0
`dcs-export-complete: true` with `crd-not-installed` entries	The listed kinds have no CRD on this cluster, so they can hold no objects and nothing was lost.	0
`dcs-export-complete: false` with a `forbidden` entry	RBAC gap in the deployment. The gateway ClusterRole is missing a rule; the cluster may hold objects of that kind and none of them are in this file.	non-zero
`dcs-export-complete: false` with an `error` entry	A timeout or API-server error hid the kind. The next export may succeed, so the difference between two backups is not a real change to the plant.	non-zero
No footer at all	The file was truncated, or came from a gateway older than #1299 that could not tell the difference. Completeness is unverified — never read this as complete.	0, with a warning

The three classes are deliberately not equivalent: only crd-not-installed leaves the export whole. A forbidden is a defect in the deployment, and the fix is to add the missing rule to the gateway ClusterRole (gateway_role.yaml) and take a fresh backup. Accepting the file fixes nothing.

A partial export is still written to disk and can still be restored, since during an incident it may be the only backup available. dcs backup crds and dcs backup audit still exit non-zero over it, and dcs backup verify refuses it. Pass --allow-partial to accept the gap deliberately. The disclosure is still printed. The gaps are also copied into the sidecar manifest (complete, skippedKinds), and the export is recorded in the audit trail as a failure. A partial backup is therefore findable later without re-reading the file.

Audit Trail Backup¶

Audit records are critical for 21 CFR Part 11 compliance. The backup strategy has two tiers:

Hot tier (etcd, last 90 days): dcs backup audit -o /backups/audit-$(date +%F).yaml
Archive tier (PostgreSQL, up to 3 years): backed up via the historian database backup

# Export current audit records from etcd
dcs backup audit -o /backups/dcs-audit-$(date +%F).yaml

The audit archival CronJob (enabled via historian.audit.archival.enabled) automatically moves records older than 90 days from etcd to the historian PostgreSQL database.

Nothing orders that CronJob after the historian database, so on a fresh cluster its first scheduled run can fire while the database is still initialising. Each run waits historian.audit.archival.databaseWait (15 minutes by default) for the database to accept connections before it fails. See Troubleshooting for the symptoms and for how to clear a run that already failed.

Archive tamper-evidence (21 CFR Part 11 §11.10(c))¶

Every archival batch the CronJob writes produces a signed manifest row (audit_archive_manifest) alongside the archived records:

Digest: SHA-256 over the canonical-JSON concatenation of the batch's records (sorted by UID). Each record carries a manifest_id foreign key so the exact covered set is recoverable for re-verification.
Signature: HMAC-SHA256 over {namespace, batch range, record count, digest} using the rotating dcs-signing-key Secret in kube-system.

Re-verify the full archive periodically or after restoring from backup:

# Verify every manifest in the last 30 days.
dcs audit verify --archived --since "$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)"

The command exits non-zero and prints the specific failure reason (digest_mismatch, signature_invalid, unknown_key, count_mismatch, or missing_records) for any manifest that fails verification. This is the detection primitive behind the "Scenario: Audit Archive Corruption" entry in the DR runbook.

missing_records on an archive written before 0.2.5 deserves one check before it is treated as a finding. Earlier archivers derived a random manifest ID per attempt, so a failed mirror upload left the records attached to the first attempt's manifest while each retry committed another that nothing pointed at. Manifest IDs are now derived from the batch digest, so a retry lands on the same row. Each archiver run clears the older orphans it can prove redundant. Run the CronJob once and re-verify. Anything still failing is real.

A supervisor holds a running production lot through the reason ceremony and the matching AuditRecord appears in the Data app's Audit Trail seconds later carrying actor, action, timestamp, and the typed reason; a filter drill reaches a QA Lead's e-signed batch-record approval showing signer, signature meaning, and HMAC digest; the System app's Archive Integrity panel reads Clean over the signed archive manifests and dcs audit verify --archived re-proves the chain from a terminal — The archive chain proven live: the System app's Archive Integrity panel reading Clean over the signed manifests, and `dcs audit verify --archived` re-proving the chain from a terminal seconds after a Hold's reason ceremony lands in the trail.

Optional immutable S3 mirror (Object Lock / WORM)¶

Detection alone is not a restore path. If PostgreSQL storage is compromised, the archiver and its signed manifests go with it. Configure an S3-compatible bucket with Object Lock in Compliance mode to mirror every signed batch as one JSON object. Until its retention date expires, that object version cannot be overwritten or deleted by any identity, bucket root included. This is the canonical restore source for the DR "archive corruption" scenario.

Retention protects a version. It does not protect a name, and the distinction has an operational consequence worth stating plainly. A DeleteObject call that names no version is accepted against a locked bundle: it appends a zero-byte delete marker as the new latest version and returns success. The retained version survives underneath it, but every read addressed by name (mc cp, mc cat, a plain ListObjects) then behaves as though the bundle had never been written. The bytes are safe. A name-addressed restore does not find them. Two things follow. Step 3 below is a hard requirement, and the DR runbook's restore step reads the bundle by version ID.

# values.yaml
historian:
  audit:
    archival:
      immutable:
        enabled: true
        endpoint: "s3.us-east-1.amazonaws.com"   # or minio service DNS
        bucket: "dcs-audit-archive"              # must be pre-created
        region: "us-east-1"
        useSSL: true
        retentionDays: 2555                      # 7 years (21 CFR Part 211)
        credentialsSecret: "dcs-audit-mirror-s3" # keys: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
        egress:
          destinationCIDRs: ["52.10.0.0/16"]     # the endpoint, as CIDRs
          ports: [443]

The egress block is what lets the archiver reach the endpoint at all, and it has no default that could be right. The archiver's NetworkPolicy declares policyTypes: [Egress], so the pod reaches exactly the destinations that policy names. A NetworkPolicy cannot resolve s3.us-east-1.amazonaws.com. The address has to be written down. Writing ["0.0.0.0/0"] allows any destination, and it can only appear as a decision someone made. With networkPolicies.enabled and the mirror on, the chart refuses to render until one is set (#1514).

For a bucket served by a pod inside the cluster (a minio Deployment, which is what the DR drill and the capture stack both run), name the pod instead. An ipBlock cannot name a Service, and the ClusterIP is not the address the policy sees:

        egress:
          destinationPodLabels:
            app.kubernetes.io/name: minio
          destinationNamespace: ""   # defaults to the release namespace
          ports: [9000]

The archiver proves the endpoint answers when it starts, before it archives anything, and fails the run if it does not. That check exists because a run that finds nothing past the retention cutoff never touches the mirror: on a fresh deployment nothing crosses activeRetentionDays for 90 days, so without it a denied egress or a wrong endpoint would report a clean hourly no-op for a quarter while the restore copy was never written. A refusal from the endpoint counts as reachable. The bucket policy in step 3 below grants the archiver s3:PutObject and nothing else, so a 403 to a HEAD bucket is the correct deployment answering.

Required bucket setup (the chart does not bootstrap the bucket because Object Lock must be enabled at creation time and cannot be added retroactively):

Create the bucket with Object Lock enabled. On AWS: aws s3api create-bucket --bucket dcs-audit-archive --object-lock-enabled-for-bucket --region us-east-1 On minio: mc mb --with-lock <alias>/dcs-audit-archive.
Set a default retention policy matching retentionDays: aws s3api put-object-lock-configuration --bucket dcs-audit-archive ... so non-archiver writers (if any leak through) inherit immutability.
Required. Apply a bucket policy that restricts s3:DeleteObject* and s3:PutObjectLegalHold to break-glass identities only. Everyday IAM users and the archiver's service account get s3:PutObject only. This is the control that stops the delete-marker case above, so it is load-bearing. No other layer covers that case. The archiver never deletes, so the restriction costs nothing operationally. Without it, anyone holding the archiver's credentials can hide every mirrored bundle in one pass, and an attacker who has already tampered with the PostgreSQL rows holds exactly those credentials.
Configure lifecycle rules to transition objects to a cheaper storage class (e.g., S3 Glacier Deep Archive) at an age beyond typical recovery windows. Do not expire objects. Let Object Lock retention govern deletion.
Legal hold: during an active investigation or regulatory inspection, stamp s3:PutObjectLegalHold on every object in the relevant namespace/date prefix. Legal hold outlives retention and is removed only by an identity with s3:PutObjectLegalHold.

Object key layout is audit-archive/<namespace>/<YYYY>/<MM>/<DD>/<manifest_id>.json, the same scheme referenced by lifecycle JSON and dcs audit verify --archived reports. A failed upload does not delete the source AuditRecord CRs from etcd. The next archiver run replays the same manifest (idempotent on the PG side) and retries the upload. The primary query path remains PostgreSQL.

What that configuration looks like once it is running, on a bucket the archiver has been writing to:

The MinIO console signed in to the audit mirror bucket, walking the archiver's audit-archive/site-plant-01/YYYY/MM/DD key layout to a bundle keyed by manifest ID; the object's detail panel reads Retention Policy compliance and Legal Hold Off, and its raw headers carry X-Amz-Object-Lock-Mode COMPLIANCE with a retain-until date a year out; a terminal beside it runs dcs audit verify --archived and names the same manifest, and an all-versions delete issued as the store's root identity is refused with a banner reading that the object is WORM protected and cannot be overwritten, the object still listed behind it — The key layout above, on a live bucket: one JSON bundle per signed manifest, carrying the Compliance-mode retention the archiver stamped at `PutObject` time, and refusing an all-versions delete issued by the store's own root identity.

Exercise the full round-trip locally before turning the feature on in production:

# Starts minio in docker, creates an Object-Lock bucket, uploads one
# batch via the real pkg/audit/archive/mirror package, and asserts both
# halves of the guarantee: a version-targeted DELETE is refused, and an
# unversioned DELETE is accepted but leaves the retained version
# readable by version ID. Requires docker + mc + jq.
./scripts/test-audit-mirror-minio.sh

The chart-level counterpart runs the archiver CronJob end-to-end under a real Helm release against an in-cluster minio bucket: test/e2e/audit_mirror_test.go and make test-e2e-audit-mirror (nightly via .github/workflows/e2e-audit-mirror.yml). That spec is the regression gate for the historian.audit.archival.immutable.* chart values themselves: if the Helm wiring stops producing a CronJob that talks to an Object-Lock-enabled bucket, the nightly run fails.

Historian Database Backup¶

The historian uses CloudNativePG (CNPG) with TimescaleDB. Enable automated backups via Helm:

historian:
  enabled: true
  backup:
    enabled: true
    schedule: "0 0 3 * * *"       # daily at 3 AM UTC — CNPG cron is 6-field (leading seconds)
    retentionPolicy: "7d"
    s3:
      endpointURL: "http://minio:9000"
      bucket: "dcs-backups"
      path: "/historian"
      secretRef: "backup-s3-creds"  # Secret with ACCESS_KEY_ID + ACCESS_SECRET_KEY
      egress:
        destinationPodLabels:       # an in-cluster minio, named by its pods
          app.kubernetes.io/name: minio

This creates a CNPG ScheduledBackup CR that uses Barman to continuously archive WAL segments and take periodic base backups to S3-compatible storage.

The egress block is what lets the database pods reach the store at all, and it has no default that could be right. Their NetworkPolicy declares policyTypes: [Ingress, Egress], so they reach exactly the destinations that policy names. A NetworkPolicy cannot resolve minio or s3.us-east-1.amazonaws.com, so the address has to be written down. Use destinationCIDRs for a store outside the cluster, and destinationPodLabels for one served by a pod inside it: an ipBlock cannot name a Service, and the ClusterIP is not the address the policy sees. Writing ["0.0.0.0/0"] allows any destination, and it can only appear as a decision someone made. With networkPolicies.enabled and backups on, the chart refuses to render until one is set (#1516).

The port is not asked for again. It comes off endpointURL. An explicit port in the URL wins. Otherwise the scheme decides, and an empty endpointURL is AWS S3 on 443, which is where CNPG itself sends the backup. Set egress.ports only when the store answers on some other port than the URL names, such as a proxy in front of it. For a pod destination the port is the one the container listens on, which is not always the Service port.

A denied WAL archive fills the data volume

PostgreSQL does not drop a WAL segment it has not archived. It keeps retrying, pg_wal grows on the data volume, and what the operator eventually sees is a full disk. The network-policy denial itself never surfaces. That is the 2026-04-12 demo failure: archiving to DigitalOcean Spaces broke, pg_wal reached 38 GB, filled the droplet, and cascaded into disk-pressure evictions of every operator on the node. Check the store for a recent base backup after enabling backups on a cluster that enforces NetworkPolicy. A chart that renders is not yet proof that the packets arrive.

CNPG cron is 6-field, with a leading seconds field

ScheduledBackup.spec.schedule follows the Go cron format (per the CNPG backup docs), not the 5-field Kubernetes CronJob format. The first field is seconds. A 5-field crontab string is not rejected. It is silently reinterpreted with the first field as seconds, so "0 3 * * *" means hourly at HH:03:00 where the author intended daily at 03:00. Always set historian.backup.schedule explicitly as a 6-field expression.

pg_dump Fallback (Administrator)¶

For environments without S3, and for the case where the object store is itself the failure domain, take a dump with dcs backup historian:

dcs backup historian -o /var/lib/dcs/backups/historian/dcs-historian-$(date +%F).sql

This is the one backup subcommand that does not go through the gateway. It runs pg_dump inside the database's primary Pod and streams the result to the local file, so it needs a kubeconfig with exec rights in the namespace the chart is installed in. Pass --kubeconfig, --context or --system-namespace when the defaults are not what you want. The output is plain SQL and is restored with psql, which is the path the DR runbook drill exercised.

The command answers four questions an out-of-band kubectl exec leaves open.

It dumps the primary. A dump taken from a standby succeeds and is as stale as that standby's replication lag, which is the quantity the dump was meant to bound. A cluster carrying only replicas is refused.
It refuses an ambiguous namespace. A CNPG restore stands a second cluster beside the live one, so during the incident that matters most there are two primaries. Name the one to dump with --cluster.
It says whether the file is whole. An evicted Pod or a dropped connection leaves a dump that is valid SQL as far as it goes and restores without complaint. The command checks for pg_dump's own completion marker and exits non-zero without it, keeping the partial file and recording it as incomplete.
It writes the integrity manifest. A shell redirection writes none, so an out-of-band dump could not be validated by dcs backup verify before a restore at all.

The dump covers every site. The historian is one database for the whole install and the site is a column in it, so --site cannot narrow this export. An installation pointed at an external PostgreSQL through historian.database.externalURL has no Pod to exec into. Back that database up with its own tooling.

Schedule it beside the CRD and audit exports:

# Daily at 02:00, retained by the same sweep as the other exports
0 2 * * * /usr/local/bin/dcs backup historian -o /var/lib/dcs/backups/historian/dcs-historian-$(date +\%F).sql

A dump is a point-in-time copy, so its RPO is the cron cadence. A healthy WAL archive gives seconds. Where S3 is available, keep the ScheduledBackup as the primary path and treat the dump as the fallback.

etcd Snapshot¶

etcd stores all Kubernetes state (CRDs, secrets, configmaps). Snapshotting it is a platform concern, and no dcs CLI feature covers it. Use your Kubernetes distribution's tooling. dcs backup crds and dcs backup audit cover the application-level state in a portable, re-appliable form regardless of platform. The etcd snapshot additionally captures secrets, Helm release state, and resource status.

Talos (the reference deployments): take a consistent snapshot from any healthy control-plane node:

talosctl -n <control-plane-ip> etcd snapshot \
  /var/lib/dcs/backups/etcd/dcs-$(date +%F-%H%M%S).snapshot

k3s with embedded etcd: use the built-in snapshot command, or the scripts/etcd-snapshot.sh wrapper which adds copy-to-backup-dir and retention pruning:

sudo k3s etcd-snapshot save
# or:
sudo ./scripts/etcd-snapshot.sh --backup-dir /var/lib/dcs/backups/etcd --retention-days 7

Note that a default k3s install uses kine/sqlite with no embedded etcd. See the backend-detection subsection in the DR Runbook.

Scheduling¶

Configure a cron job or scheduled task (on a workstation or bastion with talosctl access for Talos, or on the server node for k3s) to run daily etcd snapshots and copy them off the cluster.

Recovery Procedures¶

Restore CRDs from Backup¶

Pre-restore verification is mandatory (IEC 62443 SR/CR 7.3 RE(1), #973):

# 1. Validate the backup against its integrity manifest — refuses tampered,
#    truncated, or manifest-less files
dcs backup verify /backups/dcs-crds-2026-03-27.yaml

# 2. Restore all resources from the verified backup file
dcs restore crds -f /backups/dcs-crds-2026-03-27.yaml

If the backup file declares itself partial, the restore opens with a warning naming the kinds it cannot rebuild, before it applies anything. Those kinds were never in the file. A clean 0 failed summary says nothing about them.

The restore summary counts three outcomes:

Restore complete: 81 applied, 58 skipped, 0 failed

Skipped is the expected outcome for export-only kinds. A skip is not a partial restore. A backup captures everything the cluster held, because that is what an audit or a post-incident reconstruction needs. Restore re-applies only what a human or a GitOps repo authored. Two groups are skipped, each naming its reason on its own line:

Controller-derived — BatchRecord, ControlRecipe, RecipeRevision. Their controllers rebuild them from the parent object. Each also carries an ownerReference to a parent whose UID does not survive a restore, so a re-applied copy would be garbage-collected almost immediately.
Read-only records — material genealogy written during batch execution, and kinds whose CRDs are retained for existing objects and read paths without a write API.

Every authored kind restores. That includes CoordinationBlock, which the plant model declares and the gateway applies through the same endpoint as the rest.

Only the failed count means something went wrong, and it is the only one that makes dcs restore exit non-zero on a run that reaches the end.

A restore targets a cluster with nothing executing¶

The gateway refuses a document whose target is running (ADR 0060). Applying a Unit spec over a unit a batch holds would rebind the equipment the running phases resolve through. The apply endpoint answers 409 for that document and writes nothing.

The restore stops there. It prints the refusal with the gateway's reason and exits non-zero:

  OK: Site/main
  REFUSED: Unit/reactor-1 — Unit reactor-1 cannot be applied: unit reactor-1 is
  allocated to batch b-001. This document changes cmRoles, …

Restore STOPPED: 1 applied, 0 skipped, 0 failed

Stopping is deliberate. Skipping the refused document and carrying on would write the rest of a plant model over a live plant, which is a worse outcome than a run that halts. The documents already applied are that backup's own content, and apply is an upsert. Stop or abort what is running, then re-run the command to land the rest.

Disaster recovery restores into a rebuilt cluster, where every document is a create and nothing can be refused. A refusal means the target cluster is live, which is worth knowing before the rest of the backup lands on it.

A large restore can outrun the gateway's per-IP write budget. The CLI honours the server's Retry-After and backs off on its own when the header is absent. Throttling therefore shows up as a slower restore, with no resources shed.

An admin exports the plant's CRDs with dcs backup crds, which writes the export and its sidecar integrity manifest, and dcs backup verify reads it back OK with bytes, document count, and sha256; a copy of the export has exactly one byte flipped by dd on camera; dcs backup verify refuses the tampered copy with INTEGRITY FAILURE, both digests, do not restore from this file, and a non-zero exit pinned by echo; the pristine export then re-verifies green — The mandatory pre-restore check: one byte of a copied export flipped on camera, `dcs backup verify` refusing it with both digests and a non-zero exit, and the pristine export re-verifying green as the only file a restore may proceed from.

The restore command applies each document via the gateway's apply endpoint. The endpoint performs upserts (create if missing, update if exists). The restore is therefore safe to run against a live cluster.

Restore Audit Records¶

dcs backup verify /backups/dcs-audit-2026-03-27.yaml   # mandatory pre-restore check
dcs restore crds -f /backups/dcs-audit-2026-03-27.yaml

Note: If the AuditRecord immutability webhook is enabled, audit records that already exist in the cluster will be rejected (updates forbidden). This is by design for compliance. Only missing records will be created.

An admin picks a real AuditRecord and kubectl patches its spec: the validating webhook denies with the Part 11 clause in the message; delete is denied because there is no durable archive; all six archival attestation annotations are fabricated by hand and the write lands because attestation keys are allowlisted, but the delete is denied again with archival attestation rejected: signature-invalid; the archiver then runs with retention floored, verifies the attestation on every stamped record, reaches that same signature-invalid verdict on the forged one, re-archives it under a genuine signature, and dcs audit verify --archived re-proves the signed archive every record retired into — The immutability the restore note relies on, shown directly: an existing AuditRecord's spec edit and its deletion both denied by the webhook with the Part 11 clause in the message.

Restore Historian Database¶

Pre-restore verification (IEC 62443 SR/CR 7.3 RE(1)): backup-artifact integrity for the historian tier is delegated to CNPG/barman. The barman object-store catalogue records checksums per base backup and WAL segment, and the recovery bootstrap validates them as it replays (register D13 in the component assessment). Before initiating a restore, confirm the backup you are targeting is complete and catalogued:

# Backup CR must show completed phase and a non-empty backupId
kubectl get backups.postgresql.cnpg.io -n dcs-system
kubectl get clusters.postgresql.cnpg.io historian-db -n dcs-system \
  -o jsonpath='{.status.firstRecoverabilityPoint}{"\n"}{.status.lastSuccessfulBackup}{"\n"}'

An empty firstRecoverabilityPoint means there is no verified restore point. Do not proceed. Investigate the ScheduledBackup first. The periodic restore drill (below) is the end-to-end verification that the whole chain restores.

From CNPG backup (S3):

Create a recovery Cluster CR pointing to the backup:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: historian-db-restored
spec:
  instances: 1
  bootstrap:
    recovery:
      source: historian-db
  externalClusters:
    - name: historian-db
      barmanObjectStore:
        destinationPath: "s3://dcs-backups/historian"
        endpointURL: "http://minio:9000"
        s3Credentials:
          accessKeyId:
            name: backup-s3-creds
            key: ACCESS_KEY_ID
          secretAccessKey:
            name: backup-s3-creds
            key: ACCESS_SECRET_KEY

From pg_dump (Administrator):

Validate the dump first, then stream it back in. dcs backup verify checks the SHA-256 against the sidecar manifest and re-reads pg_dump's completion marker from the file. A dump that was cut short is therefore caught before it is restored over anything:

dcs backup verify /var/lib/dcs/backups/historian/dcs-historian-2026-08-15.sql

kubectl -n dcs-system exec -i <primary-pod> -c postgres -- \
  psql -U postgres -d historian -v ON_ERROR_STOP=1 \
  < /var/lib/dcs/backups/historian/dcs-historian-2026-08-15.sql

The dump is plain SQL, so the restore tool is psql and not pg_restore. It carries the TimescaleDB CREATE EXTENSION and the hypertable metadata, so the _hyper_*_chunk child tables are re-registered on the way in with no timescaledb_pre_restore() dance. Scale the historian collector to zero before the restore and back to one afterwards. The full procedure, including what to do about the CNPG cluster itself, is the fallback path in the DR runbook.

Full Cluster Recovery from etcd Snapshot¶

For catastrophic cluster loss, the restore path is platform-specific.

Talos (the reference deployments): follow the Talos v1.10 disaster-recovery guide. The version in that URL is the Talos release the reference deployments run, and the DR Runbook explains why the link is pinned. The procedure is:

Wipe the ephemeral partition on the affected control-plane node(s):

talosctl -n <control-plane-ip> reset --graceful=false --reboot \
  --system-labels-to-wipe=EPHEMERAL

Bootstrap from the snapshot once etcd reaches the Preparing state:

talosctl -n <control-plane-ip> bootstrap \
  --recover-from=/var/lib/dcs/backups/etcd/dcs-2026-03-27-030000.snapshot

k3s with embedded etcd:

Stop k3s:
```
sudo systemctl stop k3s
```

Restore etcd snapshot:

sudo k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=/var/lib/dcs/backups/etcd/dcs-backup-2026-03-27-0000

Start k3s:
```
sudo systemctl start k3s
```

On any platform, after the control plane is back:

Verify: Wait for all services to restart, then check the Diagnostics page in the gateway UI (System app).

Restore CRDs if needed (if the etcd snapshot is stale):

dcs restore crds -f /backups/dcs-crds-latest.yaml

The DR Runbook has the full drill-tested procedure including validation.

Validation¶

Verify Backup Integrity¶

# Cryptographic check against the sidecar manifest (#973) — run before
# every restore and periodically against stored backups
dcs backup verify /backups/dcs-crds-2026-03-27.yaml

# Content sanity: document count is also recorded in the manifest
cat /backups/dcs-crds-2026-03-27.yaml.manifest.json

dcs backup verify answers two separate questions, and both must pass:

Integrity — the file is byte-for-byte the one that was exported (SHA-256 against the sidecar manifest).
Completeness — the export was able to read every kind. A file can be perfectly intact and still be missing a kind that an RBAC gap hid from the gateway, which integrity alone will never detect. verify fails with PARTIAL BACKUP and names the kinds. --allow-partial accepts it deliberately.

A file that fails verification (or has no manifest) must not be restored. Restore-path verification is exercised end to end by dcs qualify dr (--fail-on-deviation) and recorded drills (see the Backup/Restore Drill).

IQ/OQ Test Protocol Outline¶

For qualification evidence (IQ/OQ):

IQ (Installation Qualification): Verify backup tools are installed and configured
dcs backup --help returns usage
The platform snapshot tool is present and authenticated (talosctl -n <node> etcd status on Talos, k3s etcd-snapshot --help on k3s)
Helm values show historian.backup.enabled: true (if applicable)
OQ (Operational Qualification): Verify backup/restore round-trip
Create a test recipe: dcs apply -f test-recipe.yaml
Back up: dcs backup crds -o test-backup.yaml
Delete the recipe: dcs delete masterrecipe test-recipe -s <site>
Verify the export against its manifest: dcs backup verify test-backup.yaml
Restore: dcs restore crds -f test-backup.yaml
Verify: dcs get masterrecipe test-recipe -s <site> returns the recipe
Document: record timestamps, file sizes, and resource counts for each step

A test master recipe is authored at the shell and applied with dcs apply, then listed beside the plant's real recipes; dcs backup crds writes the site export and its sidecar integrity manifest, whose JSON shows sha256, byte count, document count and export origin; dcs delete masterrecipe removes the test recipe after an interactive confirmation and the recipe list no longer shows it; dcs backup verify reads the export back OK against its manifest; dcs restore crds replays every document, printing OK per applied document and SKIP with a reason for export-only kinds, and finishes with zero failures; dcs get masterrecipe returns the deleted recipe — The OQ round-trip above, run end to end on a live plant: the test article applied, the site exported with its integrity manifest, the article deleted and proven gone, the export verified before any restore, replayed with zero failures, and read back by name.

21 CFR Part 11 Considerations¶

Audit trail backups (dcs backup audit) preserve the complete immutable record chain.
Archived audit records in PostgreSQL are protected by the historian database backup.
The archival CronJob annotates each record with audit.dcs.io/archived-to: historian before deletion from etcd, creating a traceable chain of custody.
Backup files should be stored on write-protected media or versioned object storage to prevent tampering.