Skip to content

Backup/Restore Drill — 2026-07-18

Recorded execution of the CRD-tier backup → pre-restore verification → restore chain (IEC 62443 SR/CR 7.3 incl. RE(1) backup verification, #973), mirroring the format of the archive corruption drill.

Environment

Disposable kind cluster (drill-62443, single node), DCS CRDs installed (config/crd/bases/), no operators (the drill exercises the backup/restore chain, not reconciliation). Gateway run as a local process (--auth-mode none, port 61980) against a dedicated kubeconfig. Seeded state: 2 Enterprises, 2 Sites, 1 Area, 1 ProcessCell, 1 Unit across two site namespaces — including one Enterprise carrying a semantic user label (dcs.io/tenant: eastfield).

Executed steps and observed results

  1. Backup: dcs backup crds -o drill-final.yaml → 2 971 bytes, 7 documents, sidecar manifest written (drill-final.yaml.manifest.json).
  2. Pre-restore verification (clean): dcs backup verify drill-final.yaml → OK … sha256 92f95fcd5227d21a….
  3. Tamper detection: a single byte substituted in an earlier export (byte count unchanged, so only the hash can catch it) → dcs backup verify exited non-zero: INTEGRITY FAILURE … do not restore from this file. A backup with no manifest is likewise refused (must not be restored).
  4. Simulated loss: kubectl delete unit granulator-1 -n site-newark-plant → resource gone.
  5. Restore from the verified backup: dcs restore crds -f drill-final.yaml → Restore complete: 7 applied, 0 failed.
  6. Confirmation: the deleted Unit exists again (unit.physical.dcs.io/granulator-1) and the user label survived the round-trip (dcs.io/tenant: eastfield present on the restored Enterprise).

Defect found and fixed during the drill

The first restore pass failed on the labeled resource: the gateway apply envelope rejected metadata.labels as an unknown field (HTTP 400), so any resource carrying a user label could not be restored from its own backup. Fixed in the same change (#973): the apply envelope accepts user labels/annotations and re-applies them (system-prefixed keys dropped), with a regression test pinning the round-trip. This is precisely the failure class the SR 7.3 RE(1) drill exists to catch — the backup was valid; the restore path was not.

Scope and deferred tiers

  • CRD/system state tier: exercised end to end above.
  • Audit export tier: same mechanism (dcs backup audit writes the same manifest; dcs backup verify is tier-agnostic); the archive tier's own integrity chain was drilled separately (archive corruption drill).
  • Historian tier (CNPG/barman): backup-artifact integrity is delegated to the barman catalogue (register D13 in the component assessment); a full CNPG point-in-time restore drill requires an S3-backed deployment and is the next scheduled drill on the reference cluster (tracked with the periodic-drill items in the SL-3 checklist).

Secrets exclusion check

The exported backup was inspected: custom resources only — no Secrets, no key material (62443-4-2 11.5.2). Recovery re-provisions secrets per the rotation runbook.