Upgrade and Rollback

Status: Procedures below match the current Helm chart and the physical-operator's runtime-pod behaviour. The version compatibility matrix is a placeholder -- Cloud-Native DCS does not yet have stable release tags to map against, so that table lights up once v0.x releases exist.

Procedure for upgrading Cloud-Native DCS in place on an existing Kubernetes cluster and rolling back when an upgrade fails verification. Platform-specific steps (etcd snapshots) show both Talos (the reference deployments) and k3s forms.

Before You Start

• Read the release notes for the target version. Every release lists breaking changes, CRD schema migrations, and expected downtime.
• Confirm a current backup exists -- see Backup and Recovery. At minimum:
• etcd snapshot from the last hour: talosctl -n <node> etcd snapshot pre-upgrade.snapshot (Talos) or k3s etcd-snapshot save (k3s)
• CRD export: dcs backup crds -o /backups/pre-upgrade-$(date +%F).yaml
• HMAC signing key Secret: kubectl get secret dcs-signing-key -n kube-system -o yaml > /backups/dcs-signing-key.yaml
• Verify the current deployment is healthy. Upgrading on top of a degraded system multiplies the failure modes you have to debug:

  dcs health
  kubectl get pods -n dcs-system
  kubectl get certificate -n dcs-system -l app.kubernetes.io/part-of=cloud-native-dcs
  

• Schedule the window. Even rolling upgrades briefly interrupt per-unit runtime pods (see Downtime Expectations). Plan for a quiet period when no batch is in Running or Holding on a unit you intend to roll.

Architecture Facts That Matter for Upgrades

Two chart behaviours tend to surprise first-time upgraders:

1. Helm installs CRDs but never upgrades them. The chart ships the CRDs in its crds/ directory, so a fresh helm install creates them. Helm deliberately skips the crds/ directory on helm upgrade, and helm rollback will not undo a CRD change either. On every upgrade you must apply the new CRDs yourself with kubectl apply (or make install, from config/crd/bases/) before running helm upgrade.
2. Unit runtime pods are reconciled by the physical-operator alone. Helm never touches them. They run as bare Pods in site-<name> namespaces, and the operator's reconcile loop only recreates a runtime pod when the Pod is missing, has terminated, or has spec/volume drift. It does not compare image tags (podNeedsRecreate in internal/controller/physical/unit_pod.go). Picking up a new runtime image therefore means deleting the existing runtime pod so the operator recreates it with the current image tag. This is the single most important step to get right during an upgrade.

One-time migration: per-site runtime Certificate (v0.x, issue #213)

Upgrading from a chart version that predates #213 switches the CA issuer from namespace-scoped Issuer to cluster-scoped ClusterIssuer and moves per-site cloud-native-dcs-runtime-mtls Secret provisioning from a gateway-managed copy to a cert-manager Certificate owned by the Site reconciler. After helm upgrade:

# 1. Verify cert-manager is configured with --cluster-resource-namespace
#    pointing at the DCS release namespace (usually dcs-system). Without
#    this, cert-manager cannot find the CA Secret and ClusterIssuer stays
#    NotReady.
kubectl get clusterissuer cloud-native-dcs-mtls-ca

# 2. Delete manually-copied runtime Secrets in each site namespace so the
#    Site reconciler can recreate the Certificate → Secret chain cleanly:
for NS in $(kubectl get ns -l dcs.io/site -o name); do
  kubectl delete secret -n "${NS#namespace/}" cloud-native-dcs-runtime-mtls --ignore-not-found
done

# 3. Trigger a Site reconcile so the reconciler provisions a Certificate.
#    Any metadata change works — bump an annotation:
#      kubectl annotate site <name> \
#        dcs.io/reconcile-requested-at="$(date -u +%FT%TZ)" --overwrite
#    cert-manager then issues a fresh Secret within seconds.

# 4. Bounce runtime pods once so they mount the fresh cert content:
kubectl get pod -A -l dcs.io/component=unit-runtime -o name | \
  xargs -I {} kubectl delete {} -n <namespace>

Set mtls.certManager.clusterIssuer=false and mtls.certManager.perSiteRuntimeCert=false if your cert-manager install can't be reconfigured. The pre-#213 behaviour is preserved, but you keep the manual-copy drift risk.

Archive-integrity scheduler enabled by default (issue #216)

After upgrading to a chart that includes the Phase B scheduler, the gateway runs dcs audit verify --archived on gateway.archiveIntegrity.interval (default 6h) and writes one AuditRecord per run. Set gateway.archiveIntegrity.enabled: false or env GATEWAY_ARCHIVE_INTEGRITY_ENABLED=false to keep the old quiet behaviour on downgrade-friendly test clusters. No schema change is required. The scheduler reads the same audit_archive_manifest rows written by the archiver in Phase A.

Historian disk-watchdog removed (issue #243)

The chart-shipped historian-disk-watchdog CronJob (the interim coverage for #237/#238) has been removed. The two alerts it implemented are now opt-in PrometheusRule resources gated on monitoring.prometheusRule.historianDisk.enabled (default false). On upgrade:

• Any value set under historian.alerts.watchdog.* becomes a no-op. Helm silently ignores unknown keys, so the upgrade itself does not fail. The CronJob is removed from the cluster as part of the rollout.
• If you rely on the watchdog's webhook, install kube-prometheus-stack (or any Prometheus stack) and turn the new rules on:

  monitoring:
    prometheusRule:
      historianDisk:
        enabled: true
  

  The reference deployment's Flux overlay (cndcs-deploy-demo/flux/clusters/demo/) is the worked example.
• The Slack/Discord/PagerDuty webhook URL is now consumed by Alertmanager, not the CronJob. Move the demo-alert-webhook Secret from dcs-system into the monitoring namespace, or recreate it there.

Production deployments should keep historianDisk.enabled: false and rely on TimescaleDB native retention plus a CSI-backed PVC instead. See production-deployment.md § 1, § 4.

Optional S3 Object-Lock mirror (issue #216, Phase C)

The audit-archiver chart now accepts historian.audit.archival.immutable.* values. It is off by default. Existing deployments keep the current PostgreSQL-only archive behaviour until an operator explicitly points the archiver at a pre-provisioned Object-Lock bucket plus a Secret with S3 credentials. Enabling mid- cluster is safe. No migration is required, because PG remains the primary query path and the mirror only covers batches written after the flip. Bucket Object Lock cannot be added to an existing bucket retroactively, so new buckets must be created with --object-lock-enabled-for-bucket (AWS) or mc mb --with-lock (minio). See backup-recovery.md for the full checklist.

The mirror now needs its egress destination written down (issue #1514). A deployment that already runs the mirror with networkPolicies.enabled will find the upgrade refused at render time until historian.audit.archival.immutable.egress.destinationCIDRs (or destinationPodLabels, for a bucket served inside the cluster) is set. The refusal names the value and an example. It is not a new requirement so much as an old one that was never expressible: the archiver's NetworkPolicy has always declared policyTypes: [Egress] and has never carried a rule for the mirror endpoint, so on any cluster whose CNI enforces NetworkPolicy the mirror upload was already being denied. The PostgreSQL archive kept working, and only the 21 CFR Part 11 §11.10(c) restore copy went missing. Nothing caught it because every stack the mirror had run on used a CNI that ignores NetworkPolicy entirely.

Two things to check on the way through, and they are checked in different places. Confirm the archiver's recent runs succeeded (kubectl get jobs -n dcs-system over the CronJob's history), because a run that had a batch to archive would have failed on the denied upload and left those AuditRecord CRs in etcd, waiting for a run that can complete. Then list the bucket itself. A period during which the archiver was denied is a period with no immutable copy, and no later run rewrites it: a batch is mirrored once, by the run that archives it. dcs audit verify --archived will not show that hole, because it verifies the manifest chain in PostgreSQL and never reads the bucket.

The historian database backup now needs its egress destination written down too (issue #1516). A deployment that runs historian.backup.enabled with networkPolicies.enabled will find the upgrade refused at render time until historian.backup.s3.egress.destinationCIDRs (or destinationPodLabels, for a store served inside the cluster) is set. It is the same class of gap as the mirror above, in the policy one file over: the CNPG pods' policy declares policyTypes: [Ingress, Egress] and its backup rule named no destination at all, allowing every address in the world on TCP/443, while the port the chart's own endpointURL example uses is 9000. So a store on any port but 443 was already being denied. The port now comes off endpointURL, and the hard-coded 443 is gone.

Check the store for a recent base backup on the way through, because a denial here does not look like one. PostgreSQL does not drop a WAL segment it has not archived, so pg_wal grows on the data volume until the volume fills. That fill is what the 2026-04-12 demo incident was, one cause upstream.

Version Compatibility Matrix

Cloud-Native DCS is pre-1.0 and does not yet have release tags to map against. This table will light up as v0.x releases ship.

From	To	Forward-compatible	Rollback-safe	Notes
pre-release	pre-release	n/a	n/a	No stable release yet

When releases begin: - Forward-compatible means the new operators tolerate the old CRD schemas for one minor version, so a rolling upgrade is safe. - Rollback-safe means the old operators can still read any CR that was written by the new version. A rollback-unsafe upgrade typically requires an etcd snapshot restore. A helm rollback cannot do it.

Kubernetes Version Upgrades

Upgrading the cluster underneath Cloud-Native DCS is a separate exercise from upgrading the chart, and every constraint on it comes from upstream. Cloud-Native DCS builds against the Kubernetes 1.36 API libraries and runs its envtest suites on the 1.36 control-plane binaries. The reference deployments run Talos Linux v1.10.9 with Kubernetes v1.33.6, and the documented floor for a customer cluster is Kubernetes 1.27.

The wave of upstream removals landing between Kubernetes 1.35 and 1.38 is about node prerequisites. None of it lands on the DCS control plane. Read the table before scheduling a cluster upgrade. Two of these rows fail the kubelet on a node, which is a worse outcome than failing a workload.

Upstream change	Enforced from	What a DCS cluster has to do
containerd 1.x support ends	1.36	Every node image must carry containerd 2.0 or later. Kubernetes 1.35 was the last release to support containerd 1.x, and from 1.38 an old containerd fails outright against the newer kubelet. Scrape kubelet_cri_losing_support before the upgrade to find nodes that are still behind.
cgroup v1 support phased out	1.35	Every node must run cgroup v2. failCgroupV1 has defaulted to true since 1.35, so the kubelet refuses to initialize on a cgroup v1 node. The failCgroupV1: false override still exists in 1.37, and it is only a stopgap.
Static Pods can no longer reference Secrets or ConfigMaps	1.37	Nothing on the DCS side. Static Pods below covers why, and names the one place the product touches the concept.
kube-proxy ipvs mode deprecated	1.37 logs a warning	Confirm the mode with kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}'. The mode is expected to be off by default in 1.40 and removed in 1.43, and the recommended Linux mode is now nftables. Talos and k3s both default to iptables, so a cluster nobody switched by hand is unaffected.
SELinuxMount graduates to GA	1.37	Only clusters with SELinux enabled see any effect, and only for CSI drivers that set seLinuxMount: true. Two pods with different SELinux labels sharing one volume can now fail to start where recursive relabeling used to let them coexist. Set seLinuxChangePolicy: Recursive on such a pod to keep the old behaviour.
metrics.k8s.io graduates to v1	1.37	Nothing. The gateway resolves the group version from discovery and prefers v1, falling back to v1beta1 on a cluster that serves only the beta (#1376). A cluster that later retires v1beta1 is followed, with no false report of a missing metrics stack.

Kubernetes v1.37 is planned for 26 August 2026. The v1.37 sneak peek is the source for every 1.37 row above.

The Talos version gates the Kubernetes version

On the reference deployments the Kubernetes version is not chosen independently. Talos v1.10 supports Kubernetes 1.28 through 1.33, so a cluster on Talos v1.10.9 cannot reach 1.37 at all until Talos itself is upgraded. Sequence the Talos bump first and the Kubernetes bump second.

That ordering also disposes of the first two rows of the table on Talos. Talos v1.10 already ships containerd 2.0.5, and the same release dropped cgroup v1 outside container mode. A Talos node current enough to run 1.37 satisfies both prerequisites by construction.

A Talos bump also moves two documentation links. The upstream disaster-recovery guide is linked at a version-pinned URL from the DR Runbook and from Backup and Recovery, because the Siderolabs documentation site serves no unversioned path for that page. Both links state the version they carry, and both name the release the reference deployments run. Neither one tracks the current Talos release. Move them with the bump, and confirm the commands quoted under each still match the new version's procedure.

Static Pods are not how DCS runs anything

The 1.37 kubelet prohibits a static Pod from referencing a Secret or a ConfigMap through fields like configMapRef or secretRef, and it removes the PreventStaticPodAPIReferences gate that used to let an operator opt out. Cloud-Native DCS ships no static Pods and writes nothing into /etc/kubernetes/manifests on any node. Unit runtime pods look node-pinned because they use hostNetwork and a hostPath state volume, but the physical-operator creates them through the API server like any other Pod (internal/controller/physical/unit_pod.go).

The product touches the static-Pod concept in exactly one place. The controller-removal drain skips pods carrying the kubernetes.io/config.mirror annotation, because a mirror pod belongs to the kubelet and cannot be evicted (internal/controller/physical/node_drain.go). Mirror-pod semantics are unchanged in 1.37, so that path needs no work.

Where the rule can still bite a plant is on a hand-rolled edge node. If a site runs a lightweight component as a static Pod alongside the DCS workload (a log shipper, a vendor agent, a bootstrap helper), read that manifest before the node crosses 1.37. Any configMapRef or secretRef content has to move inline into the manifest or onto the node's disk.

Pre-flight Checks

Run this checklist before touching the cluster. Anything that fails is a stop-ship.

# 1. Cluster and chart versions.
kubectl version --short
helm list -n dcs-system

# 2. All operators healthy and at the desired replica count.
kubectl get deploy -n dcs-system
kubectl get pods -n dcs-system \
  -l app.kubernetes.io/part-of=cloud-native-dcs

# 3. No batches in Running or Holding on a unit you plan to roll.
#    dcs is site-scoped — repeat per site (list sites with `dcs get sites`).
dcs get batches -s <site>
dcs get units -s <site>

# 4. Historian WAL replication is current (if historian is enabled).
kubectl cnpg status <historian-cluster-name>

# 5. Fresh etcd snapshot, younger than 1 hour.
talosctl -n <control-plane-ip> etcd snapshot pre-upgrade-$(date +%s).snapshot   # Talos
sudo k3s etcd-snapshot save pre-upgrade-$(date +%s)                             # k3s

# 6. Certificates are all Ready.
kubectl get certificate -n dcs-system \
  -l app.kubernetes.io/part-of=cloud-native-dcs

If the historian is configured with historian.backup.enabled: true, also confirm the most recent S3 backup is within its RPO window with kubectl get backup -n dcs-system.

If the window also moves the cluster to a new Kubernetes minor version, clear Kubernetes Version Upgrades before anything below. A node that fails its own prerequisites never gets as far as the chart.

If the deployment defines action-level authorization policies (gateway.auth.roles object form: allow/deny lists, ADR 0024), review the release notes' new-action list against your deny lists before upgrading: a new release can add actions to the catalog, and a deny list written against a family does not automatically cover a newly added sibling action. After the upgrade, dcs auth policy shows what the running gateway enforces.

A release can also move an existing action to a different tier, which changes who holds it without changing any role definition. The OPC UA discovery family (discovery:browse) moved from read to engineer in ADR 0062. Tooling that drove those routes under a credential below engineer has to move up, or be granted the action through an ADR 0024 allow entry on the role it holds. The same release added dcs-viewer to the shipped table, which is read alone and is the grant for an identity that only observes. A deployment supplying its own roles file does not receive it, since a non-empty gateway.auth.roles replaces the shipped table outright.

7. Dry-run the chart upgrade to catch immutable-field changes

The commands below use a local chart checkout. Upgrading against the published chart instead (oci://ghcr.io/cloud-native-dcs/charts/cloud-native-dcs) pulls from a private registry, so helm registry login ghcr.io has to have been run on the machine driving the upgrade. Credentials cached from an earlier release can be expired without any sign until the pull fails.

helm upgrade dcs deploy/helm/cloud-native-dcs \
  --namespace dcs-system --dry-run --reuse-values \
  --set global.image.tag=<target-version-tag>

The chart ships a render-time guard (templates/historian-database-immutability-check.yaml) that uses Helm's lookup function to compare the running CNPG Cluster's spec.postgresUID and spec.postgresGID against the values about to be applied. Both fields are immutable on an existing Cluster. Mismatching them otherwise trips the validating webhook, and Flux loops on RollbackFailed indefinitely. That failure mode was observed live during a 2026-04-22 demo-instance incident, which motivated this pre-flight check.

If --dry-run fails with an immutability error, the message includes the existing values and remediation. Honour it before retrying the live upgrade. The check is silent when nothing immutable changed.

The same dry-run also exercises the CNPG minimum-version guard (templates/historian-database-cnpg-version-check.yaml), which inspects the installed clusters.postgresql.cnpg.io CRD for .spec.podSecurityContext, a field the historian Cluster sets that only exists from CloudNativePG chart 0.27.0 (operator 1.28) onward. This matters most when upgrading a cluster whose CNPG was installed long ago against a loose version range: the DCS chart upgrade would otherwise fail mid-apply with .spec.podSecurityContext: field not declared in schema. Upgrade CNPG to >=0.27.0 first, then retry.

8. Validate PromQL changes against promtool

If the chart upgrade changes any rule under templates/prometheusrule.yaml, validate the rendered PromQL locally before the prometheus-operator's mutating admission webhook does. A syntax error there fails the entire Helm upgrade with Rules are not valid, and Flux gets stuck in Running 'upgrade' action until you manually unlock the release with flux suspend + helm rollback + flux resume.

helm template <release> deploy/helm/cloud-native-dcs/ \
  --set monitoring.prometheusRule.historianDisk.enabled=true \
  | python3 -c '
import sys, yaml
for d in yaml.safe_load_all(sys.stdin):
    if d and d.get("kind") == "PrometheusRule":
        print(yaml.safe_dump({"groups": d["spec"]["groups"]}))' \
  | docker run -i --entrypoint promtool \
      quay.io/prometheus/prometheus:v2.54.1 check rules /dev/stdin

helm lint and the chart's existing template tests do not catch PromQL syntax errors. Those are only surfaced by the in-cluster admission webhook. Add this check to your CI pipeline for any change under prometheusrule.yaml. See production-deployment.md § 4 "Validating PrometheusRule changes before deploy" for the recurring PromQL pitfalls (notably group_left() and the Helm-vs-Prometheus {{ }} overlap).

Upgrade Procedure

This sequence is the standard "rolling upgrade" path. Every step must complete successfully before moving to the next.

1. Drain or pause active batches

Rolling the operators and gateway doesn't interrupt running batches. State lives on CRDs, outside operator memory. Rolling a unit runtime pod does interrupt whatever is running on that unit. Two options:

• Quiet window: wait for all batches to Complete, or Hold and Stop them via the HMI before starting.
• Live upgrade: let the operators reconcile batches into Holding when their unit runtime goes NotReady. This is the default behaviour. The grace window before Hold is runtimeCrashGracePeriod in unit_controller.go. Expect each unit to lose ~30s of execution time.

Record the decision in the upgrade ticket before starting.

2. Apply the new CRDs

git fetch --tags origin
git checkout <target-version-tag>
make install        # runs kubectl apply -f config/crd/bases/
# or, directly:
kubectl apply -f config/crd/bases/

Watch for kubectl errors about incompatible schema changes -- those indicate a breaking CRD migration that the release notes should have flagged. If they didn't, stop and escalate.

3. Upgrade the chart

helm upgrade dcs deploy/helm/cloud-native-dcs \
  --namespace dcs-system \
  --reuse-values \
  --set global.image.tag=<target-version-tag> \
  --wait --timeout 10m

--wait blocks until every rollout reaches Ready. --reuse-values preserves any site-specific overrides you applied at install time. Pair it with --set for the small handful of values that need to change.

If you pin components by digest (Security Hardening § Supply Chain Verification), remember that a digest beats any tag: --reuse-values carries the old <component>.image.digest forward and that component will not move to <target-version-tag>. Update each pinned digest to the new release's digest in the same helm upgrade.

The MQTT broker is the one image that is not a DCS component. mqtt.image.tag names a Mosquitto release, and no longer the "2" series. An upgrade therefore moves it only when the chart's own default moves. That pin exists because the series tag changed the broker underneath a release nobody cut. A passwd file that made one Mosquitto accept the wrong credential made the next one terminate on load (#1579).

4. Verify each control-plane component rolled cleanly

kubectl rollout status deploy/dcs-cloud-native-dcs-physical-operator -n dcs-system
kubectl rollout status deploy/dcs-cloud-native-dcs-procedural-operator -n dcs-system
kubectl rollout status deploy/dcs-cloud-native-dcs-batch-operator -n dcs-system
kubectl rollout status deploy/dcs-cloud-native-dcs-control-operator -n dcs-system
kubectl rollout status deploy/dcs-cloud-native-dcs-gateway -n dcs-system
kubectl rollout status deploy/dcs-cloud-native-dcs-historian -n dcs-system  # if enabled
kubectl rollout status deploy/dcs-cloud-native-dcs-mqtt -n dcs-system       # or sts in HA

Deployment names are <release>-cloud-native-dcs-<component> for a release whose name does not already contain the chart name. The commands on this page assume the release is named dcs, matching the helm upgrade dcs invocation above. The reference GitOps deployment pins releaseName: cloud-native-dcs, which collapses the prefix to cloud-native-dcs-<component>. kubectl get deploy -n dcs-system shows the rendered names for your install.

Operators use leader election, so the old leader steps down as soon as its replacement is Ready. The gateway is stateless.

5. Roll the unit runtime pods

This is the non-obvious step. Because the physical-operator does not compare runtime image tags, the runtime pods are still running the previous image until you delete them.

# Delete every runtime pod, one site at a time. The physical-operator
# reconciles each unit and recreates the pod with the new runtime image
# from the Helm values (unitRuntime.image.tag).
for ns in $(kubectl get ns -l dcs.io/site -o name | sed 's|namespace/||'); do
  echo "== Rolling runtimes in $ns =="
  kubectl delete pod -n "$ns" -l dcs.io/component=unit-runtime
  # Wait for the operator to recreate and for all runtimes to go Ready.
  kubectl wait pod -n "$ns" -l dcs.io/component=unit-runtime \
    --for=condition=Ready --timeout=120s
done

If you want zero interruption per unit, roll the runtime pods one at a time: delete one pod, wait for Ready, move to the next. The physical-operator's grace window absorbs the brief NotReady gap without converting the batch to Hold.

Step 5, and why it exists: the chart upgrade lands and every control-plane check is green while the runtime pods keep running the previous image, until they are deleted. Both tags are the same build under two names on the capture stack. The filmed change is the reconciler ignoring the tag. No version moved.

6. Verify the upgrade end to end

See Verification After Upgrade. Do not declare success until every item on that checklist is green.

Rollback Procedure

Rollback is only straightforward when the new version introduced no CRD schema changes. Check the release notes for Rollback-safe: yes before choosing this path.

Rollback path A -- rollback-safe upgrade

# 1. Helm rollback to the previous revision.
helm history dcs -n dcs-system
helm rollback dcs <previous-revision> -n dcs-system --wait --timeout 10m

# 2. Roll the runtime pods again -- same step as the upgrade, because the
#    operator still does not compare image tags.
for ns in $(kubectl get ns -l dcs.io/site -o name | sed 's|namespace/||'); do
  kubectl delete pod -n "$ns" -l dcs.io/component=unit-runtime
  kubectl wait pod -n "$ns" -l dcs.io/component=unit-runtime \
    --for=condition=Ready --timeout=120s
done

# 3. Run the verification checklist.

helm rollback does not revert CRD changes applied with kubectl apply in step 2 of the upgrade. For rollback-safe upgrades this is fine. The older operators ignore new optional fields. For rollback-unsafe upgrades, use Path B.

Rollback path B -- CRD schema change or data-format migration

Once a new version has written CRs in a schema the old operators cannot read, helm rollback is not enough. You must restore etcd to the pre-upgrade snapshot:

1. Stop the DCS workloads to prevent split-brain writes:

   kubectl scale deploy -n dcs-system \
     dcs-cloud-native-dcs-gateway dcs-cloud-native-dcs-physical-operator \
     dcs-cloud-native-dcs-procedural-operator dcs-cloud-native-dcs-batch-operator \
     dcs-cloud-native-dcs-control-operator --replicas=0
   

2. Follow the etcd restore procedure in the DR Runbook for your platform: talosctl bootstrap --recover-from=pre-upgrade-<ts>.snapshot on Talos, or k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/pre-upgrade-<ts> on k3s.
3. Re-apply the old CRDs from the version tag you are rolling back to:

   git checkout <previous-version-tag>
   kubectl apply -f config/crd/bases/
   

4. helm rollback dcs <previous-revision> -n dcs-system.
5. Run the runtime-pod roll from step 5 of the upgrade procedure.
6. Run the verification checklist.

Data-format migrations (historian schema bumps, audit archive format changes) are one-way. If the upgrade ran the migration, a rollback loses any records written after the migration completed. The release notes for each release must call out which steps are one-way.

Downtime Expectations

Component	Interruption during rolling upgrade
Gateway	~5-10s per replica as each pod goes NotReady and the service endpoints update. Active WebSocket subscriptions reconnect automatically. With gateway.replicas >= 2 and a PDB, user impact is effectively zero.
Operators	None visible. State lives on CRDs; the new leader resumes reconciling where the old one left off.
MQTT broker	~5s reconnect on the clients. Telemetry queued in the runtime's store-and-forward buffer (queue.jsonl) replays automatically.
Historian	None if CNPG-managed (no schema change). Historian pod restart briefly pauses ingest; MQTT QoS 1 redelivery backfills.
Unit runtime pods	~10-30s per pod from delete to new pod Ready. During this window the unit is NotReady; if a batch is Running on the unit, the grace window absorbs the gap. If the gap exceeds the grace window, the batch transitions to Hold and can be resumed.
Webhook (audit immutability)	~5s. During the restart window, AuditRecord create calls that race the webhook may be rejected; the gateway retries.
OMF egress	Off by default, so most upgrades skip it. Where it is on, the pod is Recreate with one replica, so the gap is the pod restart. Its buffer is in memory: whatever it is holding for an unreachable endpoint at that moment is lost, and dcs_omf_egress_dropped_total records it. Upgrade while the endpoint is up. An outage it is riding out is the wrong window.

One-time restart on the upgrade that adds the OMF egress. The broker's auth Secret gains a fifth account, dcs-omf-egress, whether or not the component is enabled. The gateway, the four operators, and the historian each carry a checksum annotation over that Secret. Each therefore rolls exactly once, on that upgrade. It reads as an unexplained restart of components the release notes did not mention, and this is the reason.

Two things do not restart with them. The broker keeps running: it rehashes and SIGHUPs through its passwd-reloader sidecar, the same mechanism that makes password rotation graceful. Unit-runtime pods are recreated on a change to the runtime mTLS Secret and nothing else, and their own password did not change.

Verification After Upgrade

Every item must be green before closing the upgrade ticket.

• [ ] dcs health returns all components healthy.
• [ ] helm status dcs -n dcs-system shows the new chart version.
• [ ] kubectl get deploy -n dcs-system shows the expected replica count for every DCS deployment.
• [ ] Gateway is reachable and authenticates an OIDC user end to end.
• [ ] At least one runtime pod per site was recreated with the new image tag: kubectl get pod -n site-<name> -l dcs.io/component=unit-runtime -o jsonpath='{.items[*].spec.containers[*].image}'.
• [ ] A canary batch runs from Create → Running → Complete against a simulated unit.
• [ ] AuditRecords for the upgrade window are present and their e-signatures verify. See 21 CFR Part 11 traceability.
• [ ] No x509, TLS handshake, or 401/403 spikes in gateway or operator logs in the 15 minutes after the rollout.
• [ ] cert-manager is still healthy and all Certificate resources are Ready.
• [ ] kubectl get prometheusrule -n dcs-system shows no firing alerts attributable to DCS.

Related Documentation

• Deploy Your Own -- initial install procedure
• Backup and Recovery -- take backups before upgrading
• DR Runbook -- recovery path when an upgrade corrupts state beyond what a rollback can fix
• High Availability -- component failure modes during rolling upgrade
• Rotation Runbook -- credential rotation is a frequent trigger for upgrade-time surprises
• Historian Disk-Pressure Runbook -- upgrades that add new hypertables or change TimescaleDB versions can shift disk usage. Enable historian.prune on non-prod clusters to bound growth. Upgrades that touch historian.prune.priorityClassName or historian.prune.tolerateDiskPressure re-render the prune Pod template, which the next CronJob run picks up with no in-flight Pod restarts needed (issue #256).