Production Deployment

deploy-your-own.md gets a single-cluster install running. This guide is the gap between that and a production deployment a customer should run: what to change, what to add, and why each item matters. Every recommendation here came out of a real failure mode on a single-node reference cluster that production must not repeat.

If you are evaluating Cloud-Native DCS for a regulated pharmaceutical site, read this end-to-end before sizing the cluster. A single-node install is intentionally not configured this way. A recurring DiskPressure cascade on our own single-node demo instance (April 2026) is the stress test that produced this checklist.

Registry credentials are a production dependency

The chart and every component image come from a private GHCR namespace, so the cluster holds a pull secret. That secret is on the critical path for every pod restart, node replacement and scale-out. Two things follow. The token needs an owner and a rotation schedule alongside the other credentials in Rotation Runbook, because a silently expired one stays quiet until the next restart, where it surfaces as ImagePullBackOff. And a site that cannot reach ghcr.io from every node needs a pull-through registry, which is the same mitigation as the pull rate limit in § 9.

Deploy Your Own Instance covers obtaining access and creating the secret.

1. Storage

Use a CSI driver with quota enforcement

The single largest source of operational pain on a single-node reference cluster is local-path-provisioner. It accepts PersistentVolumeClaim.spec.resources.requests.storage as advisory metadata: a workload can write past its claim until the host filesystem is full, at which point kubelet trips node.kubernetes.io/disk-pressure:NoSchedule and evicts every BestEffort and Burstable pod on the node.

Production deployments must use a CSI driver that enforces capacity at the filesystem layer. Verified-compatible options:

Driver	Quota enforcement	Notes
AWS EBS CSI	Yes (block volume size)	Recommended on EKS
GCP PD CSI	Yes (block volume size)	Recommended on GKE
Azure Disk CSI	Yes (block volume size)	Recommended on AKS
Ceph RBD via Rook	Yes (RBD image size)	On-prem default
Longhorn	Yes (volume size)	Lightweight on-prem
TopoLVM	Yes (LVM logical-volume size)	Bare-metal LVM
local-path	No (advisory only)	Dev/demo only — see #240
hostPath	No	Never use in production

The historian PVC must never share a filesystem with the OS, container image store, or kubelet logs. A separate block volume per stateful component (one for historian-db, one for mqtt-data, one for the audit-archiver mirror) isolates blast radius. A runaway historian cannot evict ingress-nginx if its disk fill never reaches /.

Per-component PV sizing

Component	Default PVC	Production minimum	Sizing reference
historian-db (CNPG)	10 Gi	See Capacity Planning § Historian	Function of tag count × sample rate × retention
mqtt-data (mosquitto)	1 Gi	1 Gi	Persistent sessions only; minimal growth
Audit S3 mirror (optional)	n/a	Bucket sized per § Backup & DR	Object Lock retention drives bucket cost

Set historian.database.cnpg.storage.storageClass explicitly. Do not rely on the cluster default. It may be local-path on dev clusters and the production CSI on prod, with no failure if you forget.

2. Cluster topology

Minimum: three nodes for control-plane workloads

A single-node cluster is the worst structural decision you can make. When the single node hits DiskPressure every control-plane component goes down together: ingress, cert-manager, CNPG operator, every DCS operator. There is no redundancy and no fallback path.

The CNPG and cert-manager validating webhooks are single points of contagion in single-node deployments. When the operator pod is evicted, the webhook service has zero endpoints, and any Kubernetes apply that goes through that webhook stalls. Flux loops on RollbackFailed until the operator is back (observed live during a 2026-04-22 demo-instance incident).

Topology	Use case	Failure mode
1 node (Talos on a droplet)	Demo / proof-of-concept only	Webhook contagion, DiskPressure cascades
3 control-plane + N workers	Recommended production minimum	Survives single-node failure of control-plane or worker
3 control-plane + 3 storage + N workers	Stateful workloads (CNPG, MQTT) on dedicated storage nodes	Storage churn isolated from compute churn

For the unit runtime side, see reference-architectures.md. A pharmaceutical plant typically pairs a 3-node Kubernetes control plane with one or more edge devices per process cell.

Taint device nodes unit-runtime-only (zone-model placement)

Device nodes run Level-1/2 control. The historian, its database, MQTT, the gateway, and the operators are Level-3 operations services. Hosting either group on the other's compute collapses the IEC 62443 zone boundary at the host layer and puts historian compaction and query load in contention with deterministic control scans. The scheduler will happily do this to you: with no guardrail it once placed a CNPG database and the MQTT broker on a device node, and node-local storage made the placement permanent (ADR 0031, issue #1167).

The posture has two halves:

1. Taint every device node at join time:

kubectl taint node <device-node> dcs.io/role=device-node:NoSchedule

The product's unit-runtime and io-probe pods tolerate this taint out of the box. Every other pod is repelled, which is the point: platform services excluded by default, with no per-service values to forget.

Apply the taint with kubectl, using an administrative credential. Some distributions offer a declarative node-taint field in their machine or node configuration, and that field cannot always apply a taint to a node that has already joined, because the NodeRestriction admission plugin lets a kubelet set taints only at registration. Where it fails, it fails silently from the Kubernetes side: the distribution's own resources report the taint while kubectl get node shows none. Talos machine.nodeTaints behaves this way. Verify on the Node object. The configuration you supplied is the side that lies here.

Re-apply the taint whenever a device node re-joins. A taint lives on the Node object. It survives reboots, upgrades, and cold boots. It is lost when the Node object is deleted and recreated, which is what a node rebuild or a re-enrolment does. An untainted device node looks perfectly healthy until a platform service lands on it.

Give node-scoped DaemonSets an explicit toleration. A DaemonSet is not exempt from taints. The scheduler places DaemonSet pods normally, so an untolerated taint keeps them off a device node exactly as it would a Deployment. CNI and kube-proxy keep working because they carry a blanket operator: Exists. That toleration is a property of those charts, and DaemonSets as a kind carry no such exemption. Anything that must observe or service every node (node-exporter, log shippers, storage and security agents) needs the toleration spelled out:

tolerations:
  - key: dcs.io/role
    operator: Equal
    value: device-node

The empty effect is deliberate. It matches every effect, so hardening the taint to NoExecute later does not evict the agent.

Missing this is quiet and expensive. The series that disappear are the ones describing controller compute, so a later failover drill or soak run reports clean numbers while the instrument is blind on the subject under test.

Verify the taint by what it repels. Reading configuration back proves nothing here:

kubectl run taint-probe --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"dcs.io/role":"device-node"}}}' \
  --command -- sleep 5
kubectl get pod taint-probe \
  -o jsonpath='{.status.phase} {.status.conditions[?(@.type=="PodScheduled")].reason}'
# Pending Unschedulable
kubectl delete pod taint-probe

Then confirm the agents you expect are still running there: kubectl get pods -A --field-selector spec.nodeName=<device-node>.

1. Pin platform services to platform nodes. Every chart-rendered platform pod exposes nodeSelector and tolerations values (historian.* also covers its prune and audit-archival CronJobs), and the CNPG database takes historian.database.cnpg.affinity, a passthrough to CNPG's AffinityConfiguration. On a cluster whose platform tier is the (untainted-for-you) control plane:

historian:
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  database:
    cnpg:
      affinity:
        nodeSelector:
          node-role.kubernetes.io/control-plane: ""
        tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
            effect: NoSchedule

The same two keys exist on mqtt, gateway, docs, physicalOperator, proceduralOperator, batchOperator, controlOperator, and gitopsEnforcement.notification.flux.

The database placement matters most. local-path-class storage binds the PV to whichever node the pod first lands on, so a wrong first scheduling decision persists through every restart until you delete the PVC. Steer it before first install.

Provision standby edge nodes for unit-runtime failover

If you use edge-runtime failover (ADR 0006), the standby nodes are deployment-owned infrastructure. The product names them but does not provision them. For each Unit (or pool) that needs failover:

1. Enroll the standby like any controller node. It must carry the dcs.io/site=<site> label (the ADR 0004 node-adoption contract) and be Ready. An un-enrolled or virtual node is never an eligible target.
2. Give it field-network reach, bonded. The standby needs an interface on the same field VLAN(s) as the primary it backs. The runtime reconnects to the remote I/O over the network, so this is the only hard adjacency requirement. Build that interface as an active-backup bond across two ports on two switches, the same posture as the primary (ADR 0026). A standby that is single-homed inherits the failure mode failover exists to escape. Verify reachability to every IOModule endpoint before relying on it.
3. Label it for selector-based target sets (optional). A shared spare pool is cleanest to reference via availability.failoverTargets.nodeSelector (e.g. a dcs.io/standby-pool: <cell> label). The alternative, an explicit node list, must be edited per Unit.
4. Size for the busiest Unit it may host. Match unitRuntime.resources headroom and CPU/RAM to the heaviest FB network in its target set. A shared spare must also carry every field VLAN in the pool.
5. Name it in the Unit spec. Set spec.availability.failoverTargets (explicit nodes and/or nodeSelector) and spec.availability.mode (Failover for automatic re-bind, Autonomy for manual fenced failover). A dedicated 1:1 standby is the recommended pattern. See reference-architectures.md Pattern B for the topology trade-offs.

The standby stays cold until a failover re-binds a runtime onto it (no hot-standby state replication). Pre-pulling the unit-runtime image onto the standby shortens failover time and is a safe optimization.

Pin admission-webhook backers to multiple nodes

Set replicas: 2 (or more) on:

• cnpg-cloudnative-pg (the CNPG operator)
• cert-manager (controller, webhook, cainjector)
• ingress-nginx-controller

Combine with a topologySpreadConstraint on kubernetes.io/hostname so replicas land on distinct nodes. A single replica on a single node defeats the topology recommendation above.

3. Capacity planning

The full sizing math lives in capacity-planning.md. This section is a quick lookup for the historian-disk failure mode that hit the reference cluster three times in ten days.

Historian retention vs disk at common ingest rates

The TimescaleDB hypertable footprint, for (time, tag_id, value) rows on the historian schema, is approximately 31 bytes/sample raw (~12 bytes post-compression after a chunk is closed). Compression kicks in only on chunks fully outside the active retention window, so plan against the uncompressed number for the active window plus headroom.

Per-day ingest volume:

GB/day = tags × samples_per_second × 86400 × 31 / 1_000_000_000

Tags	Sample rate	GB/day	30-day retention	90-day retention	365-day retention
100	1 Hz	0.27	8 GB	24 GB	100 GB
500	1 Hz	1.34	40 GB	121 GB	489 GB
1 000	1 Hz	2.68	80 GB	241 GB	978 GB
5 000	1 Hz	13.4	402 GB	1.2 TB	4.9 TB
5 000	0.1 Hz (avg)	1.34	40 GB	121 GB	489 GB
10 000	0.1 Hz (avg)	2.68	80 GB	241 GB	978 GB

Add 30 % headroom for CNPG WAL files, the audit hypertable, and chunk-time interval rounding (a chunk older than retention is kept until the whole chunk is outside the retention window).

The reference cluster runs at ~1.3 GB/day of tag ingest (5000 simulated tags at 0.1 Hz average across three units, the table row above, plus alarm churn on top) on a 5 Gi PVC with 6-hour retention enforced by a chart-shipped prune CronJob (see issue #240). Production deployments should size the PVC for the full retention window and rely on TimescaleDB's native retention policy. The prune CronJob is a single-node safety net.

The prune CronJob template sets priorityClassName: system-cluster-critical and tolerates node.kubernetes.io/disk-pressure:NoSchedule. Production deployments that don't enable the prune Job inherit no risk from these defaults. The toleration only applies to the prune Pod itself, which is not created at all when historian.prune.enabled: false. Production deployments that do enable it (rare, generally only on bench-pilot clusters) get a Job that can schedule and survive eviction during a DiskPressure cascade, the failure mode that caused issue #256 on the reference cluster. Override historian.prune.priorityClassName: "" if your cluster's PriorityClass admission policy restricts system-cluster-critical to kube-system.

AuditRecord retention vs etcd / kine

Same sizing-or-fail story applies to AuditRecord CRs in the apiserver datastore. The chart default historian.audit.archival.activeRetentionDays: 90 is calibrated for low-volume clusters. On a busy pharma site generating hundreds of records per minute it will saturate kine sqlite (unresponsive past ~1 GB state.db) or hit etcd's --quota-backend-bytes (default 2 GB) inside a week. The 2026-04-27 incident accumulated 4,676 records in one namespace over 21 days at the old default. At the documented chart default the same site would have held ~36k records (~36 MB) and the cluster ~140k records.

Set activeRetentionDays so that records_per_day × activeRetentionDays × 1 KB stays under 20 % of your apiserver datastore budget. The records still live in PostgreSQL via the archiver, and dcs audit verify --archived queries them transparently. Full formula and the initial-drain behavior on freshly-enabled archival are in capacity-planning § Audit archive.

4. Required monitoring

The chart already emits ServiceMonitor and PrometheusRule resources by default (monitoring.enabled: true in values.yaml), on clusters that have the Prometheus Operator CRDs installed. On a cluster without them the chart installs cleanly and simply skips both (#1345). Run helm upgrade after installing the Prometheus stack and the monitors appear. Customers must run a Prometheus stack capable of consuming them. The recommended path is the upstream kube-prometheus-stack chart, deployed in its own namespace.

Alerts that must be wired

Alert	Source	Severity	Why
Two consecutive prune-job failures	kube_job_status_failed{job_name=~".*-historian-prune-.*"} for 12 h	critical	Two missed cycles let the historian PVC outgrow the disk before retention naturally drops chunks (issue #237)
Node root-fs > 75 %	node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.25 for 15 m	critical	Precedes kubelet's hard-eviction threshold (~85 %); BestEffort cleanup pods get evicted at hard-eviction, so action must come before that (issue #238)
CNPG primary Cluster not Ready	cnpg_cluster_status{value="Cluster in failed phase"} == 1	critical	Historian writes block; gateway shows stale data
Reconcile error rate	dcs_reconcile_errors_total[5m] / dcs_reconciles_total[5m] > 0.05	warning	Already shipped by the chart's prometheusrule.yaml
Gateway 5xx rate	nginx_ingress_controller_requests{service=~".*gateway.*",status=~"5.."}	warning	User-facing degradation
Cert near expiry	cert_manager_certificate_expiration_timestamp_seconds - time() < 8 * 3600	warning	Already shipped (24 h cert rotation, 8 h renewal — see rotation-runbook.md)

The first two ship with the chart as opt-in PrometheusRule resources, gated on monitoring.prometheusRule.historianDisk.enabled (default false). They are off by default because they only fire usefully on deployments that use prune-CronJob retention (non-prod) or local-path-style unbounded growth (also non-prod), the same structural shortcuts production must not take. The reference cluster turns them on. You should leave them off and size the historian PV for the full retention window instead.

If you want them anyway (e.g. a proof-of-concept that mirrors the reference-cluster shape), set:

monitoring:
  prometheusRule:
    historianDisk:
      enabled: true
      pruneStaleHours: 12   # two-strike threshold against a 6h prune schedule
      diskWarnPct: 75       # precedes ~85% kubelet hard-eviction

The remaining alerts in the table (CNPG, ingress, certs) are upstream metrics. Install kube-prometheus-stack with the corresponding ServiceMonitor discovery enabled and the rules fire automatically.

Installing kube-prometheus-stack

The HelmRelease at flux/clusters/demo/apps/kube-prometheus-stack.yaml in cndcs-deploy-demo is the worked example. It installs end-to-end with our chart's ServiceMonitor + PrometheusRule resources. For production, relax the storage knobs (a real CSI-backed PVC replaces emptyDir) and bump retention to match your alerting needs (15 d is the upstream default, and 6 h–24 h is a single-node constraint that comes from disk).

The non-obvious knobs that have to be set right:

prometheus:
  prometheusSpec:
    # Discover ServiceMonitor + PrometheusRule + PodMonitor + Probe from
    # any namespace, regardless of release label. Without these flags the
    # chart's defaults match only its own resources, and the DCS chart's
    # ServiceMonitor + PrometheusRule never get scraped.
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false

alertmanager:
  alertmanagerSpec:
    # Mount webhook URLs + API tokens via Secret references, not inline
    # in alertmanager.config. The chart's rendered config Secret is part
    # of `helm get manifest` output and any gitops backup, so inline
    # credentials leak into places they shouldn't be.
    secrets:
      - alertmanager-webhook
  config:
    global:
      # Alertmanager mounts secrets at /etc/alertmanager/secrets/<name>/.
      # The `_file` form reads the credential at notify time, so rotating
      # the secret takes effect on the next alert without restarting
      # Alertmanager. Same pattern for `pagerduty_routing_key_file`,
      # `opsgenie_api_key_file`, etc.
      slack_api_url_file: /etc/alertmanager/secrets/alertmanager-webhook/url

Single-binary Kubernetes distros (k3s, k0s, microk8s). The chart ships separate ServiceMonitors for kube-controller-manager, kube-scheduler, kube-proxy, and kube-etcd. On these distros those processes are collapsed into the apiserver, and the standalone scrapes report 0/1 up permanently. Disable them in your overlay:

kubeControllerManager: { enabled: false }
kubeScheduler:         { enabled: false }
kubeProxy:             { enabled: false }
kubeEtcd:              { enabled: false }

Alertmanager tls-assets startup race. On a fresh install the prometheus-operator creates the tls-assets Secret asynchronously, and the Alertmanager pod's first mount can fail with failed to sync secret cache: timed out waiting for the condition. The pod retries automatically once the Secret lands. Expect ~30 s of Init:0/1 on first install. Persistent past 5 min means the prometheus-operator pod itself isn't Running. Start there.

Validating PrometheusRule changes before deploy

A PromQL syntax error in prometheusrule.yaml is not caught by helm template, helm lint, or any Go-side test. It surfaces only at the prometheus-operator's mutating admission webhook in-cluster, by which point the Helm upgrade has already started and Flux is in a Running 'upgrade' action state that needs an explicit unlock to recover. Add promtool check rules against the rendered chart to your CI for any change that touches templates/prometheusrule.yaml:

helm template release deploy/helm/cloud-native-dcs/ \
  --set monitoring.prometheusRule.historianDisk.enabled=true \
  | python3 -c '
import sys, yaml
for d in yaml.safe_load_all(sys.stdin):
    if d and d.get("kind") == "PrometheusRule":
        print(yaml.safe_dump({"groups": d["spec"]["groups"]}))' \
  | docker run -i --entrypoint promtool \
      quay.io/prometheus/prometheus:v2.54.1 check rules /dev/stdin

PromQL pitfalls that have bitten this codebase:

• group_left / group_right followed immediately by ( parses as the labels-list form group_left(label_a, label_b). To use the no-labels form before a parenthesized vector, write group_left() with explicit empty parens.
• Helm and Prometheus both use {{ }} template syntax. Prometheus templates inside chart files must be escaped: {{ "{{" }} $value {{ "}}" }}. Otherwise Helm tries to interpret $value and renders an empty string into the alert annotation.

Decide the log format before a collector is pointed at the cluster

Every component logs the same way, and the release-wide logging block is what changes it (ADR 0063). The shipped default is the console encoder at info, which is what the Diagnose panel's Logs tab and dcs health --logs render.

A production cluster shipping logs to Loki, Elasticsearch or a SIEM sets logging.encoder: json. The collector then extracts fields from each entry, in place of matching on text. Do it before the collector's parsers are written, because switching later invalidates them.

Leave logging.level at its default. Setting it to debug turns on every V(1) line the product carries, and that includes one entry per HTTP request the gateway serves. Treat it as a triage setting. The per-request cost lands on the Logs tab first, since its window is 2000 lines.

See Security Operations § Log format for the accepted values and what each stream carries.

5. Operator hardening

Set priorityClassName on cluster-critical pods

Under DiskPressure or MemoryPressure, kubelet evicts in QoS order (BestEffort first, then Burstable, then Guaranteed). Even a Guaranteed pod without a priority class can be evicted before kube-system workloads. The following pods are cluster-critical and must keep running through node pressure:

priorityClassName: system-cluster-critical

Apply to:

• cnpg-cloudnative-pg (the CNPG operator) — losing it pauses every CNPG Cluster admission and breaks Flux during recovery
• cert-manager controller, webhook, and cainjector — losing the webhook pauses every Certificate apply and stalls Flux + Helm upgrades
• ingress-nginx-controller — losing it cuts user traffic
• The local-path-provisioner-cleanup helper pod (BestEffort by default upstream) if you must keep local-path on a non-prod cluster. Under DiskPressure the kubelet evicts BestEffort pods first, so the very pod that reclaims disk is the first casualty (observed during a 2026-04-22 demo-instance incident)

The DCS operators (physical-operator, procedural-operator, batch-operator, control-operator) and the gateway should run as Guaranteed QoS (limits = requests) but do not need system-cluster-critical. They are recoverable from a cold start.

Webhook failurePolicy trade-off

On a single-node cluster, the CNPG validating webhook becoming unavailable (operator evicted) breaks every subsequent CNPG apply. The mitigation is failurePolicy: Ignore on the validating webhook, which lets applies through during operator downtime at the cost of weaker validation. Production must keep failurePolicy: Fail: stricter validation is the entire point of running CNPG in a regulated environment, and a multi-node deployment with operator HA means the webhook is essentially never unavailable. Only consider Ignore on intentionally non-HA non-production clusters.

CNPG minimum version

The bundled historian database requires CloudNativePG chart 0.27.0 (operator 1.28) or newer. 0.27.0 is the first release whose clusters.postgresql.cnpg.io CRD declares .spec.podSecurityContext, which the chart sets on the historian Cluster so the TimescaleDB-HA image's UID-1000 postgres process can read its PGDATA. Against an older CNPG the apply is rejected with .spec.podSecurityContext: field not declared in schema.

Declare the floor as >=0.27.0 wherever CNPG is pinned. A wider range such as >=0.22.0 <1.0.0 happens to resolve to a new enough chart today only because Helm selects the newest match. It protects nobody who pins deliberately or installs into a cluster that already runs an older CNPG.

The chart ships a render-time guard (templates/historian-database-cnpg-version-check.yaml) that inspects the installed CRD for the field itself, skipping version-string comparison, and refuses the release with remediation before any apply.

CNPG immutable-field guard

The chart ships a render-time guard (templates/historian-database-immutability-check.yaml) that hard-fails helm upgrade --dry-run if a chart rev would mutate spec.postgresUID or spec.postgresGID on an existing Cluster. Both fields are immutable in CNPG. Without the guard, an upgrade trips the validating webhook and Flux loops on RollbackFailed. The guard is silent on fresh installs and on no-op upgrades. See upgrade-rollback.md § "Pre-flight Checks".

Node drains and the historian database

A drain evicts every pod a node hosts, and it honours the PodDisruptionBudgets those pods sit under. CNPG creates a budget over the primary that requires one pod available. At two instances or more that budget is what you want, because it makes CNPG switch the primary over to a replica before the pod is evicted. Without it a drain would drop the database. At one instance the same budget can never be satisfied, so the drain waits indefinitely for a switchover that has nowhere to go.

The chart therefore ties the budget to the instance count (issue #1466). One instance renders enablePDB: false on the Cluster and ships no budget, two or more renders enablePDB: true. Set the value yourself to decide it either way:

historian:
  database:
    cnpg:
      instances: 1
      enablePDB: false   # unset follows the instance count; an explicit value wins

Know what a drain costs you before you start one. A single-instance database is unavailable from the moment its pod is evicted until the pod is running again, and on a node-local storage class it cannot come back anywhere but that same node. The historian stops recording for the duration and the gateway's trend surfaces read empty, while batch execution and the control path are unaffected because neither reads from the historian.

Recognising the failure is worth more than the fix here, because the symptom names nothing. A drain blocked by a budget produces no error and no timeout, and neither kubectl drain nor talosctl shutdown nor talosctl upgrade mentions a budget while it waits. The node stays up and answers normally, which reads as a hardware fault. Ask the cluster instead:

# What is still on the node, and what is holding it there?
kubectl get pods -A --field-selector spec.nodeName=<node>
kubectl get pdb -A -o wide          # DISRUPTIONS ALLOWED 0 is the answer

talosctl shutdown --force and talosctl upgrade --force skip the cordon and drain entirely. That is safe for PostgreSQL, which is crash-safe and is stopped in order by the later shutdown phases regardless. It skips the drain for every other workload on the node too. Treat it as a way out of a stuck bench machine, and keep it out of site procedures.

Power provider (optional)

Kubernetes has no power verb, so rebooting a chassis from the Servers surface needs a channel below the cluster. That channel is off unless you configure it, and a deployment that leaves it off shows no power verbs at all.

The provider runs in the physical operator, which already holds every node verb. The conduit toward the management network therefore originates in one component. The gateway receives the provider's name only and uses it to decide which buttons to render. It never receives the credentials.

Two backends ship. Which one you configure decides whether the product can stop a machine at all, so choose it against the recovery you actually have.

kind	Channel	Verbs	Machine comes back by
talos	The Talos machine API, in band	reboot, and shutdown only where a person is declared at the rack	Itself after a reboot, a person pressing the power button after a shutdown
redfish	A BMC on each machine, out of band	reboot, shutdown, power-on, and a power-state readback that confirms a shutdown reached off	The product

The Talos backend

powerProvider:
  kind: talos                      # "" (default) opens no conduit at all
  attendedRack: false              # true where somebody can be at these machines
  endpoints:                       # control-plane addresses that proxy the machine API
    - 10.10.10.11
    - 10.10.10.12
  credentials:
    secretName: dcs-talos-power    # required whenever kind is set
    key: talosconfig
    mountPath: /etc/dcs/power

What to know before turning it on:

• Talos can stop a machine and can never start one. A machine that is off serves no API, so a shutdown over this channel is undone by a person at the rack and by nothing else. The product offers it only where powerProvider.attendedRack says somebody can be there, which is the recoverability rule in ADR 0034 § 4 as amended.
• The credential is a machine identity. For Talos it is a talosconfig carrying an os:admin client certificate, which can reboot, reconfigure and wipe every node in the cluster: Talos has no narrower role. Treat it as a cluster-root credential, hold it in a Secret the operator alone mounts, and rotate it on the same schedule as your other cluster-root material. The file is re-read on every action, so a rotation takes effect without restarting the operator.
• Leave powerProvider.endpoints empty to use the endpoints the mounted talosconfig declares, and failing that the target machine's own address.

The Redfish backend

A baseboard management controller has its own processor, its own network attachment and its own power. It keeps answering while the machine it manages is off. That is what makes a shutdown recoverable by the product with no person at the rack, and it is what an unattended site, a remote site and a UPS that restores a plant by itself all require.

powerProvider:
  kind: redfish
  credentials:
    secretName: dcs-bmc-power      # required whenever kind is set
    key: redfish.yaml              # the default for this backend
    mountPath: /etc/dcs/power

The Secret holds the BMC account and the trust for its certificates:

username: dcs-power
password: <from the vault>
caCert: |                          # the CA the BMC certificates chain to
  -----BEGIN CERTIFICATE-----
  ...

What to know before turning it on:

• Each Node names its own BMC. Set dcs.io/bmc-endpoint on every Node the channel should reach, as a host, a host:port or an https:// URL. The deployment layer sets it at enrolment, because Kubernetes knows nothing about a BMC and guessing is how the wrong chassis gets powered off. A node without the annotation refuses every power verb and says so. Where one BMC fronts several systems, as in a blade chassis, add dcs.io/bmc-system-id naming which one that node is. Without it the product refuses the verb outright.
• powerProvider.endpoints is refused for this backend. A BMC belongs to one chassis, so a deployment-wide list describes a channel that does not exist. The chart fails the render, because the alternative aims every action at one machine.
• Nothing here cuts power. Every verb goes out as the graceful Redfish reset type, and a BMC that will not perform one gets a refusal naming what it does offer. The product never substitutes ForceOff for GracefulShutdown.
• A shutdown here is a confirmed observation. The BMC keeps answering while the machine is dark. After a shutdown the operator reads the power state back until the machine reports off, and the NodeMaintenance records that as an observation. An in-band channel cannot make that claim, and its record says the request was accepted instead. The readback runs for one machine at a time, every 15 seconds, for at most 5 minutes from the shutdown, and it stops as soon as the machine reports off. Nothing polls your BMC estate to keep a screen fresh.
• Plain HTTP is refused. HTTP Basic puts the BMC account on the wire in every request, so the endpoint must be https.
• Supply a CA. BMCs ship self-signed certificates, and insecureSkipVerify: true is the way to start against an estate that still carries them. It costs the conduit its authentication: anything on the management network can then answer as a BMC. Move to a CA as soon as there is one, and record the interim state in your zone documentation.
• One account covers every BMC, which is the blast radius. Revoking it takes the power channel away from every machine at the site at once. The file is re-read on every action, so a rotation takes effect without restarting the operator.

Common to both:

• The chart does not bound where the conduit may reach. The operator NetworkPolicy declares ingress only. If your zone model requires the operator's egress restricted to the management subnet, add an egress policy that also covers its other destinations: the apiserver, DNS, the MQTT broker, the unit runtimes, and the OTLP collector.
• A misconfiguration fails at deploy time. A kind with no credentials.secretName fails the Helm render, and a credential the operator cannot read or parse fails its startup. Neither is discovered during an incident.

6. Backup & DR

backup-recovery.md is the canonical reference. The production decisions to make at install time:

Enable CNPG WAL archiving

historian:
  backup:
    enabled: true
    retentionPolicy: "30d"           # tune to your RPO + budget
    schedule: "0 0 3 * * *"          # CNPG cron is 6-field (leading seconds), NOT 5-field crontab
    s3:
      bucket: "<your-historian-backup-bucket>"
      path: "/historian"
      endpointURL: ""                # set for non-AWS S3 (DO Spaces, MinIO, …)
      secretRef: "historian-backup-s3"  # ACCESS_KEY_ID + ACCESS_SECRET_KEY
      egress:
        destinationCIDRs: ["0.0.0.0/0"]  # or the store's own range

A production cluster enforces NetworkPolicy, so the database pods also need the store's address written into historian.backup.s3.egress. A hostname is not something a NetworkPolicy can resolve, and those pods reach only what their own policy names. The chart refuses to render without it (#1516). The port comes off endpointURL, so a store on :9000 is allowed on 9000. That rule hard-coded 443 until #1516, which denied every non-443 store while allowing 443 to the entire internet.

Set schedule explicitly: CNPG's ScheduledBackup uses the Go cron format with a leading seconds field, and a 5-field crontab string is silently reinterpreted ("0 3 * * *" runs hourly at HH:03:00). See backup-recovery.md § "Historian Database Backup".

The reference deployment intentionally sets historian.backup.enabled: false because its data is disposable and a misconfigured WAL archiver previously filled the root disk on its own (see the comment in cndcs-deploy-demo's flux/clusters/demo/apps/dcs-release.yaml). Production should never disable backups on the historian.

Bucket sizing for WAL archive + audit mirror

Two distinct buckets, sized independently:

• WAL archive bucket — Sized as (daily WAL volume) × retentionPolicy, typically 0.5×–2× the historian PVC size depending on write churn. Lifecycle: bucket-default, with CNPG enforcing retention via retentionPolicy.
• Audit Object-Lock mirror (optional, when historian.audit.archival.immutable.enabled: true): sized for 7-year retention per 21 CFR Part 211. At ~10 batches/day with a typical batch record size this is a few GB of locked storage. It is far cheaper than the WAL bucket, because audit records compress well and are written once. See backup-recovery.md § "Optional Immutable S3 Mirror (Object Lock / WORM)".

Object Lock in Compliance mode means no identity, including the bucket owner, can delete objects until lock expiry. Verify your account allows Compliance mode. Most providers do. DigitalOcean Spaces does not as of this writing, so use AWS S3 / minio / Backblaze B2 / Wasabi there.

A production cluster enforces NetworkPolicy, so the mirror also needs its destination written into historian.audit.archival.immutable.egress. A hostname is not something a NetworkPolicy can resolve, and the archiver reaches only what its own policy names. The chart refuses to render without it (#1514). The archiver proves the endpoint answers at startup and fails the run if it does not, which is what keeps a denied egress from reading as a clean hourly no-op until the first record crosses activeRetentionDays.

7. Failure modes — single-node reference cluster vs production

Each row is a real failure observed on a single-node reference cluster and documented in the runbooks. Production deployments built per this guide do not see these because the structural preconditions don't exist.

Failure mode	Trigger	Production prevention
DiskPressure cascade	local-path PVC outgrew root disk; helper-pod-delete-pvc was BestEffort and got evicted before it could free space	CSI driver with quota enforcement (§ 1); historian on its own block volume; system-cluster-critical priority class on cleanup pods (§ 5)
Flux RollbackFailed loop on CNPG webhook unavailability	Single-node cluster with CNPG operator on the only node, evicted under DiskPressure	Multi-replica CNPG operator across multiple nodes (§ 2); failurePolicy: Fail is then safe
postgresUID/postgresGID immutability blocked an upgrade	Chart rev mutated CNPG immutable fields with no pre-flight detection	Chart-shipped immutability guard (§ 5); helm upgrade --dry-run in pre-flight (§ Upgrade)
Two consecutive prune-CronJob failures filled the disk	Chart's prune-job retry budget shorter than its scheduling interval; no monitoring on its own success rate	Production does not run the prune CronJob — TimescaleDB's native retention policy handles bounded growth on adequately sized PVs (§ 3)
Stale disk-pressure:NoSchedule taint after manual cleanup	Kubelet did not auto-remove the taint; CNPG operator wouldn't schedule	Multi-node cluster means at least one node is always schedulable; manual taint removal is rarely needed
Ghost NotReady simulation nodes accumulated in the cluster	Transient simulation pods register node objects that outlive the pod	Production unit runtimes are long-lived per-device pods, so ghost nodes stay a simulation-only artifact
ghcr.io image-pull rate limit during recovery	Mass restart on a single node hammered ghcr from one source IP	Multi-node deployments spread pull traffic; consider an in-cluster pull-through registry (Harbor, Sonatype Nexus) for air-gapped sites
kine bloat from AuditRecord accumulation	Chart-default activeRetentionDays: 90 × ~300 records/min on the reference cluster grew state.db past 1 GB; k3s/kine became unresponsive	Set historian.audit.archival.activeRetentionDays against actual record rate (§ 3 § AuditRecord retention); embedded etcd raises the ceiling but does not eliminate it
audit-archiver OOMKill on large drains	Unpaginated client.List loaded a 4.6k-record namespace into one heap, peaked past the chart's 1 Gi limit	Fixed in #245 — archiver is now paginated (listPageSize × record_size), chart default memory is back to 512 Mi

Pre-deployment checklist

Run through this before declaring a deployment production-ready:

• [ ] Storage class for stateful PVCs is a CSI driver with capacity enforcement (§ 1)
• [ ] Historian PVC sized per Capacity Planning § Historian plus 30 % headroom
• [ ] historian.audit.archival.activeRetentionDays sized against expected AuditRecord rate per Capacity Planning § Audit archive. The chart default of 90 is too high for busy sites
• [ ] Cluster has at least 3 control-plane nodes, with CNPG and cert-manager operators replicated across distinct nodes (§ 2)
• [ ] Every controller node's field interface is an active-backup bond, per Network Requirements § Physical-Layer Posture — verified by pulling one member and confirming no Unit Hold. Members on separate switches where available. See the note there on what that does and does not buy when remote I/O is single-homed
• [ ] For any critical continuous parameter, the zero-gap limitation across a controller death is understood and mitigated if required — see HA and Failure Modes § Continuous control and data integrity
• [ ] kube-prometheus-stack (or equivalent) deployed, with alerts from § 4 wired to a notification path
• [ ] system-cluster-critical priority class set on CNPG, cert-manager, ingress-nginx (§ 5)
• [ ] CNPG WAL archive enabled to a backed-up bucket (§ 6), and audit mirror enabled if 21 CFR Part 11 §11.10(c) restore is in scope
• [ ] security-hardening.md checklist completed (OIDC, mTLS, network policies, electronic signatures)
• [ ] gateway.policy.reasonMinLength set to match the site's SOP for audit-trail justifications (product default 10, see API Reference § Deployment reason policy)
• [ ] Component images pinned by digest (<component>.image.digest) per Security Hardening § Supply Chain Verification. The broker is third-party: mqtt.image.tag names a Mosquitto release, so bumping it is a deliberate edit (#1579)
• [ ] MQTT client password rotation rehearsed once on a non-prod copy. It is three helm upgrades with no client disconnect (Rotation Runbook § MQTT Client Passwords), and the third one is the step that stops the old credential working
• [ ] upgrade-rollback.md § Pre-flight Checks rehearsed end-to-end on a non-prod copy of the cluster
• [ ] If northbound OMF egress is in scope: equipment and tag names agreed with whoever owns the PI System before the first site publishes, because they become AF element and PI Point names that displays are built against and a rename orphans that work. See OMF Egress to AVEVA PI § The one-way door. The spec.northbound.publish declaration on each Site, the endpoint credential, and omfEgress.egress.destinationCIDRs are the three settings that gate it
• [ ] Disaster-recovery procedure from dr-runbook.md walked at least once with the actual storage backend and bucket configuration

Related Documentation

• Deploy Your Own — fresh-install path
• Capacity Planning — sizing math
• Historian Disk-Pressure Runbook — production-safe remediation when the historian fills its disk, codifying the demo-instance failure modes this guide is built to prevent
• Upgrade and Rollback — pre-flight checks including the CNPG immutability guard
• Security Hardening — OIDC, mTLS, network policies, electronic signatures
• Backup and Recovery — CNPG WAL archive + Object Lock audit mirror
• HA and Failure Modes — what happens when each component goes down