High Availability and Failure Modes

HA Architecture Overview

Cloud-Native DCS builds on Kubernetes-native HA primitives and adds process-control-specific mechanisms:

Mechanism	Component	How It Works
Leader election	All operators	controller-runtime Lease; only one replica reconciles at a time
State recovery	All controllers	ISA-88 state machine rebuilt from CRD .status.state on every reconcile
FB network persistence	Unit runtime	Last FB network saved to hostPath; replayed on startup for bumpless restart
Driver reconnection	Unit runtime	Exponential backoff (100ms-30s) with proactive health monitor every 5s
Grace-period health check	Unit runtime	gRPC health reports NOT_SERVING after all drivers disconnected for 60s
Watchdog Hold	Unit controller	Queries runtime status API; issues ISA-88 Hold if all drivers down during active batch
Control lease + self-fence	Unit runtime (availability mode Failover)	Runtime holds an operator-brokered lease; on expiry it self-fences (stops FB output writes, reads continue) on its own clock — works under control-plane partition (ADR 0006)
Hold-then-resume failover	Unit controller (availability mode Failover)	Re-binds a unit's runtime to a designated standby node after lease expiry + safety margin; manual fenced failover for Autonomy (ADR 0006)
Edge-local holding	Unit runtime	On a control-plane partition (heartbeat watchdog fires) an embedded SFC engine runs the armed hold chart — the active phase's holdingChart, else the UnitSpec.safeStateChart baseline — to drive a deliberate sequenced safe state by writing the unit's own control-module tags; the FB scan stays the sole I/O writer (ADR 0008). A holdingChart that addresses another unit's tags, or calls a builtin needing an operator, the apiserver or another machine, is refused when the runtime is asked to stage it, so the baseline covers that phase and the operator learns which at phase start (#1406)
Link redundancy (NIC bond)	Controller node (deployment-configured)	Active-backup bond per network zone; a cable, NIC, or switch-port fault fails over below the runtime and never surfaces as a partition. Members on separate switches where available — though single-homed remote I/O caps what that buys for the field zone (ADR 0026). Deployment-layer machine config, transparent to the FB scan and drivers
MQTT store-and-forward	Unit runtime	Buffers up to 10K messages during broker disconnection
Liveness/readiness probes	All components	Kubernetes restarts unhealthy pods; removes unready pods from service
Startup probes	All operators, gateway, historian	Allows up to 150s for initialization before liveness probe takes over
PodDisruptionBudget	All deployments	Auto-enabled when replicas > 1; prevents voluntary disruption from removing all replicas
Pod anti-affinity	All deployments	Spreads replicas across nodes when replicas > 1

Failure Mode Analysis

The coordination layer is elastic where that is safe, and deterministic where it must be. Batch state lives in the model (the Batch and Procedure resources) and never in an operator pod, so the pods above the process are ordinary replaceable Kubernetes workloads. The unit runtime scanning real I/O is pinned to its device node and never disturbed. The clip below demonstrates the first row of the table live. The procedural-operator pod (the SFC engine) is deleted mid-batch, Kubernetes replaces it, and the replacement resumes the chart exactly where the model says. The unit-runtime pods show zero restarts throughout.

Killing the SFC engine mid-batch: the procedural-operator pod is deleted while a batch runs, Kubernetes reschedules it, and the chart advances charge → inoculate under the replacement. Unit runtimes untouched, RESTARTS 0.

Failure Mode	Detection	Response	Recovery Time	Impact
Operator pod crash	Kubernetes liveness probe (20s period)	Pod restart (RestartAlways) + leader election handoff	15-30s	Reconciliation paused; running batches continue autonomously via runtime
Unit runtime pod crash	Kubernetes liveness probe	Pod restart + FB network replay from disk	20-40s	I/O ceases during restart; batch Held by watchdog if enabled
All I/O drivers disconnected	gRPC health check (60s grace) + watchdog	Watchdog issues Hold; kubelet restarts pod after health probe fails	60s grace + 15s probe	Batch placed on Hold; manual Restart after drivers recover
Single driver disconnected	ReconnectingDriver health monitor (5s)	Auto-reconnect with exponential backoff	100ms-30s	Partial I/O loss; affected channels return stale values
MQTT broker down	Store-and-forward queue + reconnect; dcs_mqtt_connected == 0 on operators/gateway	Runtime buffers messages and replays on reconnection; operators and gateway retry in the background (capped exponential backoff, ≤30s) and resume publishing automatically — including when the broker first comes up after them	Transparent	Historian data gap; HMI shows "No MQTT" until reconnect; no control impact
etcd unavailable	Kubernetes API errors	Controllers back off; running batches continue	Kubernetes-dependent	New batches cannot start; running processes unaffected
Network partition (operator-runtime)	Watchdog HTTP timeout (5s); lease renewal failure (Failover mode); control-plane heartbeat watchdog (edge-local holding, ADR 0008)	Autonomy: log warning, runtime continues autonomously; if the partition outlasts availability.holdGraceSeconds the embedded SFC engine runs the armed hold chart to a deliberate sequenced safe state and keeps holding (no fence — Autonomy is the legitimate single writer). Failover: runtime self-fences on lease expiry — on its own timer for a symmetric partition, and on the operator's reported acknowledged-renewal age for a reverse-path (asymmetric) partition where forward POSTs still land but responses are dropped (#579) — running the armed hold chart as a bounded sequence first, then self-fencing so the re-bind margin (≥ hold duration) keeps one writer; operator may re-bind after the safety margin	Hold grace then pod-restart dependent (Autonomy); lease duration + margin (Failover)	Autonomy: operator cannot issue new commands; the runtime drives the unit's own CM tag space to a sequenced safe state and holds. Failover: bounded hold sequence, then outputs held on self-fence, then failover to a standby. On reconnect the runtime's self-held status reconciles to Held; recovery is an explicit ISA-88 Restart
Controller link fault (cable / NIC / switch port) to a live node	Bonded (recommended, ADR 0026): bond driver detects member link-down; no control-plane symptom. Single-homed: indistinguishable from a node partition — watchdog HTTP timeout / lease renewal failure	Bonded: traffic moves to the surviving member; the runtime keeps regulating and keeps publishing. Single-homed: escalates down the network-partition row above — hold chart and/or self-fence, then a failover of a controller that never actually failed	Bonded: sub-second, below the runtime. Single-homed: as for network partition	Bonded: none — no control or data gap. Single-homed: a full Hold/failover cycle for what is only a cable fault. This row is the reason link redundancy is the standard posture
Edge node failure — Autonomy (default)	Kubernetes node controller (40s default); unit watchdog/self-Hold	Unit lands in Held; failover is a manual operation (dcs unit failover --confirm-fenced once the node is powered off / field-disconnected) re-binding the runtime to a designated standby	Manual (operator-paced)	One unit's control loop down until failover or hardware replacement; running phase survives in Phase.Status
Edge node failure — Failover (opt-in)	Lease expiry observed by the operator	Runtime self-fenced at lease expiry; a unit still Running mid-batch is placed on ISA-88 Hold by the operator itself once expiry outlasts the hold bound (½ lease, clamped 5–30s); operator auto-re-binds to a free eligible standby after lease + safety margin (½ lease, clamped 10–60s); new epoch stamped	Lease duration + margin (tens of seconds)	Held outputs during the gap, then hold-then-resume on the standby; recovery is an ISA-88 Restart
Lease loss / self-fence (Failover mode)	Runtime local timer, or operator-reported acknowledged-renewal age ≥ lease duration (two-way liveness, covers reverse-path partitions, #579); dcs_runtime_fenced == 1; operator marks lease Expired	Runtime drops FB output writes (reads continue), reports NOT_SERVING, publishes retained fenced status; critical Alarm + AuditRecord	Immediate (self-fence)	Field outputs hold last value / device fail-safe; one-writer guaranteed before any standby takes over
Gateway pod crash	Kubernetes liveness probe	Pod restart; WebSocket clients must reconnect	10-20s	UI temporarily unavailable; no control impact
Control Module FB network down (single CM)	dcs_cm_fb_network_running == 0 + retained MQTT health topic	Runtime publishes raw driver reads at 1 Hz with quality: "Raw" so CM PVs stay visible; ControlProgram reconciler redeploys on generation/pod change	Runtime-dependent	PV engineering scaling unavailable for that CM; operator sees raw value flagged Raw; other CMs unaffected
Single FB block error (e.g. transient driver write fault on one address)	CM health state Degraded + dcs_cm_program_degraded_total increments	Faulting block is skipped and flagged; the scan continues and keeps driving (and interlocking) every healthy output — the program does not halt (ADR 0009)	Self-clearing when the block recovers (next scan)	Faulted output holds its last value; all other regulation and protection continues; persistent degradation is the operator signal
Control program deliberately removed	Operator/reconciler removal	Output blocks driven to their configured fail state once before teardown (failState: safeValue default, or holdLast; ADR 0009, IEC 62443 SR 3.6 #972)	Immediate	Outputs go to their configured predetermined state on removal; a hot-swap redeploy holds instead, to avoid glitching live outputs
Graceful runtime stop for a redeploy (rolling update, image bump, node-pressure eviction)	Pod recreated by the reconciler	Nothing is written: output continuity is preserved on purpose, the same reasoning the hot-swap path uses (ADR 0009, #1283)	Runtime returns in seconds	Outputs hold their last commanded value across the gap; safing here would bump the field on every deployment
Graceful runtime stop for an outage (terminal stop armed)	POST /api/v1/stop/arm on the runtime before the pod is stopped	Output blocks driven to their configured fail state once, while the drivers are still connected and before the runtime exits (ADR 0009 amendment, #1283)	Bounded by the safe-write budget, inside the pod's termination grace period	Outputs reach their configured predetermined state; the runtime queues a safe-stop record that the operator materializes into an AuditRecord, so an outage's trail can assert the field was safed
Ungraceful runtime process death (OOM-kill, node loss, kill -9)	No software runs on this path	The device's own comm-loss watchdog acts, if one is declared. Declare it with spec.failSafe on the IOModule (ADR 0068); an IOModule that declares nothing keeps whatever its device shipped with, and says so through the FailSafeDeclared condition. The declared timeout has a floor of 45 s, three io-probe read cadences, because any traffic feeds the watchdog and a shorter deadline fires on a routine gap (ADR 0071)	Declared timeout	What a device can do is per model. A WAGO 750 coupler clears the whole node and offers no substitute value; a cleared 4-20 mA output sits at 4 mA, which reads downstream as a valid 0% and not as a fault

Continuous control and data integrity across a controller failure

The failure table above is written from a batch and procedural perspective, where ISA-88 Hold/Restart is the sanctioned exception path and a bounded gap is immaterial. Continuous control at the control-module level has a different exposure, and it is worth stating plainly.

Most of the layering already protects the record:

• A broker, historian, or downstream-network fault is covered by store-and-forward. The runtime buffers and replays, so there is no permanent record gap.
• A single CM's FB network going down while the runtime lives leaves the PV visible: the runtime publishes raw driver reads at 1 Hz flagged quality: "Raw".
• A cable, NIC, or switch-port fault is masked entirely on a controller with a bonded field interface (ADR 0026), the recommended posture.

What is not covered is the death of the controller node itself. On failover, a continuous PID loop stops and re-establishes on the standby: integral state is held in the runtime process and persisted to a node-local hostPath, and a hostPath does not follow the pod to another node. The loop restarts from its configured initial conditions. Where it left off is gone with the node. During the RTO window outputs hold last value, go to their device fail-safe (ADR 0009), or are sequenced by an armed SafeStateChart (ADR 0008), and no reader is on the field. The result is a bounded data gap as well as a control gap.

We do not guarantee zero-gap continuous control or an unbroken record across a controller death. For most processes this is immaterial. For a critical continuous parameter under 21 CFR Part 11 / ALCOA+ it may not be. What the product does guarantee is that the gap is bounded, timestamped, and cause-attributed, bracketed by a Hold event, a failover AuditRecord, and an explicit ISA-88 Restart. The bracket is what makes the gap a documented one. It is also carried inside the batch production record. At batch-terminal time the reconciler materializes each failover into the record's spec.failoverEvents[], with the control gap (Hold → Restart) and the data gap (fence → lease re-established) bounded separately and each bound referencing its evidencing AuditRecord. A QA reviewer reads the explained gap in the record itself, with no operator log to chase (ADR 0041, #1308, and Batch Production Records).

If your process needs the unbroken record, the decided answer (ADR 0041) is redundant-collector sourcing: a redundant PLC or OPC UA server with historical buffering that the historian backfills on reconnect. That keeps the record whole even when control briefly holds, and in regulated contexts the record is more often the binding requirement than bumpless regulation. Raise it during design. It is a deployment topology, and no product setting turns it on. Hot-standby FB-state replication is deferred behind the reversal triggers ADR 0041 records.

RTO figures in the table above are derived from lease and probe timings and have been exercised on kind/CI. Metal has not measured them yet. Measured hardware numbers for a continuous loop, and a demonstration that bonding masks a cable-pull with no gap, are tracked in #942 under the bench epic #400.

IEC 62443 Availability Targets (FR 7)

Component	Availability Target	Basis
Control operators	99.9% (8.7h/year downtime)	Leader election + 2 replicas
Unit runtime (Autonomy)	99.5% (43.8h/year downtime)	Single pod per unit; restart + replay; manual fenced failover on node loss
Unit runtime (Failover)	99.9% (8.7h/year downtime)	Adds lease-based self-fence + automatic re-bind to a designated standby (ADR 0006); requires standby hardware with field-network reach
Gateway	99.9%	Multiple replicas + autoscaling
MQTT broker	99.5% (single) / 99.9% (HA)	Single instance by default; optional HA mode (mqtt.ha.enabled) deploys StatefulSet with configurable replicas and per-instance storage

These targets assume the HA profile (values-ha.yaml) is deployed.

Deploying for High Availability

Enable HA by applying the overlay values file:

helm install dcs ./deploy/helm/cloud-native-dcs \
  -f deploy/helm/cloud-native-dcs/values-ha.yaml

This enables: - 2 replicas for every operator - PodDisruptionBudgets (auto-enabled when replicas > 1, with explicit enable also supported) - Gateway autoscaling (2-5 replicas) - Pod anti-affinity (spread across nodes) - PrometheusRule alerting - MQTT HA mode (StatefulSet with per-replica storage, enabled with mqtt.ha.enabled=true)

Alert Runbooks

DCSRuntimeUnhealthy

Severity: Critical Condition: dcs_runtime_healthy == 0 for > 2 minutes

What happened: All I/O drivers on a unit runtime have been disconnected. The runtime's gRPC health check is reporting NOT_SERVING.

Actions: 1. Check the Diagnostics page for runtime status 2. Check driver reconnection metrics: dcs_driver_reconnect_failures_total 3. Verify field device connectivity (network, power, protocol endpoint) 4. If the runtime pod was restarted by kubelet, check if FB network replayed successfully

DCSRuntimeDriverDown

Severity: Warning Condition: dcs_runtime_driver_connected == 0 for > 5 minutes

What happened: A specific I/O driver has been disconnected for an extended period. The ReconnectingDriver is attempting automatic reconnection.

Actions: 1. Check which driver: look at driver_name and protocol labels 2. Check IOModule status: dcs get iomodules -s <site> 3. Verify the field device is reachable on the network 4. Check for protocol-specific issues (Modbus TCP port 502, OPC UA port 4840)

DCSRuntimeDriverConfigRefused

Severity: Warning Condition: dcs_runtime_driver_config_valid == 0 for > 1 minute

What happened: The runtime accepted the driver and refused its configuration. This is not a connectivity failure and it does not look like one. The driver connects, answers every read, and reports dcs_runtime_driver_connected == 1. What it does not do is run the profile the module declares, so a simulation module holds every channel at its initial value and injects no faults. The plant presents as a set of tags that have stopped moving.

The reachable causes are a behaviour or fault parameter the driver would not parse, a dependency cycle among behaviours, and a profile whose JSON does not unmarshal. The gateway refuses each of those in an authored document (ADR 0064). A module in this state usually arrived through kubectl apply, through POST /api/v1/apply, or from a document written before that guard shipped.

Actions:

1. Read the message. It is not a metric label. dcs get runtime -s <site> --unit <unit> prints it in the CONFIG column, and the Diagnose panel's Runtime tab shows it in the I/O Drivers table.
2. Fix the IOModule's spec.simulation block or the SimulationPreset it binds, and apply it. The unit controller pushes the corrected config to the running runtime, which re-applies the profile without a pod restart.
3. Confirm the gauge returns to 1 on the next watchdog poll.

One failed behaviour costs the whole module's profile, so the message names the first refusal and the module runs none of its behaviours until that is fixed.

DCSOperatorDown

Severity: Critical Condition: One or more operator up metrics absent for > 5 minutes

Actions: 1. Check the Diagnostics page for operator health status 2. Contact your system administrator if operators are unhealthy 3. Verify RBAC: missing permissions cause silent failures

DCSReconcileErrorRate

Severity: Warning Condition: Reconciliation error rate > 0.1/s for 5 minutes

Actions: 1. Check operator logs for error messages 2. Common causes: missing CRDs, RBAC issues, etcd timeouts 3. Check if a recent deployment introduced a regression

DCSMQTTQueueBacklog

Severity: Warning Condition: MQTT queue depth > 1000 messages for > 5 minutes

Actions: 1. Check the Diagnostics page for MQTT broker status 2. Verify broker connectivity from runtime: check dcs_runtime_mqtt_publishes_total, whose outcome label separates what reached the broker from what is queued or dropped 3. If queue is approaching the 10K limit, messages will be evicted (check dcs_runtime_mqtt_queue_evicted_total)

DCSWatchdogHoldTriggered

Severity: Critical Condition: Watchdog Hold triggered in the last 5 minutes

What happened: The unit controller's watchdog detected that all I/O drivers on a unit were disconnected while a batch was actively running. The unit was placed on ISA-88 Hold to prevent uncontrolled process behavior.

Actions: 1. Investigate why all drivers disconnected (see DCSRuntimeDriverDown runbook) 2. Once drivers are restored, issue a Restart command to the held batch running on the unit: dcs command Batch <batch-name> Restart -s <site> 3. Verify the batch resumes correctly 4. Review the Alarm CR created by the watchdog for timestamps and details

Unit runtime fenced (control lease lost)

Severity: Critical Condition: dcs_runtime_fenced == 1 for > 1 minute (availability mode Failover)

What happened: A Failover-mode unit runtime could not renew its control lease and self-fenced. It stopped writing FB outputs (reads continue) and reports NOT_SERVING. The physical operator lost contact with the runtime. The unit's RuntimeBinding.leaseState shows Expired and a critical lease-expired Alarm was raised. Field outputs hold last value (or device fail-safe). This is the designed one-writer guarantee at work, and no fault in itself. It does mean that unit is not controlling.

Actions: 1. A brief 1 at runtime startup is normal (Failover runtimes start fenced until their first lease grant). Only sustained fencing is an incident. 2. Determine whether the edge node is dead/partitioned or the operator↔runtime path is broken: dcs get unit <name> -s <site> (RuntimeBinding + FailoverRequest conditions), node status, and dcs_runtime_lease_expirations_total. 3. If the node is genuinely down, the operator auto-re-binds to a free eligible standby after the safety margin. Confirm a standby exists (dcs unit failover <name> lists eligible targets) and is enrolled (dcs.io/site label) and Ready. 4. If no free standby exists, a NoFreeTarget condition is set. Provision or free a standby, or run a manual co-located failover with dcs unit failover <name> --to-node <node> --allow-colocation --reason "<justification>". --reason is required with --to-node and lands in the audit trail. 5. Recovery on the standby is an ISA-88 Restart. Verify the batch resumes and the lease returns to Held.

DCSCertExpiryWarning

Severity: Warning Condition: A TLS certificate file observed by a DCS component expires in <48h

What happened: cert-manager normally renews certs 8h before NotAfter (24h duration, 8h renewBefore). If the gauge dcs_cert_expiry_seconds - time() has fallen below 48h, renewal has not yet landed, most commonly because cert-manager is wedged or the component is an off-cluster edge node that cert-manager doesn't manage.

Actions: 1. Identify the affected cert: $labels.component, $labels.host, $labels.file. 2. Check cert-manager: kubectl get certificate -A (look at Ready + Renewal). 3. Describe the Certificate: kubectl describe certificate <fullname>-<component>-mtls. 4. Check cert-manager logs: kubectl -n cert-manager logs deploy/cert-manager. 5. Confirm the reloader picked up the last rotation. Grep component logs for TLS certificate material reloaded. 6. For edge nodes (unit-runtime devices joined at the deployment layer), the in-cluster cert-manager does NOT renew. An automated edge renewal loop is not yet implemented, so rotate manually per the rotation runbook.

DCSCertExpiryCritical

Severity: Critical Condition: A TLS certificate file expires in <8h

What happened: Renewal has failed. mTLS handshakes will break at NotAfter. Unit-runtime ↔ gateway, gateway ↔ historian, and MQTT will all cascade.

Actions: 1. Force-renew: kubectl cert-manager renew <fullname>-<component>-mtls. 2. If cert-manager itself is broken, the emergency mitigation is a redeploy with --set mtls.enabled=false. Accept the IEC 62443-3-3 SR 4.1 finding and file an incident. 3. Automatic renewal should have fired at two-thirds of the certificate lifetime. Check the cert-manager controller logs for why it did not, or the forced renewal will expire the same way.

DCSHistorianPruneJobFailing

Severity: Critical Condition: No successful run of *-historian-prune-* in over 12h and at least one failed run on record (opt-in rule, default off, enabled via monitoring.prometheusRule.historianDisk.enabled: true).

What happened: The historian prune CronJob runs every 6h. Two missed cycles let TimescaleDB chunks accumulate beyond the 6h target, and on local-path-style PVs the historian PVC begins to outgrow the node root disk. Left unattended this triggers a DiskPressure cascade across the cluster (observed end-to-end in a 2026-04-22 demo-instance incident, with the prevention guidance codified in Historian Disk Pressure).

Actions: 1. Inspect recent failed Jobs: kubectl -n {{ $labels.namespace }} get jobs -l app.kubernetes.io/component=historian-prune --sort-by=.metadata.creationTimestamp. 2. Get the most recent failure's logs: kubectl -n {{ $labels.namespace }} logs job/<name>. Common causes: CNPG primary unavailable, role/grants missing on the prune SQL user, statement timeout under load. 3. Once the underlying cause is fixed, kick a manual prune: kubectl -n {{ $labels.namespace }} create job --from=cronjob/<release>-historian-prune prune-manual-$(date +%s). 4. If the PVC has already grown into the warning band, escalate to the Historian Disk-Pressure runbook before DCSNodeRootDiskHigh fires. On a single-node reference cluster, hack/demo-provision.sh recover is the fast path.

DCSNodeRootDiskHigh

Severity: Critical Condition: A node's root filesystem is >75% used for 15min (opt-in rule, default off, enabled via monitoring.prometheusRule.historianDisk.enabled: true).

What happened: Root-fs usage has crossed the NodeHasDiskPressure-precursor threshold. Kubelet's hard-eviction threshold is around 85%. Cleanup pods that need to run during DiskPressure (local-path-provisioner-cleanup, historian-prune) are typically BestEffort and get evicted first once the taint lands.

Actions: 1. SSH to the node and find the offender: ssh root@<node> 'du -h -d 2 /var/lib/rancher/k3s/storage/ /var/log /var/lib/containerd | sort -h | tail -20'. 2. If the historian PVC is the source, check DCSHistorianPruneJobFailing first. That's the upstream cause. Free space without fixing the prune loop and the alert will re-fire within hours. 3. If logs/containerd is the source, run journalctl --vacuum-size=500M and crictl rmi --prune. 4. On a single-node reference cluster, the codified recovery is hack/demo-provision.sh recover (issue #239), idempotent and safe to re-run mid-flight. 5. Production deployments should not see this rule fire at all (it's opt-in and only meaningful on storage without filesystem-quota enforcement). If it does, the deployment is missing a CSI driver (see production-deployment.md § 1).