ADR 0006: Edge-runtime redundancy is hold-then-resume failover to a designated standby node — no hot-standby state replication¶

Status: Accepted Date: 2026-06-12 Issue: #566

Context¶

The competitive baseline for a diligence reader is DeltaV, where controller redundancy is a standard orderable product feature: a controller can have a dedicated 1:1 redundant partner receiving continuous state synchronization over a local redundancy link (the documented topologies mount the pair on a shared carrier assembly), giving bumpless switchover. The pairing is per controller, not per unit — DeltaV imposes no controller:unit cardinality, and a controller commonly executes control for many units. Our product today has no controller-level redundancy feature — a failed edge IPC means that unit's control loop is down until the hardware comes back or is replaced (docs/ha-failure-modes.md node-failure row: "manual intervention"). Issue #566 asks Engineering to scope what failover should look like.

What exists today, precisely:

One runtime owner per physical unit. The unit runtime is a bare pod (<unit>-runtime) created by the Unit reconciler (internal/controller/physical/unit_pod.go), pinned to a single edge node via UnitSpec.NodeSelector (convention: dcs.io/device=<unit>), hostNetwork: true. There is no lease, leader election, or fencing — the node pin is the one-writer guarantee.
Node:unit cardinality is not enforced. Like other control systems, the product is unopinionated about how many units a controller node serves: runtime ports are per-unit configurable (UnitSpec.RuntimeHealthPort/RuntimeGRPCPort), so several runtimes can share a node, though the colliding port defaults make 1:1 the default path. The recommended practice is 1:1 controller:unit — it keeps the failure blast radius to one unit and lets someone unfamiliar with the control system power down one piece of equipment without taking out an unrelated one.
Restart-replay autonomy. The last deployed FB program is persisted to hostPath (/var/lib/dcs/runtime/last-program.json, internal/adapter/persist.go) and replayed on pod restart; during a control-plane partition the runtime keeps regulating autonomously.
Detection and safe-state already exist. A runtime outage is absorbed for a 60 s grace window, after which the phase self-Holds (internal/controller/procedural/phase_controller.go); the unit watchdog issues an ISA-88 Hold when all drivers are down during an active batch (unit_pod.go); Held phases are exempt from the 15 min stuck-phase force-abort. A dead edge node mid-batch therefore lands in Held, not Aborted.
Phase logic does not live at the edge. SFC/phase execution runs in the procedural operator (phase_controller.go instantiates the engine), which is already HA via leader election. Mid-chart execution state — active steps, completed steps, variable values, fired transitions — is published to Phase.Status.SFCStatus every second and the engine resumes mid-chart from it (sfc.WithRestoredState). The edge node holds only the FB scan loop and the I/O driver connections.

Three architectural facts shape the option space:

The controller↔I/O binding is a network connection. Our I/O is remote Ethernet I/O (Modbus TCP, OPC UA, EtherNet/IP) — a TCP client connection from the runtime, with no physical attachment between the controller node and the I/O hardware. Moving a unit's runtime to different hardware is therefore a software-only operation: any enrolled node with reach into the unit's field network can take the role, with no rewiring and no physical adjacency requirement on the standby.
Partition autonomy and automatic failover are mutually exclusive per unit. Kubernetes cannot distinguish a dead node from a partitioned one, and our runtime is designed to keep controlling while partitioned. Auto-failover without fencing means two writers on the same field devices — on Modbus, for example, there is no session exclusivity at all; last write wins, undetected.
ISA-88 already defines the recovery shape. Part 1 Clause 7.4 lists control equipment malfunction as a canonical exception event; the standard's response is HOLD (bring equipment to a known safe state) and RESTART with recipe-defined restarting logic. The standard does not ask for invisible failover, and batch processes tolerate Hold/Restart by design.

Decision¶

Edge-runtime redundancy is hold-then-resume failover: re-binding a unit's runtime to a deployment-designated standby node, with the one-writer guarantee protected by an explicit per-unit availability policy. The product builds the re-binding and fencing mechanism; it does not replicate runtime state to a standby and does not claim bumpless switchover. Which nodes are eligible standbys — and whether a standby is dedicated to one primary or shared — is deployment configuration, not product opinion (product-vs-deployment split).

The scoped mechanism:

Failover targets — UnitSpec.availability names the eligible standby node(s) (explicit list and/or label selector). Targets are enrolled nodes (ADR 0004 adoption contract) with field-network reach to the unit's I/O. The recommended pattern is a dedicated standby per primary controller node: a deterministic failover target is easier to qualify, and it preserves the 1:1 controller:unit practice. A spare set shared across several units is equally supported as a cost option; its consequences are the deployment's to accept consciously — after multiple failures, units can co-locate on one node. The re-bind operation surfaces a co-location guard: it warns (or refuses, per policy) when the target already hosts another unit's runtime.
Unit re-binding as a first-class operation — dcs unit failover <unit> --to-node <node> (and a UI equivalent): the physical operator fences the old binding, rebinds the runtime pod to the target node, and redeploys the control program from the control-plane source of truth (CRDs — no hostPath migration); drivers reconnect to the remote I/O. The action is audit-logged.
Hold-then-resume semantics — the running phase survives in the control plane; the unit is Held during outage and re-bind (the existing watchdog/self-hold path); recovery is an ISA-88 Restart whose recipe-defined restarting logic re-establishes process conditions. Field outputs hold last value (or device fail-safe) during the gap, exactly as for a pod restart today.
One-writer via availability policy — UnitSpec.availability.mode:
Autonomy (default; today's behavior): on partition the edge keeps controlling. Failover is manual only and requires explicit operator confirmation that the old node is fenced (powered off or disconnected from the field network).
Failover (opt-in): the runtime holds a control lease; on lease expiry it self-fences (stops FB output writes, disconnects drivers), and the operator may auto-rebind to a standby after lease timeout plus margin. The lease extends only on confirmed two-way liveness — a renewal carries the operator's acknowledged-renewal age, and the runtime self-fences once that age reaches the lease duration even while renewal POSTs keep arriving, so a reverse-path (asymmetric) partition cannot leave the runtime driving while a standby is re-bound (#579). An epoch/fencing token in the runtime API and write path is defense-in-depth against stale owners.

The two modes are mutually exclusive by construction — a unit cannot have both partition autonomy and automatic failover. The product ships the mechanism and the safe default; the deployment chooses the policy per unit.

Alternatives Considered¶

Hot-standby state replication with bumpless switchover (DeltaV-style) — a standby runtime continuously receiving FB and variable state from the active one, taking over sub-second. Not chosen: continuous state-sync machinery, a switchover protocol, and a fencing story would be built for a guarantee ISA-88 does not require for batch — Clause 7.4 Hold/Restart is the standard exception path, and batch phases tolerate seconds of held outputs. Note the rejection is of state replication, not of dedicated standby hardware — a dedicated standby per controller is the recommended topology under this ADR; it just stays cold until re-bind. Revisit only if a design partner has a unit where held outputs are process-destructive; that is a new ADR.
Plain Kubernetes rescheduling (drop the node pin, let the scheduler move the pod) — not chosen: it breaks the one-writer guarantee on partition. Kubernetes cannot tell dead from partitioned, and the partitioned node's runtime keeps controlling by design, so unfenced rescheduling creates dual writers on live field devices.
Status quo (restart-replay only) — not chosen as the terminal posture: detection and Hold already work, but a dead IPC strands the unit until hardware replacement, re-binding is undocumented manual surgery, and the diligence comparison against DeltaV fails.
Protocol-level exclusivity as the fencing primitive — not chosen as the universal guarantee, because support varies by protocol. Modbus has no session-ownership concept at all: any client may write, last write wins. OPC UA locking exists only as an optional companion-spec facility (OPC UA DI LockingServices), so it is server-dependent. EtherNet/IP does have a real mechanism — a CIP output assembly accepts a single exclusive-owner connection and rejects a second with an ownership-conflict error (extended status 0x0106) — but ownership is freed after a connection timeout (RPI × timeout multiplier), so under partition it is liveness-bounded rather than absolute, and a recovering stale owner races the new one for re-ownership. The one-writer guarantee therefore lives above the protocol layer (availability policy + lease/epoch fencing); EtherNet/IP exclusive-owner connections are used as defense-in-depth where the device supports them.

Consequences¶

API surfaces that move: UnitSpec gains availability (mode, lease duration, failover targets); the adapter API and FB write path carry a lease/epoch token; the physical operator gains fence, re-bind, and co-location-guard logic; dcs gains a unit failover verb; UI placement to be confirmed before implementation (no inline controls on HMI cards). Alarm + AuditRecord events cover fence, re-bind, and lease loss (21 CFR Part 11).
The running phase survives a failover. Phase/SFC state — down to active steps, variable values, and fired transitions — lives in Phase.Status in etcd, not on the edge node; the batch record shows the exception (Hold, Restart) rather than a vanished phase.
Hardware guidance changes: reference architectures stop calling the edge node an unmitigated single point of failure and instead document the failover-target patterns — dedicated standby per controller (recommended) vs. shared spares (cost option, co-location trade-off). docs/ha-failure-modes.md, docs/reference-architectures.md (Pattern B), and docs/production-deployment.md update when the mechanism ships; IEC 62443 FR 7 availability targets for the unit runtime can rise accordingly.
Marketing posture: until this ships, the accurate claim remains autonomy + restart-replay + supervised Hold. Once shipped, the claim is controller failover (hold-then-resume) to designated standby hardware — never "bumpless redundancy" or redundant-pair parity with DeltaV.
Default behavior is unchanged: Autonomy mode preserves today's semantics exactly; everything else is additive and opt-in.
Followups: the implementation epic (#574). Warm-standby pre-staging (image pre-pull, a pre-created fenced pod on the designated standby) is a compatible later optimization that shortens failover time without state replication.
Reversibility: high before implementation (this is a posture); moderate after — the availability policy field becomes a public CRD contract, and withdrawing automatic failover would be a customer-visible regression requiring a superseding ADR.

Amendment (2026-08-07, #1320): the automatic path issues the Hold itself¶

Hold-then-resume as implemented waited, before re-binding a Running unit, for "the crash detector or watchdog" to issue the Hold. Both of those live in the pod-lifecycle pass of the same reconcile, downstream of the failover check — so the wait starved the only pass that could end it, and an automatic failover of a unit Running a batch parked forever (found live: eight minutes on a dead node with no Hold, no alarm, no re-bind).

The decision is unchanged; the sequencing within it is corrected. A lease observed Expired means the runtime has self-fenced, which is the crash evidence itself, so the automatic path now issues the ISA-88 Hold directly — through the same code path, alarm, and RuntimeCrashDetected condition as the crash detector — once the expiry outlasts the ADR 0008 hold bound (½ lease, clamped 5–30s). Inside the bound it waits, so a runtime that recovers and renews its lease rides through with no Hold, preserving the crash detector's grace semantics with a lease-sized budget. The re-bind still waits for the safety margin (margin ≥ hold bound by construction), and the lease expiry is still reported (status, alarm, audit record) before any Hold or re-bind. The manual path's refusal of Running units is unchanged.