Skip to content

ADR 0026: Link redundancy is the standard controller network posture; zero-gap continuous control is deferred to bench measurement

Status: Accepted Date: 2026-07-19 Issue: #962 Amended: 2026-07-19 — added the "What separate switches do and do not buy" limitation under Consequences. The decision is unchanged; the amendment corrects an overstatement in how the separate-switch guidance was originally justified.

Context

ADR 0006 scoped edge-runtime redundancy as hold-then-resume failover to a designated standby, explicitly rejecting hot-standby state replication. Its reasoning was framed around batch and procedural control, where ISA-88 Clause 7.4 Hold/Restart is the sanctioned exception path and a bounded failover gap is immaterial: the phase survives in Phase.Status, outputs hold last value or go to device fail-safe, and recovery is an operator-supervised Restart.

Designing the hardware-validation bench network (#400) surfaced that this rationale under-addresses two things: continuous control at the control-module level, and data integrity for regulated industries.

Most failure modes already degrade gracefully. The behavior is layered, and only the residue matters here:

  • Downstream faults (broker down, historian down, network path to the historian broken) — the unit runtime's store-and-forward buffer (≤10K messages) replays on reconnect. Bounded backlog, no permanent record gap.
  • A single control module's FB network down while the runtime lives — the runtime falls back to publishing raw driver reads at 1 Hz with quality: Raw. The PV stays visible and recorded; only derived and regulated values stop.

Two cases still produce a real gap:

  1. A network-path fault to a live, healthy controller — a pulled cable, a failed NIC, a dead switch port. ADR 0006 has no notion of a network-path fault distinct from a node fault: everything is "the node is dead or partitioned." A single-homed controller whose field link drops is, to the control plane, indistinguishable from a dead node, so it escalates to unit Hold and failover. That is heavy-handed for a cable fault, and it moves a controller that never needed to move.
  2. Full controller or node death — hold-then-resume means a continuous CM PID loop stops and re-establishes on the standby. This is not a modelling assumption; it is what the code does. pkg/fbruntime/blocks/pid.go holds the integrator in process memory (b.integral), and while the block implements back-calculation bumpless transfer for tracking mode, that only covers TRK release within a living process. Runtime state persists to a hostPath mount (cmd/unit-runtime/main.go, #731), and a hostPath does not follow the pod to another node — so on failover the integral resets. During the RTO window outputs hold last value, go to device fail-safe (ADR 0009), or are sequenced by an armed SafeStateChart (ADR 0008), and there is a bounded data gap because no reader is on the field.

For critical continuous parameters in regulated industries (21 CFR Part 11, ALCOA+ completeness), an unbroken control loop and an unbroken record may be required. We do not guarantee zero-gap across a controller death today. The gap is bounded, timestamped, and cause-attributed — a Hold event, a failover AuditRecord, and an explicit Restart bracket it — which makes it defensible as a documented gap. It is still a gap, and the honest posture is to publish it rather than imply otherwise.

One architectural tension constrains any answer. Today's reference architecture spends both NICs of a dual-homed edge IPC on segmentation — one on the L2 supervisory network, one on the L1 field network (docs/reference-architectures.md, Pattern B hardware notes) — and the IEC 62443 posture leans on that separation (SR 5.1 RE(1); delegation register D8 in docs/compliance/iec62443-4-2.md). Link redundancy therefore cannot be had for free by re-purposing the second NIC; it costs ports.

Decision

Link redundancy is the standard controller network posture, and the zero-gap question is deferred to measurement rather than answered by architecture.

  1. Bonded interfaces are the documented default for controller nodes. A Talos-native bond interface in active-backup mode, one bond per network zone, with the two member ports landing on different switches where the deployment has them. Active-backup is chosen over LACP/802.3ad deliberately: it needs no switch-side configuration and no MLAG/stacking to span two switches, which is what actually buys survival of a switch failure rather than just a port failure.

  2. Zones stay physically separate; bonding is applied per zone. The posture does not collapse the supervisory and field zones onto one trunked bond carrying both VLANs. That would trade a link-redundancy win for a segmentation loss and weaken the SR 5.1 RE(1) claim. In NIC budget:

  3. Minimum posture (3 ports): bond the field interface (2 ports), leave the supervisory interface single-homed. The field path is the control-critical one — losing it is what stops regulation; losing the supervisory path only costs control-plane reachability, which Autonomy mode already rides out.

  4. Full posture (4 ports): bond both zones.

  5. Bonding is deployment-layer configuration, not a product feature. Per ADR 0004, node join and machine config belong to the deployment. The product ships the topology, the rationale, and an example Talos stanza; it does not grow a CRD field, a controller, or a health check for bonds. This is deliberate and it is also why the change is cheap: a bond presents a single logical interface, so the FB scan, the drivers, and hostNetwork: true port binding are all unaware of it. No product code changes.

  6. Case (1) becomes a non-event; case (2) is published as a bounded gap. With a bonded field interface, a cable, NIC, or switch-port fault is masked below the runtime and never reaches the control plane as a partition. We claim nothing more: a controller death still produces a hold-then-resume gap in both regulation and record, and that stays documented in docs/ha-failure-modes.md as a real limitation.

  7. Hot-standby FB-state replication stays rejected — pending measurement, not permanently. ADR 0006's rejection stands unchanged for now. The revisit is gated on two inputs that do not exist yet:

  8. Measured RTO on metal for a continuous CM loop, and a demonstration that bonding masks a cable-pull with zero control and data gap (#942, under bench epic #400). Real numbers, not the "tens of seconds" the docs currently assert from kind/CI.
  9. A business input on whether zero-gap continuous control is a contractual requirement in the regulated segments being targeted. How hard to commit is partly market positioning and belongs to Strategy (cndcs-business); Engineering owns the mechanism and its cost, not the requirement.

When those land, the choice is between building state replication and redundant-collector sourcing — a redundant PLC or OPC UA server with historical buffering, backfilled by the historian on reconnect. The latter keeps the record unbroken even when control briefly holds, and is often the pragmatic regulated answer, because the compliance driver is usually the unbroken record rather than bumpless regulation. Either outcome is a new ADR.

Alternatives Considered

  • Build hot-standby FB-state replication now (DeltaV parity) — not chosen, for the same reason ADR 0006 gave plus a new one. The cost is not just a state-sync channel: it has to interact correctly with the lease/epoch fencing that provides the one-writer guarantee, and a standby holding live FB state is precisely the thing that turns a reverse-path partition into a dual-writer incident. Committing that budget before anyone has measured the actual RTO, or confirmed a customer contractually needs bumpless regulation, is building to a guess.
  • Redundant-collector sourcing instead of bonding — not chosen as a substitute, though it remains the leading candidate for the deferred decision. It addresses the record, not the control loop, and it does nothing for case (1): a cable fault would still escalate to Hold and failover. Bonding is cheaper, orthogonal, and fixes a case redundant collectors cannot.
  • One bond trunking both zones (2 ports, VLAN-tagged) — rejected. It is the only posture that fits today's dual-NIC IPC BOM without new hardware, which makes it tempting, but it puts supervisory and field traffic on shared physical media and reduces the zone boundary to a VLAN tag. The compliance register explicitly delegates "dedicated NICs/VLANs for field buses"; silently downgrading that to "VLANs" to save two ports is not a trade worth making.
  • LACP / 802.3ad — rejected as the default. It gives aggregate bandwidth we do not need (the bandwidth envelope in docs/network-requirements.md is nowhere near a 1 GbE link), requires matching switch-side configuration, and needs MLAG or stacked switches to survive a switch loss. Active-backup survives more failure modes with less coupling to the switch vendor. Deployments that already run MLAG everywhere may use LACP; it is a local choice, not the guidance.
  • Detect the link fault in software instead — teaching the runtime to distinguish "my field link is down" from "I am partitioned," and suppress the escalation. Rejected: it adds product complexity and a new failure-classification heuristic to solve a problem the physical layer solves outright, and it would still leave the control loop dead for the duration of the fault. Bonding keeps regulating.
  • Status quo — no posture, per-deployment choice — rejected. Silence is what produced the gap: nothing in docs/ mentions bonding, and the reference architecture's only nod to NIC redundancy is an adjective in a hardware shortlist with no configuration behind it. A deployment following our guidance literally builds a single-homed controller.

Consequences

  • The hardware BOM changes. Edge IPCs need 3 ports minimum, 4 for the full posture; docs/reference-architectures.md currently specifies dual-NIC boxes, and the bench epic (#400) shortlists "2× industrial PC (fanless, dual-NIC)". Bench hardware must be re-specified before purchase, or the bench validates a posture we do not recommend. This is the one consequence with a procurement deadline attached.
  • Docs that move: docs/network-requirements.md gains a physical-layer section (it has none today — the doc is entirely logical); the docs/ha-failure-modes.md failure table gains an explicit network-path-fault row and its HA-mechanism table gains link redundancy; docs/reference-architectures.md Pattern B states the posture normatively; docs/production-deployment.md gains a link-redundancy pre-deployment check.
  • What separate switches do and do not buy (amended 2026-07-19). The guidance above says to land a bond's members on different switches, and that is right — but it must not be read as "bonding plus two switches survives a switch failure." For the field zone it largely does not, because remote I/O is single-homed: a Wago coupler, an ADAM drop, or a PLC each has one port on one switch. When that switch dies the controller's surviving bond member has nothing left to reach, so controller-side redundancy buys nothing. Splitting the I/O across two switches converts a total field outage into a partial one, which is worth something but is not redundancy.

Genuine field-network switch redundancy needs the far end to be redundant too — a ring topology (MRP, RSTP, or DLR) or dual-homed field devices. That is a materially larger design than controller bonding, most remote-I/O couplers in our reference hardware cannot do it, and it is out of scope for this ADR. If a deployment needs it, it is a new ADR.

Where separate switches do pay off is the supervisory zone: the other end is the control plane, which is already multi-node and can be split across both switches, so a bonded supervisory interface plus two switches genuinely survives losing one. Prioritize accordingly when ports are scarce.

None of this weakens the case for bonding itself. Case (1) — a cable, NIC, or switch-port fault on a live controller — is the failure this ADR exists to eliminate, and bonding eliminates it on a single switch. A second switch is an increment on top, not a precondition. - The failover runbook's fencing step gets sharper, not looser. "Disconnect the node from the field network" is the load-bearing manual fencing action, and on a bonded controller that is now two cables. A half-fenced node is a dual-writer hazard, so the runbook must say so explicitly. - Compliance posture: the IEC 62443 FR 7 availability targets in docs/ha-failure-modes.md are today justified purely by node-level mechanisms; link redundancy is an additional basis, but the targets are not revised until #942 produces measured numbers. SR 5.1 RE(1) and delegation D8 are strengthened (per-zone bonds, not weakened by trunking). - Marketing claim boundary is unchanged and now explicit: bonded controller networking plus hold-then-resume failover. Never "bumpless," never "zero-gap," never redundant-pair parity with DeltaV — until and unless the deferred decision is made and shipped. - No product code changes and no CRD surface. Reversibility is therefore high: this is deployment guidance, and withdrawing it breaks no contract. The deferred hot-standby question is where the irreversible commitment lives, which is exactly why it is deferred. - Followups: #942 gains the bonded cable-pull drill and a continuous-loop RTO measurement; the Strategy input on contractual zero-gap requirements; a superseding ADR if hot-standby or redundant-collector sourcing is adopted.