Skip to content

ADR 0041: Redundant-collector sourcing answers the controller-failover gap; hot-standby FB-state replication is deferred behind a written requirement

Status: Accepted Date: 2026-08-06 Issue: #1018 (founder ruling 2026-08-06; supersedes the deferred half of #962)

Context

ADR 0026 settled the near-term half of #962 — bonded interfaces as the standard controller network posture — and deliberately deferred the harder question: what to build for continuous control and the data record across a controller death. Across that failure, a continuous control-module PID loop stops and re-establishes on the designated standby. The integrator lives in process memory (pkg/fbruntime/blocks/pid.go), runtime state persists to a hostPath that does not follow the pod (#731), and the loop restarts from configured initial conditions. The result is a bounded control gap — outputs hold last value, go to device fail-safe (ADR 0009), or are sequenced by an armed SafeStateChart (ADR 0008) — and a bounded data gap, because no reader is on the field during the RTO window. The gap is bounded, timestamped, and cause-attributed, and it is published in docs/ha-failure-modes.md rather than implied away.

#1018 carried the deferred decision between two candidate answers:

  • A. Hot-standby FB-state replication — bumpless continuous takeover, DeltaV parity. Expensive in a specific way: a standby holding live FB state interacts with the lease/epoch fencing that provides the one-writer guarantee, and it is precisely what turns a reverse-path partition into a dual-writer incident.
  • B. Redundant-collector sourcing — a redundant PLC or OPC UA server with historical buffering, backfilled by the historian on reconnect. It keeps the record whole even when control briefly holds, and it is a deployment-topology choice rather than product code.

The decision was gated on two inputs. The Strategy input landed: cndcs-business#30 was answered on 2026-08-04 (decisions/zero-gap-control-requirement.md), and its recommendation is exactly B. Its findings carry the weight here:

  • No regulation requires control-loop continuity. The binding texts are about records — 21 CFR 211.188, 211.68(b), Part 11 §11.10(b), Annex 11. ALCOA+ Consistent forbids unexplained gaps, and an explained gap is contemplated by the principle itself. The obligation is therefore to make the gap self-describing inside the batch record, not to eliminate it. A gap attributed only in a system log a QA reviewer never opens is still unexplained as far as the batch record is concerned.
  • The decisive finding is architectural rather than numerical. For every prospect in the August sequence, we are the orchestration layer above OEM control islands. The PID loops live in the skid vendors' controllers, and we do not replace them, so a failover in our tier cannot stop a loop we were never closing. That holds at any gap magnitude.
  • "Contractual" arrives through the URS chain, not through a regulator. A URS clause is a document we can ask for, and the discovery sheet now asks for it directly.

The second gate — measured bench numbers from #942 drills 9 and 10 — remains open behind procurement, and the ruling holds that it no longer gates. A bench number would matter only if the decision turned on how large the gap measures. The architectural finding makes the answer the same at any magnitude, so waiting for it is delay rather than caution. B is also cheap and reversible, which is exactly the kind of choice that tolerates deciding before the last input lands.

Decision

Redundant-collector sourcing is the answer to continuity across a controller failure. Hot-standby FB-state replication is deferred — not rejected forever, but parked behind named reversal triggers. Founder ruling on #1018, 2026-08-06. This supersedes the deferred half of #962 and completes the open item ADR 0026 § Decision (5) left behind.

  1. Redundant-collector sourcing is the recommended topology for an unbroken record. A redundant PLC or OPC UA server with historical buffering covers the data gap; the historian backfills on reconnect. It is deployment topology, not product code — the product ships the guidance, and no CRD field or controller grows from this decision.

  2. Hot-standby FB-state replication stays deferred. ADR 0006's rejection stands. The expensive option would trade a proven safety property — the one-writer guarantee that lease/epoch fencing provides — for a capability no committed prospect requires, and that trade is bad independent of the engineering cost.

  3. The reversal triggers are explicit, adopted from Strategy's answer:

  4. Any URS or RFP specifying controller redundancy. That is the mechanism by which a DeltaV capability becomes an "industry requirement" with no regulator involved. It is negotiable at requirements review and ruinous at OQ, and a URS clause is a document we can ask for.
  5. A continuous-manufacturing prospect (Continuus, Bright Path, Phlow, ODP) entering a real conversation. That is the only segment where we would close the loops ourselves and where the question is live.

  6. #942 reporting a gap materially larger than ADR 0026 assumes.

  7. The actionable engineering falls on the record, not the loop. Per the ALCOA+ reading, the failover gap must be self-describing inside the batch record rather than explained only in a system log. That work is #1308, and it is separate from the collector topology.

  8. Positioning: never claim bumpless parity. Redundancy is the incumbent's chosen ground, and asserting equivalence invites the one comparison we lose. We keep publishing the quantified gap in docs/ha-failure-modes.md, we quantify it further once #942 unblocks, and we put it in the diligence pack deliberately. In a room whose job is data integrity, a stated limitation is a better artifact than an unfalsifiable claim about a rare event.

  9. The IEC 62443 FR 7 availability targets stay deferred and ride with #942, not with this decision. ADR 0026 parked them until measured numbers exist, and nothing here settles them — the next reader should not treat them as revised by this ADR.

Alternatives Considered

  • Build hot-standby FB-state replication now (Option A). Rejected on three grounds. No committed prospect can require it, because we do not close their loops. It trades the one-writer safety property for bumpless takeover, which is a bad trade at any price. And its cost is continuous — state sync plus a correctness argument against the fencing model — where B's cost is a topology diagram.
  • Wait for the #942 bench numbers before deciding. Rejected by the ruling itself. The drills sit behind bench procurement, and the number they would produce cannot change an answer that rests on which loops we close. Drill 10 stays valuable as verification and as a diligence artifact, no longer as a gate.
  • Decide B but market it as redundancy parity. Rejected as positioning. DeltaV has shipped redundant pairs for twenty years, and the buyer's automation lead has run them. The quantified, cause-attributed gap is the stronger artifact in a data-integrity room.
  • Treat the published gap as sufficient and do nothing further. Rejected. ALCOA+ Consistent is satisfied by an explained gap, and today the explanation lives in the audit trail rather than in the batch record a QA reviewer actually reads. Closing that distance is the concrete obligation this decision produces (#1308).

Consequences

  • #1308 is the follow-up obligation: make the failover gap self-describing inside the batch record, bracketed the way ADR 0026 describes — a Hold event, a failover AuditRecord, and an explicit Restart — but carried where the QA reviewer looks. That is engineering work in the product, unlike the collector topology itself.
  • docs/ha-failure-modes.md keeps the limitation published and gains the bench numbers when #942 lands. The redundant-collector guidance there is now the decided posture rather than a suggestion raised during design.
  • The marketing claim boundary from ADR 0026 is unchanged: never "bumpless", never "zero-gap", never redundant-pair parity with DeltaV.
  • The FR 7 availability targets in docs/ha-failure-modes.md are not revised. They ride with #942, per ADR 0026 and restated here deliberately.
  • No product code changes and no CRD surface, so reversibility stays high. The deferred hot-standby question remains where the irreversible commitment lives, and the reversal triggers above are the conditions under which it reopens.
  • One caveat is inherited from Strategy and recorded here: zero discovery calls have been held, and no prospect has sent a URS. The finding rests on regulatory texts, the prospect dossier, and what our architecture touches. That settles which loops we close; it does not close the positioning question permanently. Strategy reports back after the first three discovery calls, or immediately if a URS clause turns up.