Skip to content

ADR 0008: Edge-local holding logic — partition-triggered safe-state sequencing at the unit runtime

Status: Accepted Date: 2026-06-16 Issue: #578 Related: ADR 0006 (refines its partition-gap context), ADR 0007 (extends its three-layer protection model), #566 (edge redundancy scoping)

Context

ADR 0007 established a three-layer protection model and was explicit about which layer survives a control-plane partition:

Layer Where it runs Survives partition Role
Device interlock (DO/AO ILCK) FB scan, edge Yes Force one output to one static safe value while a trip is true
Phase SFC guard (interlock: true) Procedural operator, control plane No Procedural safe-state sequencing — ordered multi-output response (vent first, then cut heat), parking the procedure in a resumable held step
AlarmDefinition.exceptionAction Alarm controller, control plane No Annunciation, audit, batch-level Hold/Stop/Abort

The gap #578 names is the middle row. A device interlock forces a single output to a single value — it cannot sequence. The only layer that can run a deliberate, ordered safe-state response (the thing a process actually needs: close the charge valve, then stop the agitator, then open the vent) is the phase SFC, and that executes in the procedural operator on the control plane (internal/controller/procedural/phase_controller.go instantiates the engine; each scan's READ/WRITE builtins cross HTTP to the runtime via pkg/stbridge). On a control-plane partition:

  • the edge FB scan keeps regulating to whatever setpoints were last commanded (the autonomy claim — accurate, ADR 0006), but
  • the phase SFC stalls mid-step at an arbitrary point, so the equipment is frozen wherever the sequence happened to be — mid-charge with the charge valve held open is frozen, not safe, and
  • no sequenced holding response can execute at the edge, because that logic lives in the control plane.

ADR 0006 made this precise and accepted it as the v1 posture: "Field outputs hold last value (or device fail-safe) during the gap, exactly as for a pod restart today" (ADR 0006 §Decision.3). The compliance traceability says the same — docs/compliance/isa88.md Clause 7.4 row: "Held outputs hold last value / device fail-safe during the gap." Frozen, not driven to a deliberate safe state.

Competitive + standards baseline (researched 2026-06-12, #578):

  • DeltaV's shipped default on Batch Executive ↔ controller comm loss is the phase placed in Held via a watchdog, with Holding logic executing locally in the controller with full I/O access — deliberate safe-state actions, not frozen outputs (Emerson Batch Executive PDS). Rockwell FactoryTalk Batch documents the same watchdog → HELD philosophy. Note this is the non-redundant comm-loss path; it is orthogonal to controller redundancy (ADR 0006), which is a separate feature.
  • ISA-88 Part 1 deliberately does not mandate execution placement (Clause 6.6.3: recipe/equipment separation is logical, physical separation optional; Clause 6.6.4: equipment procedural element internals are out of scope). What it expects is the Clause 7.4 exception response: malfunction → Hold to a known safe state → Restart with recipe-defined restarting logic.

So the gap is not "move the phase engine to the edge" — that would put mid-phase durable state on the node ADR 0006 deliberately allows to die, and complicate the 21 CFR Part 11 audit trail. The gap is narrow and specific: the edge has no locally executable, sequenced holding response.

Five constraints from the existing code shape the option space:

  1. One writer owns the drivers. ADR 0007 rejected a separate watchdog goroutine forcing driver writes precisely because "two writers to one output address race each other; the FB scan already owns output ordering and the one-writer guarantee." Any edge holding response must respect this — it cannot become a second driver writer.
  2. SFC encapsulation rule. All phase state logic must be SFC charts; no bare flat ST (project convention). A sequenced hold response is phase state logic.
  3. The SFC engine is already standalone. pkg/sfc imports only the CRD types, the ST interpreter, and a clock; it accepts an injectable StepHandler and supports WithRestoredState. It does not today have a driver-bound execution path at the edge — its READ/WRITE resolve over HTTP in the control plane.
  4. Failover self-fence and a local hold would fight. ADR 0006's Failover mode self-fences on lease expiry — it stops FB output writes so a standby can take over (internal/adapter/lease.go fencedDriver drops writes). A holding response that writes during that window contradicts the fence and risks dual writers against the standby.
  5. Durable Held state lives in etcd, and the phase self-holds anyway. Phase state lives in Phase.Status (ADR 0006); the procedural operator's existing 60 s grace → self-Hold path (phase_controller.go) independently drives the phase to Held when it sees the runtime unreachable. Whatever the edge does must converge with that, not diverge from it.

Decision

The unit runtime gains an armed local-hold program: an SFC chart executed by an SFC engine embedded in the runtime when the edge's control-plane heartbeat watchdog fires. The hold chart drives a deliberate, sequenced safe state by writing the unit's own control-module tag space — the FB scan remains the sole I/O writer, so no new write path and no dual-writer is introduced. This makes ADR 0007's procedural safe-state layer partition-tolerant. Durable Held state continues to live in the control plane (ADR 0006); recovery is a normal ISA-88 Restart (Clause 7.4) — no silent resume. This refines ADR 0006's partition gap; it does not supersede it.

The scoped mechanism:

  1. What is armed. Two sources, in priority order:
  2. Per-phase (preferred): at phase start the procedural operator downloads the active phase's HoldingChart to the runtime as the currently-armed local-hold program — the same chart that runs the ISA-88 Holding state in the control plane, now also staged at the edge. This reproduces the DeltaV default (phase-specific holding logic) and is correct precisely because a mid-charge partition arms the charge phase's holding logic.
  3. Unit baseline (fallback): a new UnitSpec.safeStateChart (SFC), deployed with the unit's control program and always armed when no phase is active. This closes ADR 0007's idle-unit / manual-mode coverage gap for the sequenced case, the same way device interlocks closed it for the single-output case.

Edge-armable constraint. A chart is armable at the edge only if it READ/WRITEs the unit's own control-module tag space (no cross-unit, no control-plane-only data). A chart referencing out-of-edge-scope data cannot be staged; the runtime falls back to the unit baseline and the operator surfaces a warning. A lint/validation enforces this.

  1. Execution model. The runtime embeds the pkg/sfc engine with a StepHandler/tag binding whose READ/WRITE resolve directly against the local control-module tag space (the FB network's variable bindings) rather than over HTTP. The hold chart manipulates FB inputs (setpoints, valve commands) exactly as the phase SFC does today via pkg/stbridge; the FB scan regulates to them and remains the only writer to drivers. Device interlocks (ADR 0007) still sit underneath and can override even the hold chart's commanded outputs. The SFC encapsulation rule is honored — the hold response is an SFC chart.

  2. Trigger contract — a unified control-plane heartbeat. The local-hold trigger is "no control-plane heartbeat within the timeout," evaluated by a purely local timer at the edge (no network reach required to decide):

  3. In Failover mode the heartbeat is the existing lease renew (/api/v1/lease/renew, internal/adapter/lease.go) — reuse it, do not add a second watchdog.
  4. In Autonomy mode there is no edge watchdog today (the lease guard is inert). Add a lightweight liveness heartbeat the procedural/physical operator issues each reconcile (carrying "phase X is live") — no fencing, no lease semantics, just a timestamp the edge watchdog consumes.

  5. Timeout is the unit's availability.leaseDurationSeconds in Failover, and a new availability.holdGraceSeconds (default mirrors the control plane's 60 s phase self-hold grace) in Autonomy.

  6. Mode interaction (resolves constraint 4).

  7. Autonomy: on heartbeat loss past the grace window, the edge runs the armed hold chart to drive a deliberate safe state, then keeps holding (the FB scan continues regulating to the safe-state setpoints the chart established). No fence — the edge is the legitimate single writer.

  8. Failover: on lease expiry the edge runs the armed hold chart as a bounded safe-state sequence, then self-fences (stops writes) so a standby can take over cleanly. The standby re-bind already happens only after lease-expiry + margin (ADR 0006); the margin must be ≥ the hold sequence's bounded duration, so the partitioned node finishes its safe-state actions and fences before the standby writes — no dual writer. A node that is truly dead (not merely partitioned) cannot run the hold; that case falls back to ADR 0006's existing behavior — outputs at device fail-safe / last value until the standby re-binds and Restart drives them. Local-hold strictly improves the alive-but-partitioned case; it does not regress the dead-node case.

  9. Reconciliation on reconnect. The runtime reports, via its status (MQTT status + a gRPC/HTTP field), that it self-held, which program it ran, and when. On reconnect the operator reads this and reflects Held on the Phase/Unit state machines. This converges with the control plane's own 60 s self-hold path (constraint 5): during a partition both sides independently move toward Held — etcd marks the phase Held, the edge holds outputs — and on reconnect they agree. Recovery requires an explicit ISA-88 Restart (Clause 7.4); there is no silent resume.

  10. Audit at the edge. Local-hold executes while the control-plane audit path is unreachable. The runtime emits hold-lifecycle events (armed, triggered, each safe-state action, self-fenced) onto the existing MQTT store-and-forward queue (internal/adapter/queue, /var/lib/dcs/runtime/queue/), replayed on reconnect and materialized into AuditRecord CRs by the audit bridge. This preserves the "audit must not block the control loop" rule and reuses the historian store-and-forward precedent rather than giving the runtime direct apiserver write access.

Alternatives Considered

  • Download a compiled safe-state FB network instead of an SFC chart (engine already at the edge — issue option 2). Rejected as the primary: a flat FB network cannot sequence (vent-then-cut-heat), which is the entire gap — ADR 0007 keeps sequencing in the SFC layer for exactly this reason, and the FB layer already provides the static single-output safe state via device interlocks. Re-encoding ordered sequences as FB logic would duplicate the SFC engine's job and violate the SFC encapsulation rule.
  • Move the whole phase/SFC engine to the edge. Rejected: it puts durable mid-phase state (active steps, fired transitions, variable values) on the node ADR 0006 deliberately allows to die, breaks the etcd-resident phase state that lets a running phase survive failover, and complicates the Part 11 audit trail. The fix is a fallback safe-state responder at the edge, not relocating sequencing ownership.
  • Unit-level static safe-state chart only (no per-phase arming). Rejected as the sole answer: it loses phase-specific safe states (mid-charge wants a different response than mid-heat), which is the DeltaV parity the issue is chasing. Kept as the baseline for the idle/no-phase case.
  • Status quo — frozen outputs + device interlocks only (ADR 0006 v1 posture). Rejected as terminal: a device interlock cannot sequence, and frozen-at-last-value is not a deliberate safe state. The diligence comparison against the DeltaV/Rockwell shipped default fails on exactly this point.
  • A separate watchdog goroutine forcing driver writes at the edge. Rejected for the same reason ADR 0007 rejected it for interlocks: two writers to one address race; the FB scan owns the one-writer guarantee. The embedded SFC engine writes the FB variable space, never the drivers directly.

Consequences

  • API / code surfaces that move:
  • UnitSpec gains safeStateChart (SFC, the unit baseline) and availability.holdGraceSeconds (Autonomy trigger timeout).
  • The unit runtime embeds the pkg/sfc engine with a driver-bound tag handler (internal/adapter/, cmd/unit-runtime/); a new edge watchdog consumes the heartbeat; new self-held status field on the adapter API.
  • The adapter API gains an arm-hold-program endpoint (download the active phase's HoldingChart) and, for Autonomy, a liveness heartbeat endpoint; Failover reuses /api/v1/lease/renew.
  • The procedural operator downloads the phase HoldingChart at phase start and reconciles edge-self-held → Held on reconnect (phase_controller.go).
  • The audit bridge consumes edge hold-lifecycle events from the MQTT queue and materializes AuditRecord CRs.
  • New metrics: hold armed/triggered counters, hold-sequence duration, edge-self-held gauge.
  • Compliance:
  • docs/compliance/isa88.md Clause 7.4 row (control equipment malfunction → Hold → Restart) is refined: the partition response becomes a deliberate sequenced safe state, not frozen-at-last-value. The "Held outputs hold last value" caveat narrows to the dead-node / no-armed-chart case. docs/library/alarms-and-interlocks.md's three-layer table gains a note that the procedural safe-state layer is now partition-tolerant at the edge.
  • This is a candidate execution vehicle for the existing ProcessException safeState.structuredText row (currently "Partial — validated and logged, not executed against equipment"): an armed unit baseline chart is where a process-exception safe state could actually run. Noted for the implementation epic, not decided here.
  • 21 CFR Part 11: the edge-buffered, replayed audit trail records the hold actions taken while the control plane was unreachable.
  • Docs that update when this ships: docs/ha-failure-modes.md (partition row: deliberate safe-state, not frozen), docs/architecture.md (autonomy section), docs/adr/0007 three-layer narrative cross-link.
  • Marketing posture: now that this has shipped (epic #597, merged), the accurate partition claim is "watchdog-triggered local holding logic drives a deliberate safe state at the edge during a control-plane partition" — the DeltaV/Rockwell default reproduced. The pre-ship claim ("autonomy + restart-replay + supervised Hold, outputs frozen at last value") now applies only to the dead-node / no-armed-chart case. Never claim bumpless redundancy (that remains explicitly rejected, ADR 0006).
  • Default behavior unchanged: with no safeStateChart and no phase HoldingChart, the edge behaves exactly as today (frozen outputs). The feature is additive and opt-in per unit/phase.
  • Reversibility: moderate now that this has shipped — UnitSpec.safeStateChart is a public CRD contract and the heartbeat is a runtime API surface. (It was high before implementation, when this was still a design posture.)
  • Follow-ups (implementation sub-issues): embed driver-bound SFC engine at the edge; UnitSpec.safeStateChart + holdGraceSeconds API; Autonomy liveness heartbeat + edge watchdog; per-phase HoldingChart arming at phase start; Failover hold-then-fence ordering + margin guard; edge-self-held status + reconnect reconciliation to Held; edge audit buffering of hold events; edge-armable chart validation/lint; docs + compliance updates.

Amendment (2026-08-10, #1406): edge-armable covers capabilities, not only data

The edge-armable constraint in §Decision.1 was written about data — the chart may address only the unit's own control-module tag space. It said nothing about capabilities, and the two fail the same way. Eleven of the ST dialect's builtins cannot execute at the edge at all, because each one needs something the partition has taken away: an operator (PROMPT, PROMPT_CHOICE, PROMPT_VALUE), the gateway an external system delivers a measurement through (AWAIT_RESULT, ADR 0055), the apiserver (MODE), the phase state machine the control plane drives (COMMAND), control-plane state about where the action chart stopped (STEP_ACTIVE), or a network session to another machine (CALL_SERVICE and the three MTP builtins, ADR 0045).

A chart calling one of them passed every gate, ran correctly on the control-plane path every time it was exercised, and errored mid-sequence only during the partition it was armed for. That is the worst moment to discover it, and it was discovered as an ST runtime error inside a hold, with no operator and no control plane to report it to.

The decision is unchanged and its scope is stated fully. Such a chart is not armable, which is the outcome §Decision.1 already defines: the runtime refuses to stage it, the phase falls back to the unit baseline safe-state chart, and the operator is told — while the control plane is still up to be told through. Authoring one remains legitimate, because a holding chart that prompts the operator, or commands a PEA service to hold, is exactly right for the ordinary control-plane hold it will normally run in. What changes is that its edge cover is one posture coarser, and that fact now arrives at arming time rather than during the partition.

ValidateEdgeArmableBuiltins (api/procedural/v1alpha1/edge_armable.go) is the static check, the runtime applies it when a chart is staged (internal/adapter/hold.go), and make lint-edge-armable enforces it over the example corpus alongside the tag-space half. Because the two halves must not drift, TestEdgeUnavailableBuiltinsMatchHoldRuntime runs every builtin the ST package implements through the real hold environment and fails if the list and the runtime disagree in either direction.