Skip to content

ADR 0007: Device-level interlocks run in the FB scan on the edge — three-layer protection model

Status: Accepted Date: 2026-06-12 Issue: #567 Related: ADR 0002 (amended by this ADR), #566 (edge redundancy scoping)

Context

ADR 0002 designated the phase SFC transition guard (interlock: true, priority <= -1) as the "primary, deterministic interlock" and AlarmDefinition.exceptionAction as its audit-traceable companion. Issue #567 identified the flaw in that stance: if interlock functionality does not exist at the layer that actually drives devices, it is not actually an interlock. Three gaps, in increasing order of severity:

  1. Coverage. A phase guard evaluates only while its phase executes. An idle unit, a unit in manual mode, and an operator-commanded output (a tag write landing as a runtime output override) are all unprotected — nothing stops a pump being commanded against a closed inlet valve when no phase chart is active.
  2. Scoping. Phase guards are recipe-scoped; a pump/valve cross-check is equipment-scoped. Encoding it per phase means every recipe author re-implements (and can forget) the same protection.
  3. Partition tolerance. The SFC engine executes inside the procedural operator on the control plane (internal/controller/procedural/phase_controller.go), with each scan's READ/WRITE builtins crossing HTTP to the unit runtime via pkg/stbridge. During a control-plane outage or network partition — exactly the scenario the edge is designed to survive autonomously — phase guards stop evaluating, while the edge FB scan keeps driving outputs. ADR 0002's "deterministic one-scan interlock" was therefore overstated even on its own terms: deterministic, yes; always-on, no.

The only layer that keeps executing through all three cases is the IEC 61131-3 function block runtime on the edge node (the autonomy story: the unit runtime persists its program to local disk and replays it on restart). ISA-88 Part 1 supports this placement directly: interlocking is a form of basic control (Clause 3.1.5, Clause 5.2.1), and the control module is "the lowest level grouping of equipment in the physical model that can carry out basic control" and "contains the interfaces to the physical equipment" (Clause 3.1.13). Traditional DCS platforms (DeltaV and PLC practice generally) implement device interlocks the same way: a boolean input on the output block that forces a defined safe state while true.

Decision

Output function blocks implement device-level interlocks, evaluated every scan in the FB runtime on the edge. Protection is layered in three tiers, each with a distinct, documented role:

Layer Where it runs Latency Survives control-plane partition Role
Device interlock (DO/AO ILCK input, interlockAddress param) FB scan on the edge node One scan (100–200 ms), always-on, mode-independent Yes Equipment protection: force a single output to its configured safe state while the trip condition is true
Phase SFC guard (interlock: true transition) Procedural operator (control plane) One SFC scan, while a phase executes No Procedural safe-state sequencing: ordered multi-output response (vent first, then cut heat), parking the procedure in a resumable safe-hold step
AlarmDefinition exceptionAction / ProcessException Alarm controller (control plane) 10–60 s No Annunciation, ISA-18.2 / 21 CFR Part 11 audit record, batch-level Hold/Stop/Abort

Mechanics of the device layer:

  • DO and AO gain an optional ILCK boolean input port (wired logic; compose AND/OR/NOT/SR upstream for complex or latching conditions) and an optional interlockAddress parameter (a trip signal the block reads from the device driver itself each scan, with interlockInvert to flip its sense). The two sources are OR'd.
  • While tripped, the block writes safeValue (DO default false; AO default outMin, validated within [outMin, outMax]) instead of its IN value — including when IN is fed by an operator override. The ILCK_ACTIVE output port exposes the trip state for faceplates and alarm definitions.
  • Fail-safe: an unreadable or quality-Bad interlockAddress read trips the interlock rather than erroring the scan (a block error halts the whole program, freezing every output — strictly worse).
  • The interlock is non-latching; latching is composed upstream with the existing SR block.
  • PID gains a TRK input that freezes its integrator, wired from the same trip signal, so a forced downstream AO does not wind the controller up.
  • Library ControlModuleTemplates pass safeValue/interlockAddress/interlockInvert through to their output blocks as {{.params.*}} expressions. Unset parameters are dropped at compile time, so existing instances are untouched and an instance opts in by setting spec.parameters.interlockAddress — no template rewiring, no CRD change.

ADR 0002 is amended, not replaced: its alarm reaction-time table and the tagEvalInterval stance remain in force. What changes is the interlock designation — the phrase "phase guard = primary interlock" no longer applies. The phase guard's role is procedural safe-state sequencing, which a device interlock cannot do (a device interlock forces one output to one static value; it cannot order a vent-then-heat sequence or park a procedure in a resumable held state). The alarm layer is unchanged. The BPCS-not-SIS disclaimer applies to all three layers: none of this is a safety instrumented function; safety-critical trips remain hardwired or in a certified safety PLC.

Framing reconciled by ADR 0011

The "safety-critical trips remain hardwired or in a certified safety PLC" phrasing here predates ADR 0011. The boundary recorded in this ADR is unchanged. The accurate framing is that CNDCS device interlocks, phase guards, and alarms are BPCS-layer protection functions that can be credited as independent protection layers (IEC 61511 Clause 9.2); whether a hazard also requires an independent SIS is a per-process PHA/LOPA decision, not a blanket product requirement.

Bypass stance (v1): the runtime's existing audited tag-write path can override the source of a wired ILCK signal, which constitutes an interlock bypass and is captured by the write audit trail. A first-class bypass (permission-gated, time-boxed, annunciated) is deliberately deferred — tracked as a follow-up issue.

Out of scope: cross-unit device interlocks. Each unit runtime is a separate pod with its own I/O scope; a trip signal spanning units either belongs to whichever unit drives the protected output (read it as that unit's input) or needs the control-plane layers. Cross-unit edge-to-edge signalling, if ever needed, should be scoped together with edge redundancy (#566).

Alternatives Considered

  • Status quo (control-plane-only interlocks). Rejected — see Context. The coverage and partition gaps are not closable from the control plane by construction.
  • A dedicated Interlock CRD (cluster-side resource compiled into edge logic). Attractive for auditability — interlocks become enumerable objects — but it adds a CRD, a controller, a compilation path, and a sync protocol before the first interlock can exist, and the result still has to land in the FB scan to be real. The block-level mechanism is the substrate such a CRD would compile to; it can be added later without reworking this layer.
  • Interlock evaluation in the unit runtime but outside the FB scan (a separate watchdog goroutine forcing driver writes). Rejected: two writers to one output address race each other; the FB scan already owns output ordering and the one-writer guarantee.
  • Renaming the SFC interlock field to reserve the word for the device layer. Deferred — it costs a CRD field deprecation cycle and library-wide doc churn for a terminology cleanup; the docs now qualify the SFC flag as a procedural guard wherever it appears.

Consequences

  • Code: pkg/fbruntime/blocks/{qx,ao,pid,interlock}.go, pkg/blockcatalog/catalog.go, gateway ST/LD renderers. No CRD shape change (block params are map[string]string); no migration.
  • Templates/examples: the six library templates with output blocks pass the interlock parameters through; the pharma-plant fixture interlocks mt1-outlet-valve against the inlet valve's feedback with a paired TagTrue: ILCK AlarmDefinition as the audit companion.
  • Docs: docs/library/alarms-and-interlocks.md restructured around the three-layer table; reaction-time table in docs/alarm-management.md gains the device row; docs/architecture.md autonomy section cross-links here; docs/compliance/isa88.md gains the basic-control/interlocking row.
  • Compliance: strengthens the ISA-88 Clause 5.2 basic-control story (interlocking now exists at the control module level, where the standard places it). The 21 CFR Part 11 record of a trip is the paired AlarmDefinition, unchanged in mechanism.
  • Operations: a tripped interlock is visible as ILCK_ACTIVE on the faceplate and (where paired) as an alarm. Operators recover by clearing the trip condition; there is no latch to reset unless the template composed one.
  • Follow-ups unlocked: first-class gated bypass; PID back-calculation output tracking; fail-safe output behavior when a runtime errors or stops (today outputs freeze at the device on program halt — adjacent gap, filed separately).
  • Reversibility: the ports and params are additive and optional; unwired blocks behave exactly as before this ADR.