ADR 0009: Fail-safe output behavior on program halt — per-block fault tolerance, safe-state write-out on removal, device watchdog backstop

Status: Accepted Date: 2026-06-16 Issue: #577 Related: ADR 0007 (its fail-safe philosophy and safeValue are the substrate this builds on), ADR 0006 (the hold-then-resume posture this is consistent with), ADR 0008 (the partition-response layer that runs inside the scan this keeps alive)

Context

ADR 0007 added device interlocks to the output blocks and, in passing, named a gap it did not close: when the FB scan itself halts, "outputs freeze at the device on program halt." That is the gap #577 addresses.

Two halt paths exist in the runtime today, and both leave field outputs frozen at their last commanded value:

1. A single block's Execute returns an error. pkg/fbruntime/runtime.go set ProgramStateError and the scan loop exited. One transient driver write error on one address (a momentary Modbus timeout on a single coil) therefore stopped the entire program — every other output froze, and interlock evaluation (ADR 0007) and any in-scan holding logic (ADR 0008) stopped with it, because they all run inside the same scan. ADR 0007 already called this out as "strictly worse" when it chose to trip an interlock rather than error on a bad read.
2. A program is removed (not replaced). internal/adapter/network_manager.go Remove stopped the runtime and wrote no terminal state. A pump commanded on stayed on until something else acted.

This is the layer underneath ADR 0007 and ADR 0008. A device interlock forces one output to its safe value while the scan runs; an edge-local hold chart sequences a safe state while the scan runs. Neither helps once the scan has stopped — they are scan-resident. So the foundational question is: keep the scan alive through survivable faults, and define what "stopped" drives the outputs to when it genuinely stops.

A third path — ungraceful process death (pod OOM-kill, node loss, kill -9) — runs no Go code at all: cmd/unit-runtime exits, and the network manager never sees a Remove. No software safe-state write is possible on that path by construction.

Traditional DCS/PLC practice frames the answer. A PLC does not stop the CPU scan when one I/O module faults — it sets a fault bit, drives that module's outputs to their configured fault state, and keeps scanning every healthy module. And every serious remote-I/O module (Modbus, EtherNet/IP, OPC UA) ships a comm-loss watchdog that drives its outputs to a configured fault/safe state when it stops hearing from its master — the backstop for the case where the master is simply gone.

Decision

Three complementary mechanisms, matched to the three halt paths:

1. Per-block fault tolerance (survivable faults keep the scan alive). A block's Execute error no longer halts the scan. The runtime records the faulting block and its message, logs it, and continues executing the remaining blocks; the faulted block's outputs hold their last value (its Outputs() are still collected). When any block faulted on a scan the program reports a new Degraded state (FaultedBlocks in the status names which blocks and why); it returns to Running when they recover. The scan's only fatal exit is context cancellation. This keeps regulation and interlock/hold evaluation alive for every healthy output when one address misbehaves — matching the PLC fault-bit model.

2. Safe-state write-out on deliberate removal. When an operator removes a control program, the runtime drives every output block to its configured safeValue (the same safeValue ADR 0007 defined — DO false, AO outMin unless overridden) exactly once, after the scan goroutine has exited. The stopped scan guarantees the write-out is the sole writer, preserving the FB scan's one-writer invariant (the same invariant ADR 0007 and ADR 0008 protect). Removal is the unambiguous "this program should no longer drive anything" signal, so frozen-at-last-value is wrong there.

3. Field-device comm-loss watchdog as the backstop (deployment-level complement). For ungraceful process death — where no software runs — the remote-I/O module's own comm-loss watchdog is the safe-state mechanism. The product documents this as a deployment requirement and surfaces the recommended fault-state configuration; the deployment instance owns the actual module settings (consistent with the product-vs-deployment split). This is defense in depth: software safe-state covers graceful halts; the device watchdog covers the halts software cannot.

What deliberately does not go safe — continuity paths. Two stop paths must hold last value, not drive safe:

• Hot-swap (Replace). A redeploy stops the old runtime and immediately starts the new one with captured output overrides re-applied. Writing safeValue between the two would glitch live outputs for no reason. Replace skips the safe-state write-out by design.
• Pod shutdown / failover handoff. A graceful SIGTERM (node drain, rolling restart) or a Failover-mode handoff should hold, consistent with ADR 0006's hold-then-resume posture and ADR 0008's failover ordering — the standby (or the restarted pod replaying last-program.json) resumes regulation, and a momentary safe-state bump on every benign restart is the wrong behavior. The runtime does not write safe state on process exit; the device watchdog is the backstop if the exit was a crash.

The result is a clean mapping:

Halt path	Software runs?	Output behavior	Mechanism
One block Execute errors	yes	healthy outputs keep regulating; faulted output holds last value; program Degraded	per-block tolerance
Operator removes the program	yes	every output driven to safeValue once	safe-state write-out
Hot-swap redeploy (Replace)	yes	outputs held, replacement takes over	continuity (no safe write)
Graceful pod shutdown / failover handoff	briefly	outputs held; standby/replay resumes	ADR 0006 hold-then-resume
Ungraceful process death (crash, node loss)	no	device drives its own fault state after comm-loss timeout	field-device watchdog

Alternatives Considered

• Safe-state write-out on any halt, including block errors (issue option B). Rejected as the primary: keep halt-on-first-block-error but write safeValue before exiting. This still surrenders regulation and interlock evaluation for every output the instant one address has a transient write error — the exact "single failing block halts protection for the whole program" complaint the issue raises. Per-block tolerance is the better primary; safe-state write-out is retained only for the deliberate-removal case where stopping is the intent.
• Per-block tolerance only, no terminal safe state (issue option A alone). Rejected: it leaves the program-removal gap open — a removed program's pump stays commanded on. The two mechanisms address different halt paths and are both needed.
• A separate watchdog goroutine in the runtime that forces driver writes on halt. Rejected for the same reason ADR 0007 and ADR 0008 rejected it for their layers: two writers to one output address race. The safe-state write-out runs only after the scan goroutine has exited, so it is the sole writer; it is not a concurrent second writer.
• Treat graceful pod shutdown as a removal and drive safe. Rejected: it contradicts ADR 0006 (hold-then-resume) and ADR 0008 (failover hold-then- fence) — a benign restart or a standby handoff would bump every output to safe and back. Shutdown holds; only explicit removal goes safe.
• Make the device watchdog a product feature rather than a documented deployment requirement. Rejected: comm-loss fault states are configured on the I/O module itself (vendor-specific), which the product does not own. The product documents the requirement and recommended values; the deployment instance configures the hardware.

Consequences

• Code: pkg/fbruntime/runtime.go (per-block tolerance, Degraded aggregate state, WriteSafeOutputs, WithStateObserver), pkg/fbruntime/types.go (ProgramStateDegraded, FaultedBlocks, SafeStateOutput interface), pkg/fbruntime/blocks/{qx,ao}.go (WriteSafeState), internal/adapter/network_manager.go (Remove safe-state write-out; Replace documented as deliberately holding; runtime state observer wired to CM health), internal/adapter/cm_health.go (CMStateDegraded). No CRD shape change — safeValue already exists on the output blocks from ADR 0007.
• Behavior change: a block Execute error now degrades rather than halts the program. ProgramStateError is retained but the scan no longer enters it on a block fault. This is strictly safer (healthy outputs keep regulating and protecting) and matches PLC practice; the trade-off is that a persistently faulting output holds its last value rather than the whole program stopping — visible as Degraded + FaultedBlocks + the CMStateDegraded health topic.
• Observability: new Degraded CM health state on the existing health topic; a dcs_cm_program_degraded_total counter; dcs_cm_fb_network_running now reads 1 while Degraded (the scan is still executing). A persistently degraded program is the signal operators alert on.
• Compliance: strengthens the ISA-88 Clause 7.4 control-equipment- malfunction story — a single device fault no longer collapses basic control for the whole equipment module, and deliberate program removal drives a known safe state rather than freezing. docs/compliance/isa88.md and docs/compliance/iec61131-3.md updated.
• Docs: docs/architecture.md autonomy/port sections, docs/library/alarms-and-interlocks.md (the three-layer table gains a note that the scan now survives single-block faults so the layers keep evaluating), docs/ha-failure-modes.md (program-halt and removal rows, device-watchdog backstop), docs/monitoring-metrics.md (new metric).
• Relationship to ADR 0008: ADR 0008 keeps the partition response sequenced and alive inside the scan; this ADR keeps the scan itself alive through single-block faults and defines its terminal behavior. ADR 0008's hold chart and ADR 0007's interlocks both depend on the scan running, so per-block tolerance directly benefits both.
• Reversibility: high. Per-block tolerance is an internal scan-loop change; the safe-state write-out reuses the existing safeValue; the device-watchdog guidance is documentation. No public CRD contract changes.
• Follow-ups: wire Degraded into the gateway faceplate/Controller detail surface; consider a per-block fault counter (vs. the per-program degraded counter shipped here) if operators need block-level fault rates; the device-watchdog configuration guidance may grow per-protocol recommended values as drivers mature.

Amendment (2026-07-18, #972): configurable fail state per output

The IEC 62443 reconciliation epic (#967) traced this ADR's safe-state write-out to SR/CR 3.6 (deterministic output: "set outputs to a predetermined state if normal operation … cannot be maintained") and closed the one delta the clause surfaced: the predetermined state was configurable in value (safeValue) but not in kind. Output blocks (AO, DO) now take a failState parameter:

• safeValue (default) — drive the configured safeValue once on program halt. The shipped defaults remain de-energized-equivalent (outMin for AO, false for DO).
• holdLast — deliberately write nothing; the output stays at its last commanded value. For outputs where a mid-process step change is more hazardous than holding (e.g. a cooling valve during an exothermic hold).

Invalid values are rejected at Init, so a typo fails deployment, not the fail-safe. A truly unpowered state cannot be produced by a commanded write — that remains the I/O hardware's own comm-loss watchdog/fail-safe configuration, exactly as the device-watchdog backstop section above documents. Interlock trip behavior (ADR 0007) is unchanged: a tripped interlock always forces safeValue — failState governs only the program-halt write-out. Traceability: docs/compliance/iec62443.md SR 3.6 and docs/compliance/iec62443-4-2.md CR 3.6. Relationship to #962 (continuous-control redundancy): standalone — during a hold-then-resume RTO gap the outputs behave per this configuration; #962 decides whether that gap shrinks, not what outputs do inside it.

Amendment (2026-08-05, #1283): the fourth halt path, and the distinction it needs

This ADR enumerated three halt paths and gave two of them a software safe-state write. The third, ungraceful process death, had none available by construction, which is why the device comm-loss watchdog is named as its backstop.

A fourth path was not argued: a graceful stop of the whole runtime. cmd/unit-runtime waits for SIGTERM and calls Adapter.Stop, which publishes an offline status and disconnects the drivers. It never reaches NetworkManager.Remove, so the safe-state write-out this ADR added does not run and every field output is left frozen with only the device watchdog behind it. That is the same outcome as the ungraceful path, arrived at from a path where software was running and could have acted.

The reason it went unargued is that the right answer depends on something the runtime could not see. A rolling update, an image bump and a node-pressure eviction all arrive as SIGTERM with the runtime expected back within seconds, and this ADR's own hot-swap reasoning applies: safing there would bump the field on every deployment. An outage is the same signal with the opposite correct answer, because nothing is coming back.

The decision is therefore a distinction rather than a behaviour change. A stop is terminal only when something declared it so before the signal arrived, through POST /api/v1/stop/arm on the runtime's own API (the shape ADR 0008 already uses for arming a hold). The arming is in memory: a runtime that comes back is by definition not terminally stopped, and a persisted flag would safe the field on the next unrelated redeploy.

The halt-path table now reads:

Halt path	Safe-state write	Backstop
One block's Execute errors	Scan continues; block holds last value	Program reports Degraded
A program is removed	Yes, on Remove	—
Graceful stop, unarmed (redeploy)	No, by decision: output continuity is preserved	The runtime returns within seconds
Graceful stop, armed (outage)	Yes, before the drivers disconnect	Device comm-loss watchdog
Ungraceful death	Impossible by construction	Device comm-loss watchdog

Ordering inside Adapter.Stop is part of the decision. The scan goroutines exit first, so the write has the sole-writer position WriteSafeOutputs requires; the write happens while the drivers are still connected, because one issued afterwards reaches nothing and reports success; and the pod's termination grace period is pinned above the write budget, because a write the kubelet SIGKILLs partway through leaves the field in a state nothing chose. failState: holdLast is honoured here exactly as it is on program removal.

The write-out is recorded. The runtime queues a safe-stop event onto the same store-and-forward buffer ADR 0008 gave hold events, and the physical operator materializes it into an AuditRecord stamped with the edge time. The buffer, rather than a direct apiserver write, is the load-bearing choice: a terminal stop is usually happening because the control plane is going away too, so the record most worth having is the one most likely to be lost. A partial safe is recorded as a failure, because an audit trail that graded it a success would be the fabrication the record exists to prevent.