Skip to content

ADR 0080: A restarting runtime restores what it was commanded

Status: Accepted Date: 2026-08-22 Issue: #1776 Related: ADR 0006 (restart-replay autonomy, and the sentence this makes true), ADR 0077 (the file discipline this second file follows), ADR 0074 (the held reading a restart used to discard), ADR 0068 (what the device does while we are gone, which is what decides whether a restore can step anything)

Context

The bench cut the field bus under a running batch (#942 drill 2a, on v0.5.1). The watchdog Held the unit at T0 + 21.1 s, the phase's hold_at_last_sp chart froze the outputs, and the coupler held 60 % and 50 % on two channels for the whole 138-second partition. That is #1683 behaving exactly as documented.

Then the kubelet killed the pod at T0 + 124 s, it came back at 127 s, and the first write to land when the connector went home at 138.7 s was zero on both channels. The zero was ours.

Two separate defects produced that, and the product published that neither existed.

The runtime restored its programs and nothing else

NetworkManager.saveAll persists the deployed programs to {DataDir}/networks/all.json and RestoreAll redeploys them from disk alone. That is the restart-replay autonomy ADR 0006 describes, and it works.

What it never carried is the state that decides what a program writes on its first scan. That lives in two places, both constructed empty:

  • External output overrides — every operator- and recipe-commanded value. A commanded analog output holds its value in a REAL_CONST whose compile-time default is zero, and the operator's 50 % is an override on that block's OUT port. Runtime.OverrideSnapshot/ApplyOverrides already existed to carry these across a hot-swap, with a doc comment saying why. A restart discarded them just as thoroughly and nothing carried them.
  • Block operating points — for a regulating loop, the PID's integrator. This is the half that surprises: restoring the setpoint is not enough. With the setpoint restored and the process already at setpoint, the error is zero, a fresh integrator contributes nothing, and the loop computes zero. It drives the final element to the bottom of its range while every input to it looks correct.

Four documents published the opposite. ADR 0006 and ADR 0008 both say field outputs "hold last value (or device fail-safe) during the gap, exactly as for a pod restart today". docs/failover-runbook.md repeats it. And docs/ha-failure-modes.md uses it as the rationale for #1283's decision not to safe outputs on an undeclared stop: "Outputs hold their last commanded value across the gap; safing here would bump the field on every deployment." That rationale was false. The redeploy bumped the field anyway, three seconds later, to a value nobody chose, with nothing announcing it.

The restart should not have happened

The kubelet killed the pod on a probe timeout, and the graceful all drivers disconnected beyond grace period, reporting NOT_SERVING line was written two seconds later while the process was already shutting down.

Adapter.HealthCheck calls IsConnected on every field driver. Every field driver in this tree held one mutex for the whole of an exchange and took the same mutex to answer IsConnected: pkg/driver/modbus takes d.mu around readRegister and around the reconnect inside it, whose dial carries a five-second default timeout. The gRPC liveness probe has one second.

So the graceful degrade the design describes was unreachable under exactly the condition it exists to report. The 60-second grace was chosen so the ReconnectingDriver gets two full reconnect cycles before Kubernetes acts. It never got to expire, because Kubernetes acted on timeouts long before. This could not happen before #1743, because the grace could never expire at all.

Decision

A restarting unit runtime restores the state its programs were commanded with, before the first scan. IsConnected reports a remembered state and never waits for one.

What is restored

{DataDir}/networks/state.json carries, per network, the external override map and the operating point of every block that has one. It is a second file beside all.json on purpose: the program changes when someone deploys, this changes every scan, and rewriting the program file at that cadence would be pure cost. Both halves have exactly one writer and one reader, which is ADR 0077's rule.

fbruntime.StatefulBlock is what a block implements to carry an operating point. Which blocks implement it is a deliberate line:

  • PID carries the integrator, the previous error and the initialised flag. The integrator is the operating point.
  • AO and DO carry the value last driven to the device. That is what their OUT port publishes, which is what a PID's TRK_VAL reads back for its back-calculation (ADR 0007) and what ADR 0054 calls the effective command. A held loop resumes onto whatever that port says, so an AO rebuilt at zero tracks the loop onto zero however perfectly the integrator was restored.
  • AI carries the last trusted reading and everGood. This one looks wrong at first, because the block reads the device every scan. Since #1738 it also holds its last trusted value when a reading is untrusted, and that hold is the only thing standing between an unwired loop and the rail. A restart under a dead instrument republishes zero as a reading, and every protection #1738 built is spent again from scratch. RAW deliberately does not travel. It is what the device last said, and after a restart the device has said nothing.
  • Timers, counters, flip-flops and edge detectors do not. They carry sequencing state, and whether a restart should include the downtime has opposite right answers per authored use. A reaction timer should include it and a watchdog timer must not. Nothing shipped asks for one, and none of them reaches the field except through a block above that does implement this.

The state is installed between Load and the scan goroutine. Applying it after Deploy returns is a race the caller usually wins, because Run waits out one full tick, and losing it means the first scan writes a compile-time default. Replace held that race from the day it was written and now goes through the same path.

Age governs the annunciation and never the restore

The first draft of this decision bounded the restore by a maximum snapshot age, so a remembered value could not be re-driven onto a plant that had safed while the pod was down. That bound cannot buy what it appears to.

The hazard exists in exactly one declared IOModule configuration (ADR 0068):

Declaration What the device did while we were gone What the restore does
hold, or undeclared Held the operating point Nothing at the terminals
clear + recovery: latch (the default) Refuses process data until its watchdog is deliberately restarted The write is refused; the module goes to Fault
clear + recovery: resume Cleared the outputs Re-drives the commanded value

Only the last row can step a plant. In that row, withholding the restore does not avoid a step. It substitutes a guaranteed step to the bottom of range, which is the defect this ADR exists to fix. A cap therefore makes the one bad case worse, and every other case worse too.

The honest bound is not a number anyone picks. It is the declared fail-safe timeout of the modules a network drives, which the runtime cannot see: IOModuleFailSafe never reaches the pod, and one network's addresses can span several modules. Plumbing it is a follow-up.

So age is reported and never acted on. The threshold is physicalv1alpha1.FailSafeTimeoutFloor, 45 s, the shortest timeout admission accepts. Below it no declared fail-safe can have expired, and the restore is provably lossless. Above it the restore still happens and says so. dcs_runtime_network_state_restored_total carries a freshness label, and the runtime logs the age.

A liveness verdict never queues behind field I/O

connected is an atomic on every driver: Modbus, EtherNet/IP, OPC UA, the simulation driver, and the adapter's own TagDB driver. It is written under the lock that already guarded it, beside the client handle it describes, and read without one. Nothing about its truth value changes. It was always a memory and never a measurement, which #1755 settled, and the io-probe is what measures. What changes is that reading it no longer waits for a dial.

TestIsConnectedNeverWaitsOnIO reads the source of every IsConnected in pkg/driver and internal/adapter, following one delegation hop into the concrete client a driver hands off to, and fails any that takes a lock. There is no allowlist. A driver that genuinely cannot answer from memory has a design problem and not a lock problem.

Alternatives Considered

Annunciate the loss and restore nothing. The issue offered this as the minimum. It is not neutral: it retires four written claims, one of which is load-bearing for a separate shipped decision. Rejected.

Refuse to write until the unit leaves Hold. Also offered by the issue, and not implementable from what the runtime knows. Nothing tells the runtime the unit's ISA-88 state, and the heartbeat carries only a timestamp and a phase name. The edge's own selfHeld was false on the drill, because the Hold came from the control-plane watchdog and the lease rides a network that was never cut. Wiring it would put a control-plane fact on the critical path of edge autonomy, which is the one place it must not be.

Read the output back and adopt what the device holds. This is the best answer on paper. Nothing is remembered, so nothing can be stale. A device that went to its fail-safe is adopted where it now sits. ADR 0066 makes readability a declared property of an output port, and one template in the whole corpus declares it. Neither of the bench's does, so this would have done nothing on the rig that found the bug. Worth revisiting as readBack spreads.

Consequences

  • The four documents that publish hold-last-value across a pod restart are now true. #1283's rationale for not safing an undeclared stop is now true as well, which it was not.
  • A restart during a transient resumes from up to one snapshot interval earlier in that transient, one second by default. A settled loop resumes exactly.
  • An orderly stop resumes exactly whatever the transient, because Adapter.Stop takes the final snapshot and waits for it before anything else in shutdown moves (#1808). The bullet above is what remains for a SIGKILL, a power cut and an OOM kill. That is where it always belonged, and it was the answer for every restart until #1808. The snapshot loop promised a final snapshot on context cancellation. cmd/unit-runtime ends in os.Exit, which does not run deferred functions. The cancel was deferred, so the final snapshot was never taken.
  • A number makes a round trip through JSON as a float64 whatever Go type wrote it. Every consumer inside a block goes through toFloat64 or toBool. The widening is therefore invisible to the scan. RAW is the one value where it would be visible, and RAW is deliberately not restored.
  • The graceful degrade is reachable. Under total field loss the runtime now reports NOT_SERVING after its grace and goes unready. Before this it was killed on probe timeouts before the grace could expire. On the bench sequence there would have been no restart at all. A redeploy, an image bump, an OOM kill and a node reboot each still restart the pod. The health half does not make the continuity half redundant.
  • Adding a block that carries an operating point means implementing StatefulBlock on it. Adding one that does not means saying so, in the StatefulBlock doc comment, beside the four that already do.