ADR 0082: A promoted runtime resumes from the plant¶

Status: Accepted Date: 2026-08-23 Issue: #1797 Related: ADR 0080 (the same question one node over, and the argument that does not carry), ADR 0006 (the failover this is about, and the sentence this makes true), ADR 0007 (the back-calculation a promoted loop resumes through), ADR 0066 (a different read-back, and why it is the wrong instrument here)

Context¶

The bench cut power to the active node under a running batch, three valid reps on 2026-08-23 (#942 drill 1, on v0.5.1). The coupler held both output values for the whole dead-node window at quality Good, which is ADR 0006 and #1683 behaving exactly as documented. Then the standby was promoted, and both outputs stepped to zero about four seconds before the unit reported itself controlling.

Rep	Cut	Outputs → 0	Standby controlling
1	ipc-2	+52.1 s	+55.6 s
2	ipc-1	+52.1 s	+55.9 s
3	ipc-2	+53.5 s	+57.6 s

Rep 1 alone cannot say what the zero is, because that recipe's own shutdown phase ran five seconds earlier and the recipe ending is the obvious reading. Reps 2 and 3 refute it: the shutdown phase runs 30 s and 66 s after the outputs have already zeroed, while the offset stays put.

ADR 0080's argument is a same-node argument¶

ADR 0080 made a runtime restarting in place come back on the values it was commanded with, and it deliberately put no age bound on the restore. Its reason is written in internal/adapter/network_state.go: with hold declared or undeclared the device is still holding the value we last drove it to, so restoring is a no-op at the terminals.

That reason holds because the snapshot and the terminals are two copies of one fact. A promotion breaks the identity. The snapshot on the standby's disk is what this node last commanded, in a stint that ended when the unit was re-bound away from it. The terminals hold what the node that just died commanded, seconds ago.

So the fix for the restart made the failover worse, and nothing said so. A standby that has hosted the unit before finds its own leftover state.json and re-drives it. That is the bench's own topology, since the two IPCs trade the unit back and forth. Reproduced off the rig: node A holding the plant at 50 %, the standby takes over and writes 20 % from a four-hour-old file, annunciated as a successful continuity restore with a beyond-failsafe-floor freshness label. Age was the only thing recorded, and age is the wrong instrument. The four-hour-old snapshot reads as "old but ours". It was never ours.

Five published sentences were false¶

ADR 0006 §Decision.3, ADR 0008 (twice), docs/failover-runbook.md and both node-failure rows of docs/ha-failure-modes.md all say outputs hold last value across the gap and the gap ends on an ISA-88 Restart. Row 70 is the sharpest: "Held outputs during the gap, then the unit is re-bound to the standby and the procedure running on it holds with it."

ADR 0006 §Decision.3 compares the failover gap to a pod restart, in the words "exactly as for a pod restart today". ADR 0080 fixed the pod restart. That moved the comparison and left the failover half false in a new way.

Decision¶

A promoted unit runtime adopts the operating point the device is holding. Where the device cannot answer, it writes nothing at all.

Founder ruling of 2026-08-23, taken against the measured option set.

A promotion is a binding change, and the binding epoch says so¶

state.json carries the binding epoch that wrote it. The epoch increments on every re-bind (unit_failover.go) and deliberately does not move on a pod restart (podAvailabilityDrift says so in as many words), so "this snapshot is mine" is exactly an epoch match. Nothing else in the snapshot could answer it: savedAt measures age, which is the question ADR 0080 already settled.

An equal epoch is an in-place restart and ADR 0080 applies unchanged. Epoch 1 is a unit's first binding and epoch 0 is a runtime started with no flag, and neither can be succeeding anybody. That is what keeps this off the ordinary path: a first boot, an image bump, an OOM kill and a node reboot all leave the epoch where it was.

--binding-epoch therefore goes to every unit-runtime pod and not only to a leased one. An Autonomy-mode unit is re-bound by the same dcs unit failover --confirm-fenced path and its epoch increments the same way, so without the flag its promoted runtime would read another binding's operating point as its own. Passing it enables nothing else. NewLeaseGuard fences in Failover mode alone.

The read-back is already built, and it is not ADR 0066's¶

ADR 0080 considered adopting what the device holds, called it the best answer on paper, and declined it because one template in the corpus declares ADR 0066's readBack. That is the wrong instrument. readBack is a program-authoring concept that wires an input block to an output's address. What a promotion needs is the address, and the product has carried it per channel since #1689:

Layer	What it holds
`pkg/deviceprofile/wago750`	The documented `0x0200` read-back offset (750-352 manual §11.2.4, Table 101)
discovery → `IOChannel.ReadbackAddress`	`holding.N` → `holding.N+512`, `coil.N` → `coil.N+512`
`internal/adapter/ioconfig.go`	`buildReadbackMap`
`internal/adapter/server.go`	`ReadValue` substitutes the read address transparently

It was cross-checked against a multimeter in #1502 and again in #1712, and ha-drill.sh samples holding.512/holding.513 once a second for the whole drill. The instrument that measured this defect is the read-back itself.

The drivers are connected before the restore runs. a.Start(ctx) is at cmd/unit-runtime/main.go:246, and RestoreAll runs inside NewHTTPServer at line 315. So a promotion can read the terminals in the same slot ADR 0080 installs a snapshot into, after Load and before the scan goroutine.

That premise held on the bench where this was written and failed on the first failover measured after it. The withhold is still in that slot. The read is not, since #1935, and the amendment at the end of this document says where it runs now.

Adoption is offered only where a channel declares a read-back. Reading an undeclared output address does not fail. On a WAGO coupler the write address answers from the input process image, at Good quality, carrying another channel's number. The drill runbook records 141 of 172 samples byte-identical to input.0. An opportunistic read is not a best-effort adoption. It is ADR 0076's hazard with a different address on it. A reading the block cannot express is refused rather than clamped, for the same reason: the block's own range is the last check there is, and clamping would launder a wrong-image reading into a plausible operating point.

Three output phases¶

Withholding and driving are the obvious pair. Between them sits a block that knows exactly what the plant is at and still has no idea what it is supposed to be, and that is every output commanded through a constant. The operating point lives in the constant. A promoted runtime's copy of it is a compile-time default. A block that adopted 60 % and then passed IN through would write zero on its very next scan, which is the defect wearing one more layer.

Phase	What the block writes
`driving`	What `IN` carries, every scan. Every block starts here and an ordinary program never leaves it
`holding`	What it adopted from the device, ignoring `IN`, until something commands it
`withheld`	Nothing. The device goes on holding whatever it has

A withheld output is a silence about the operating point and not a silence on the wire. The device's own watchdog is fed by any telegram at all, whatever its trigger mask says. ADR 0068 measured that on the live coupler. With FC3 excluded from the mask, once-a-second FC3 reads held the watchdog open for twenty seconds on a five-second timer, and the mask governs only what may restart it after a timeout. The scan goes on reading that coupler's inputs while an output withholds, and the io-probe reads it on its own cadence. That cadence is the heartbeat ADR 0071 measures its timeout floor against. So a coupler declared clear does not fail-safe underneath either phase. Adoption is the better of the two answers because it continues control, and not because withholding would cost the plant its watchdog.

The phase is half the operating point¶

Holding and withholding are long-lived states. They are not moments. An output commanded through a constant holds what it adopted until something commands it, and one whose channel declares no read-back withholds on the same terms. The first consequence below says "for ever". For four days it was not true (#1800).

aoState and qxState persisted the last value and not the phase. The next Load therefore brought the block back in driving, carrying a restored operating point that nothing in that phase reads. The first scan wrote IN, which is the constant's compile-time default. That is the step this decision exists to prevent.

Two ordinary events reach that Load. An engineering hot-swap carries the outgoing runtime's own state across. The kubelet restarting the pod in place classifies as a restart and not as a promotion, because the binding epoch has not moved. Neither event is a promotion, so neither re-adopts. The step lands on a plant the coupler has been holding correctly since the other node died.

The phase is saved and restored beside the value now. It travels as its own word and not as its position in the list above. The file outlives the build that wrote it, and one of the three answers is to write nothing. A word this build cannot name is refused, because the cost of refusing is one block restoring nothing and the cost of guessing is a write. A snapshot written before the field existed carries no phase at all. That decodes as driving, which is what it always meant.

The release had to move with it. The cone an external command releases through was built by WithholdOutputs, and only a promotion calls that. An output restored into withheld would have been deaf to every command for the life of the pod. Load builds the cone now, because a block can be non-driving without this runtime having withheld anything.

A loop resumes through the back-calculation it already has¶

Seeding the output block does not carry a regulating loop across, and ADR 0080 is where that is written down: with the setpoint restored and the process already at setpoint the error is zero, a fresh integrator contributes nothing, and the loop computes the bottom of its range while every input to it looks correct. A promoted runtime cannot recover the integrator, because that is precisely the state the dead node had.

PID already back-calculates its integrator onto TRK_VAL while TRK holds (ADR 0007). A promotion arms one scan of that, so the loop starts on the value the device is holding. It reuses the existing machinery. A second implementation would compute the integrator a second way, and a promoted loop now resumes exactly as an interlock release does. One armed scan is where this started and it was not enough on its own, for the reason the section below sets out.

It is armed only where TRK_VAL is wired to an output that actually adopted. Both halves are load-bearing. Unwired, Bus answers nil, and nil coerces to zero. Arming would then drive the loop to the bottom of its range by the road this exists to close. And a source still withholding has never been told anything either, so its OUT port is the compile-time default. A loop that will bump is reported at promotion, because that is the only moment anybody can be told.

A loop tracks its held output for as long as that output holds¶

The armed scan is a moment, and holding is not. That is the same mistake as the one two sections above, one layer over, and it survived four days for the same reason (#1815).

resumeFromPlant logs the promise in as many words. It writes a promoted loop will resume on the value its output is holding at promotion, and the resume it describes cannot happen at promotion. The output it names is holding, so it re-writes what it adopted every scan and ignores IN until something commands the loop back. On the bench nothing does so for a minute or more. By the time something does, the sentence is false.

ArmTrackInitialisation arms exactly one tracking scan. PID.TRK is wired from the AO's ILCK_ACTIVE, and that signal is false. So from the second scan the loop free-runs against the setpoint constant, and a promoted runtime's copy of a constant is its compile-time default. The integrator winds away from the operating point the AO is faithfully keeping, and anti-windup pins it a couple of scans later. Then somebody commands the loop back, the AO switches to driving on the same command, and its very next write is that wound value.

	`pid.OUT`	`write_out.OUT`	device
after promotion	29.95	59.95	19644
holding, with the setpoint still the compile-time 0	0	59.95	19644
first scan after the release	53.95	53.95	17678

That is a 6.00 percentage-point step down at the terminals, at the instant control resumes, against a loop whose measured operating band is 0.029 sigma (#1794). The step is the same at 0.5 s, 2 s, 10 s and 30 s of holding, because anti-windup bounds it. What sets it is the tuning and the distance between the compile-time default and the held point. How long the node was down does not come into it. The AO half of this decision worked perfectly throughout, and the plant sat at 19644 for the whole window.

A loop tracks its held output for as long as that output is not driving, and the release is one last tracking scan. The condition is a runtime state and not a plant fact, so the PID asks for it each scan through FBContext.SourceOutputPhase. Declaring it on the document would be ADR 0074's shape, and it would buy authoring visibility this does not need at the price of re-wiring every loop in every corpus. No document changes, then, and no gate to hold the wiring. It also makes the log line above true whenever the resume actually arrives.

The release is a tracking scan and not simply the end of tracking. The command that releases the output is usually the same command that installs the real setpoint, so back-calculating one more time is what puts the loop on the held value at the setpoint now in force. Ending tracking a scan earlier leaves an integrator computed against the compile-time setpoint. Measured through the runtime on the bench's own tuning, that reverses the sign of the step and keeps its size: 84 % onto a plant held at 60 %, where the untracked loop handed over 24 %.

Withholding is deliberately not tracked onto, and that is the line ArmTrackInitialisation already drew. A withheld output has never been told anything either, so its OUT port carries a compile-time default. A loop above one goes on running, and the promotion log says it will resume from a fresh operating point and may step the field. That is the honest answer when nothing anywhere knows what the plant is at.

The one armed scan stays for the case it was written for, which is an output that is already driving on the first scan.

The release, and what is never withheld¶

A withheld output that never writes again is a silence. Nobody should read it as a safe state. The release is somebody standing behind a value: the ISA-88 Restart's own restarting logic, an operator command, a recipe WRITE. It is scoped to the commanding block's cone, meaning every block whose value can reach that output. So a recipe that commands one loop does not put a second one back on the field on no evidence. The closure guards against revisiting, because a tracking wire makes the graph cyclic on purpose.

An interlock trip and the program-halt fail state are never withheld. Each names a value the block does stand behind, and a withheld protective action is worse than a bump.

Alternatives Considered¶

Replicate the dead node's last value. The runtime already publishes every FB output tag off-node at 5 Hz. Making a standby able to consume it means a retained topic plus an ACL grant (#1781's rule). It also amends ADR 0006's "does not replicate runtime state to a standby". Considered seriously and not chosen, on one asymmetry: a replicated snapshot says what the dead node wanted, and a read-back says what the plant is. They agree under a power cut, where the last copy is at most a second old. They diverge under a cluster-path partition, where the node kept controlling and then ran its bounded hold chart and published none of it. That is exactly the failure the fencing machinery exists for. It also leaves the staleness question open, which is the question that produced this issue.

Keep writing the compile-time default where the device cannot answer. Today's behaviour, and the cheapest. It leaves the bench's headline exactly as measured on any channel with no declared read-back, so the five published sentences would have to be rewritten to say the field is bumped on every failover. Rejected in the ruling.

Refuse to promote when the operating point cannot be established. The loudest and the strongest safety posture. It converts a recoverable failover into an outage on any deployment that has not declared its read-back addresses, which today is every deployment including the bench. Rejected.

Bound the restore by age instead. This is the bound ADR 0080 already examined and declined, and the failover case does not rehabilitate it. Age was never the thing that made the standby's snapshot wrong. Provenance was.

Consequences¶

The five documents that publish hold-last-value across a failover are now true. On a channel with a declared read-back the gap ends on a continuation. On one without, it ends where ADR 0006 always said it did, on the recipe's own Restart.
A deployment has to declare where its outputs read back to get the good path. Discovery does it automatically. A hand-authored IOModule does not. The bench's own is hand-authored. dcs_runtime_output_adoption_total reports no-readback for every channel that has not, per promotion.
A loop above a held output is tracking, so its CV is the plant's own value and not a computed one. That is what the promotion log line has always claimed and what the 5 Hz telemetry, the HMI and anything wired downstream of the loop now read. The loop starts deciding again on the scan that releases its output, and that scan is a back-calculation.
A promoted output that nothing ever commands holds or withholds for ever. That is the ruling, and since #1800 the phase survives a hot-swap and a pod restart, which is what makes "for ever" mean it. Both long-lived phases are annunciated, and since #1816 that sentence is true of holding as well. See the section below.
ADR 0080 is unchanged wherever the epoch matches, which is every restart that is not a re-bind.
The lease endpoint does not wait on the coupler. Since #1935 the read-back runs beside the scan, and the listener the grant arrives on is up before the coupler has answered. See the amendment below.
Adding a block that writes to a device means implementing EstablishableOutput on it. TestEveryBlockThatWritesADeviceCanBeWithheld reads the source of every block reaching ctx.Driver.WriteValue outside its fail-state path and requires it. A hand-kept list would agree with the runtime by assumption (#1655).

A holding output is writing, so it had to be counted separately¶

The sentence above said "annunciated and never silent" from the day it was written, and for holding it was false for four weeks (#1816). Measured on the bench on 2026-08-25, a unit promoted an hour and forty-five minutes earlier with both its outputs adopted and holding ever since:

dcs_runtime_output_adoption_total{outcome="adopted"} 4
dcs_runtime_outputs_unestablished{network="loop-pid-logic"} 0
dcs_runtime_outputs_unestablished{network="aux-output-logic"} 0
dcs_runtime_writes_total{outcome="ok"} 115252
dcs_runtime_writes_resumed_total 1

Every one of those readings is correct and none of them is the answer. Four things to know.

Writing is what makes it invisible. A withheld output is a silence, and a silence can be counted by counting the blocks that are silent. A holding output writes at scan rate. It moves the same counter a program driving the plant moves, at the same rate, carrying the same ok outcome. ADR 0081's fenced outcome separates a write that never reached the field. That is a different case and it does not cover this one. There is nothing to subtract and nothing to compare, so the discriminator has to come from the block.

A counter cannot say a state is still current. dcs_runtime_output_adoption_total records that an adoption happened. The question an operator brings is whether the block is still in the phase that adoption put it in, and no monotonic series can answer it. That is what dcs_runtime_outputs_holding is for.

The gauge is swept. Both gauges were set once per deploy, which was already wrong for the unestablished one and would have been useless for this one. The release is an operator command, a recipe WRITE or the ISA-88 Restart's own restarting logic. None of those is a deploy, so a block released a second after promotion would have gone on being reported as holding for the life of the pod. Both are recomputed once a second off what each runtime reports. That is level-triggered for #1806's reason: the phase changes inside the runtime on whichever goroutine a command arrives on, and a gauge wired to one caller's edge is a gauge the next caller defeats by existing.

No condition and no alarm, which is the line #1669 drew for configError. The phase reaches the gauge, the runtime's diagnostics response, dcs get runtime and the unit's Diagnose panel. It reaches no Unit condition and raises no alarm. Nothing here changes control behaviour, and holding is not a fault: the plant is where the promotion found it and the coupler is holding it there. What is wrong is that no part of the program is driving it. That is a deployment finding about the recipe, on the same footing as a refused simulation profile. The recipe that produced the bench reading is the case in point. bench-hold-at-setpoint writes aux.CMD and never loop.SP, so the loop stayed open from the failover to the end of the batch through an ISA-88 Restart the drill recorded as recovered. That was fixed in the bench recipe. The product cannot fix it for anybody else, and now it can say it.

While correcting one false sentence: a withheld output does not reach the unit's health either, whatever UnestablishedOutputs said about itself before #1816. Adapter.HealthStatus is computed from field-driver connectivity and from nothing else (ADR 0075), and the unit controller's watchdog reads the status API, which carries no phase.

The plant is asked beside the scan, and the lease does not wait on it¶

(#1935 amendment, 2026-09-03.) The read-back section above placed the read between Load and the scan goroutine, on the premise that the drivers are connected before the restore runs. On a failover replacement that premise is the thing most likely to be false. The runtime is starting on a node the unit was not running on, the coupler may be unreachable from it, and the driver's connect in Adapter.Start fails and defers to the first I/O. The promotion's read-back is the first I/O. So the read dials, and the dial is bounded by the transport's own timeout or by the kernel's ARP clock, neither of which the caller set.

Two deliberate failovers on the bench, chart 0.7.5, read the same six seconds between runtime started successfully and starting runtime HTTPS server. The health, TLS and metrics listeners between those two lines came up inside 12 ms. The rest was two Modbus dials to a WAGO the replacement node had no route to, 3.07 s each, one per output block. They were made from inside deploy with the manager's mutex held, inside RestoreAll, inside NewHTTPServer, before the listener the lease grant arrives on existed. The operator's re-deploy of every network after the grant paid the same price again, inside the HTTP handler. A coupler that answers ARP but drops the connection would have cost the transport's full timeout per output, 5 s for Modbus and 10 s for OPC UA. A unit with a dozen outputs would have reached the 120 s start-up grace with no lease endpoint to be granted on. That is #1933's shape one function over: a network round trip on the goroutine control depends on, bounded by nothing.

What the slot protected was the first scan's write, and the withhold is still in the slot. WithholdOutputs runs between Load and the scan goroutine as before, and a withheld output writes nothing however many scans pass. The asking moved. resumeFromPlant runs on the entry's own goroutine and context, beside the scan. Each output adopts when its answer arrives. Runtime.AdoptDeviceValue and Runtime.ArmTrackInitialisation take the scan lock for the reason StateSnapshot does, so an adoption lands between two cycles. The tracking scan is armed after the adoption, however late that is. That is what #1815 requires. A Replace or a Remove cancels the entry's context. An answer that arrives for a network no longer running goes nowhere: it is not counted, and the phase gauges are set only if the entry is still the one the manager runs under that name. NetworkEntry.Adopted is closed when the adoption has finished, and nothing on the control path waits on it.

On a coupler that answers, the read takes a few milliseconds and the first scan comes a full interval after Run starts, so in practice the adoption still lands before the first scan. The change is visible only where the coupler is slow or away. There the outputs withhold for exactly as long as before, while the lease endpoint listens from the start.

The driver connect inside Adapter.Start is the same class and is still on the start path. It took 3.07 s on the same reading, before the health server listened. That is #1936.