ADR 0048: A holding action deferred by a dead runtime is commanded automatically on reconnect, from a named sub-state of Held rather than from Held itself

Status: Accepted Date: 2026-08-10 Issue: #1397, #1387

Context

A phase whose unit runtime dies while it is holding cannot run its HoldingChart, because the chart's writes need the runtime that is gone. #1387 settled that phase in Held rather than aborting the lot, and that decision stands: the AbortingChart needs the same runtime, so an abort commands nothing either and only the lot's survival differs.

This ADR is about the state it settles into, which nothing ever finished.

The hold posture the HoldingChart describes was never commanded, on either side. reconcileEdgeSelfHold reconnects, finds the runtime not self-held, and returns — correctly, because it exists to reflect an edge hold, not to drive one. armPhaseHoldChart is reached only from the ActionChart path, and a Held phase runs no ActionChart. armBaseline in the physical operator's heartbeat broker restores the unit baseline and says why in its own comment: the runtime "likely restarted and lost its armed program". So the phase rested Held with the field frozen wherever the interrupted run left it (ADR 0006), and with no phase-scoped hold armed at the runtime that came back — losing, by resting, the cover ADR 0008 arms a phase chart for precisely this case.

Whether to command that posture late is a real question and not a technicality. The equipment may have been uncommanded for minutes, the runtime has restarted, and a technician may have already intervened by hand. The founder's instinct was to command it, and the standard both supports that and constrains where from.

What ISA-88 actually says

ANSI/ISA-88.00.01-2010 Clause 7.5.1 divides the example procedural state model into two kinds of state. In an acting state (the -ING names) the element "may orchestrate a defined set of actions", and transition out of it "occurs either upon completion of its defined task or upon receipt of a suitable command". A waiting state is one "for which the procedural element has previously achieved a defined set of conditions and is not permitted to direct any immediate actions".

Table 2 assigns the roles: HOLDING is where the equipment is put "into a known safe state", and "upon completion, control passes to HELD"; HELD "waits for a RESTART command". ISA-TR88.00.02-2022 draws the same line for PackML — HOLDING is Acting and transitions to HELD "upon state complete", HELD is Wait — and adds that the setpoints and procedure status needed to restart correctly "must be saved in the machine controller when executing the HOLDING procedure".

Two conclusions follow, and they point in opposite directions from the implementation we had:

1. A phase that settled Held with its holding action un-run is not entitled to plain HELD at all. It has not "previously achieved a defined set of conditions"; the platform was claiming something it could not substantiate, in a status field an operator and a Part 11 reviewer both read.
2. Running the HoldingChart while the element sits in HELD is a waiting state directing actions, which the same clause forbids.

Clause 7.4 supplies the way through, and it is explicit: exception handling "can direct processing to specific states, for example sub-states of the HOLDING and HELD state to cope with the particular situation arising from certain exceptions". A deferred holding action is exactly such a situation.

Decision

The deferral is a named sub-state of HELD, recorded in Phase.status.holdingActionDeferredSince. Not a Status.Message substring — nothing can branch on prose, and the message was the only trace before this. A phase carrying the field is Held-with-the-posture- uncommanded, which is a different thing from Held, and a distinct hold-deferred alarm annunciates it: "the phase failed" and "the equipment is sitting where the failure left it" are different facts, and only the second decides whether somebody walks to the vessel.

The deferred holding action is commanded automatically on the first reconnect, and the run is driven against a state machine positioned in HOLDING. That is the mechanical expression of the reading above: the interrupted HOLDING transition finishes late, and its state-complete lands in the Held the phase is already in. The phase's own state machine never leaves Held and no ISA-88 transition is invented.

The phase HoldingChart is re-armed whenever the reconnect poll finds that what the runtime reports armed is not this phase's program. A Held phase is still engaged — it is Restartable, and disarmPhaseHoldChart deliberately excludes Held and Paused — which is what entitles it to its own sequenced posture for the next partition. The re-arm is convergence against the runtime's reported armedProgram, not an unconditional POST on every tick, and it reads that value off the status call the reconnect poll already makes, so it costs no extra round trip.

One case commands nothing. A runtime still reporting itself self-held on this phase's own program established the declared posture during the partition, so re-commanding it would tell the field nothing new. The deferral is settled by observation, and the audit record says so rather than claiming a command. A unit-baseline self-hold does not qualify: it is edge cover one posture coarser than the phase declared.

Commanding the safe state is not resuming. Recovery still requires an explicit Restart, exactly as for an edge self-hold. The command that takes the phase out of Held clears the deferral, because the chart that command runs owns the posture from then on.

The late command is audited. The equipment posture changed without an operator asking for it in the moment. "Who commanded this" has to answer "the control plane did, this late, for this reason", which is the class of event 21 CFR Part 11 expects on the record.

Alternatives Considered

Stay in HOLDING until the runtime returns. This is the more literal reading of Clause 7.5.1 — an acting state transitions out on completion of its defined task — and it is not available to us. HOLDING is an active state, so the stuck-phase watchdog force-aborts anything parked there once the threshold passes, which is the lot loss #1387 exists to prevent. Table 3 also allows only STOP and ABORT out of HOLDING, so a phase parked there could never be Restarted even after its runtime came back. The standard's cleanest path is closed by our own liveness machinery, and suppressing the watchdog for this case would trade a loud failure for a silent indefinite one.

Re-enter HOLDING with a fresh HOLD command on reconnect. Rejected: Table 3 does not permit HOLD from HELD (HELD accepts STOP, RESTART and ABORT), and PackML 2022 adds SUSPENDED → HOLDING but likewise not HELD → HOLDING. Inventing the transition would put a non-conformant edge in the state machine to describe a conformant situation the sub-state already covers.

Re-arm only, and leave the posture to the operator. The smaller change, and it closes the lost-edge-cover half on its own. Rejected as the whole answer: it leaves the equipment frozen mid-charge indefinitely, with the declared safe state never commanded by anything, which is the defect rather than a policy about it. Nothing here stops an operator from acting first — a Stop or Abort out of Held supersedes the deferral.

Refuse RESTART while a deferral stands, on the ground that HELD's RESTART is conditioned on "the associated permissives are satisfied" and an un-run hold is an unsatisfied permissive. Rejected: a Restart runs the RestartingChart, which re-establishes the running posture the operator is asking for, so refusing it strands a recoverable lot to protect a posture the operator has just decided against.

Surface it as an alarm only, and require an operator to act. Kept, as half of the decision rather than instead of it. The alarm is how the uncommanded window is annunciated; it is not a reason to leave the window open.

Consequences

A phase held by a dead runtime reaches its declared safe state as soon as the runtime answers, and carries phase-specific edge cover again for the next partition instead of only the unit baseline. Both were claimed by the documentation before they were true — the docs/phases.md paragraph corrected under #1387 explained why the posture was not commanded, which was accurate and is no longer the whole story.

status.holdingActionDeferredSince is the first status field that qualifies an ISA-88 state rather than reporting one. Anything that reads state: Held and treats every Held alike is now reading one bit short. The gateway and the HMI show the state and the alarm, so an operator sees the qualification through the alarm; a consumer that automates on Held should read the field.

armPhaseHoldChart no longer decides what the operator is told. It returns the runtime's not-edge-armable refusal and each caller chooses: the ActionChart path writes the warning to Status.Message, and the re-arm path deliberately drops it, because on a Held phase that message is the diagnosis the operator is acting on and overwriting it costs them the failing transition, the duration and the read error — the #1387 lesson, at a new call site.

The window is measurable. holdingActionDeferredSince is a first-deferral timestamp, not a snapshot, so the message that closes it can say how long the equipment was uncommanded, and a fleet can ask that question across lots.

One case was left uncovered here and closed separately in #1404. Both halves ride the ADR 0008 reconnect poll, which a phase only runs while status.runtimeUnreachableSince is set — and only a chart execution sets that. A phase resting Held after a holding action that did run has no outage recorded and runs no chart, so it has no clock of its own, and the reconciler watched Phase objects rather than pods. A runtime pod that restarted underneath such a phase produced no reconcile at all, and the phase silently kept only unit-baseline cover until something else woke it. The fix filed and priced separately was deliberately not the obvious one: polling every Held phase with a holdingChart on a fixed cadence is one runtime GET per held phase per interval for as long as it is held, and Held has no timeout, so the cadence would have been a standing plant-wide cost and a number chosen rather than a defect fixed. #1404 watches the runtime pod instead. The pod becoming ready as a new process is the event, it produces exactly one reconcile, and every decision that reconcile makes stays in rearmPhaseHoldIfLost, which was already correct on any reconcile it got.