Skip to content

ADR 0067: A dwell counts only time the control system was present

Status: Accepted Date: 2026-08-18 Issue: #1685 Related: #1349 (the cold-boot drill that found it), #1683 (the remote I/O held its last commanded output throughout), ADR 0008 (the armed hold, which needs a live process and so cannot cover a machine that lost mains), ADR 0047 (the persisted position a resume is taken from)

Context

A step's elapsed time is now - activatedAt. The activation time is persisted in the Phase status and restored on resume, which is deliberate and is what lets a phase survive a controller restart at the step it was on.

Nothing subtracted the interval in which no scan ran. On the bench, during #1349's cold-boot drill, five computers were cut at the PDU while a phase was holding at setpoint. They were dark for eleven minutes and came back on their own. The hold step's guard is hold_sp.T >= T#120s, and the record reads:

Field Value
Phase hold-at-setpoint
Start 2026-08-18T14:48:15Z
End 2026-08-18T15:01:54Z
Scans 2

Two scans. The step was active for about a minute before the cut. The first scan after recovery found the elapsed time well past 120 seconds and fired the transition. The batch then ran to Complete with nobody in the loop.

The process was not idle during those eleven minutes. The field zone is on a separate supply, so the remote I/O held its last commanded output. It measured 13.58 mA at the terminals, which is the 60 percent the loop was left at. The loop really was driving with nothing supervising it, and the batch counted that as hold time.

The product already accepts the principle for the other reason a chart stops. resumeAfterSuspension shifts every step's activation time forward by the interval a phase spent held or paused, so an operator who takes twenty minutes over an intervention does not return to a step budget sized for a fifty-second drain. That compensation is driven by the marker a Hold or a Pause sets. A machine losing mains sets nothing, so it never ran.

The asymmetry runs the other way too. A phase whose runtime is unreachable for longer than a minute self-holds and raises an alarm. A phase whose controller was absent for the same minute cost nothing at all.

Decision

A step's dwell counts only time in which a scan of its chart ran. An interval in which none did is a control gap, and what the dwell does about one is a property of the step.

The engine stamps lastScanAt at the top of every scan, from its own clock, and the Phase status publisher persists it. The distinction matters and is the one #1071 already draws. A publisher tick proves the ticker goroutine fired. Only the engine can say a scan ran.

A run resuming from a persisted position measures the distance from that stamp before it re-enters the chart, which is the one moment the question is answerable. Inside a run the engine is scanning by definition. The threshold is one minute, the same minute the runtime outage grace uses, because it is the same claim read from the other end.

steps[].onControlGap carries the verdict:

Verdict What it does What it is for
Hold Takes the gap back out of the dwell and self-holds the phase The default. It is the only verdict that is not wrong for some dwell, because it does not decide anything about the lot
Extend Takes the gap back out of the dwell and carries on A mix time, where the interruption costs nothing but the clock
Fail Ends the run and aborts the phase A dwell whose conditions cannot survive an interruption at all
Count Counts the gap as dwell time A dwell anchored to wall clock, such as a 24-hour age, that an outage genuinely does not interrupt

Where the active steps disagree, the strictest verdict governs the phase. A phase cannot both abort and carry on.

The gap is recorded on the Phase, audited, and alarmed for every verdict, including Count. That the lot spent eleven minutes with nothing controlling it is true whatever the recipe author decided to do about it.

Alternatives Considered

  • Extend every dwell silently, with no per-step field. Rejected. It is the right answer for a mix time and a false one for a sterilization hold. In the second case the temperature fell while nothing was watching, and a silent extension declares the hold valid on the strength of a clock. One global answer cannot be right, because the question is about the process the step runs.

  • Fail every dwell that spans a gap. Rejected for the mirror reason. It destroys a lot for an interruption that cost it nothing. Most of the outages this covers are a controller restart, and the plant that lost mains is the rare one.

  • Hold the phase with no per-step field at all. Rejected, though it is the closest miss. It is the right default and it takes the ruling away from the recipe author, who is the only one who knows whether the dwell is a sterilization hold, a mix time or an age. The field defaults to it, which keeps the safety and returns the ruling.

  • Detect the gap from status.lastProgressAt, adding no new field. Rejected. That field is the stuck-phase watchdog's liveness heartbeat. It is deliberately unconditional, and several writers advance it. A tick proves the publisher goroutine fired and says nothing about the chart, which is the distinction #1071 exists to draw. Reusing it would have measured the wrong interval and made the watchdog's meaning depend on a second reader.

  • Derive the threshold from the step's own dwell, so a gap that is a small fraction of a long hold is ignored. Rejected. It is defensible in principle. In practice it makes the verdict depend on arithmetic an operator cannot check against anything, where the one-minute grace is a line the product already draws elsewhere for the same reason.

  • Refuse the resume outright and require an operator to Start a fresh run. Rejected. It throws away the position, which ADR 0047 exists to preserve. The lot is usually recoverable anyway. Held is the ISA-88 state for "a human must rule on this", and it keeps the position while they do.

Consequences

A timed dwell now asserts what a batch record has always claimed it asserts. The product spent the stated time under stated conditions, with a control system present to state them.

Hold being the default means a control-plane outage longer than a minute lands every running phase in Held. That is the disruptive answer and it is chosen on purpose. Extend is wrong for a sterilization hold in which the temperature fell. Fail is wrong for a mix time. Count is the defect. Only Hold is safe to have not thought about, and a recipe author who has thought about it writes down which of the other three applies.

Recovery from a Hold verdict is the ordinary ISA-88 Restart. The compensation has already run by then, so the step gets its full declared dwell measured from the resume.

Count keeps the pre-decision behaviour reachable. Reaching it is a claim about the process, and the schema makes it a choice the recipe author writes down. That is the whole difference between this and what shipped before.

The gap is measured once. Acting on one closes the marker it was measured from. A phase held on a gap is therefore not held again on the same interval by the pass after its Restart. resumeAfterSuspension closes the same marker for the hold it compensates, because the two compensations measure the same kind of interval from different markers.

This covers the control plane alone. A remote I/O module that holds its last commanded output while every controller is dark is #1683. Neither ADR 0008's armed hold nor ADR 0009's failState selector reaches it, because both are driven by software that is no longer running. ADR 0009 did reach the case as a decision. It lists ungraceful process death in its coverage table, assigns the backstop to the field-device watchdog, and rejects owning that watchdog because comm-loss fault states are configured on the I/O module and the product does not own them. #1683 revisits that assignment. Its grounds are that the documentation half of it was never written, so it amends 0009 and does not contradict it.