Skip to content

ADR 0053: An IOModule the probe could not measure is Unknown, not Offline

Status: Accepted Date: 2026-08-11 Issue: #1477

Context

IOModule.status.state is derived from an io-probe pod: a per-Controller one for a module that names a controllerRef, or a namespace-shared one for a controller-less network module (ADR 0021, ADR 0042). The probe answers two things about a device, and the reconciler mapped them faithfully: connected became Online, and reachable-but-not-connected became Fault.

Every other outcome also produced a state, and that is where the mapping stopped being faithful. A probe pod that was missing, that had no IP, that did not answer the query, or that had not yet loaded the module into its config all produced Offline — a claim about the device, made at the moment the DCS had lost the only instrument it had for making claims about the device.

The bench found the cost on 2026-08-11. A node shutdown left the probe pod in a terminal phase, nothing replaced it, and wago-750-352 read Offline for twenty-three hours. The coupler was answering the whole time: the unit runtime held its own Modbus connection and published input.0 and input.1 every second. An operator reading Offline, or reading the dcs io read error that named the probe pod's missing IP, is pointed at the network and the hardware. Both were fine.

Offline is not only a display value. Fault and Offline are the auto-HOLD set: on the transition into either from a non-degraded state, the IOModule reconciler annotates every Running unit allocated to a batch with Hold (ISA-88 Clause 7.4, equipment fault → automatic procedural response). So a monitoring sidecar dying could hold production on equipment that was running correctly, and the io-probe is exactly that — a sidecar. It commissions and watches; it is not in the control path. The runtime that drives the unit reaches the device over its own connection and goes on scanning while the probe is down.

Unknown already existed in the enum and already carried this meaning everywhere except in the code. docs/io.md documents a newly created module transitioning Unknown → Online, and the gateway maps an empty state to Unknown when it renders. Only the reconciler never produced it.

Decision

Offline is a measurement. Unknown is the absence of one.

The reconciler returns Unknown for every branch in which the probe did not answer for the module: no probe pod name, a pod that is missing, a pod in a terminal phase, a pod with no IP, a query that errored, and a module the probe has not loaded yet.

Offline is kept for the cascade that is genuinely determined — the Controller is missing, is not Joined, its node is not Ready, or a simulation Controller's heartbeat has gone stale. In each of those the DCS has no path to the device and knows it.

Fault is unchanged. It is the probe reaching the device's endpoint and being refused, which is a measurement and the one the auto-HOLD exists for.

Two consequences follow directly and are part of the decision:

  • Unknown is outside the auto-HOLD set. Losing the instrument is not an equipment fault, and holding a batch on it is a false trip.
  • Unknown is annunciated. The alarm generator raises the same AlarmTypeSystem alarm at the same Medium severity that the Offline branch raised, with a message naming the probe rather than a communication loss. Without this the correction would have traded a wrong annunciation for no annunciation, which is worse: nothing would say the plant had lost sight of a device.

Alternatives Considered

  • Leave the mapping and fix only the pod replacement. The pod-replacement bug in #1477 is real and is fixed independently: a probe pod in a terminal phase is now treated as absent and recreated. But the mapping is wrong on its own terms and has other ways to be reached — a probe still starting, a probe unschedulable on a zoned OT network (ADR 0042), a config that has not propagated. Every one of them would still have reported a healthy device as Offline and could still have held a batch.
  • Derive state from the unit runtime when one is scanning the device. The runtime is the strongest available evidence that a device is answering, and it is what made the bench diagnosis so slow — the DCS held the refuting evidence the whole time. Rejected here as a larger change than the correction needs: it couples IOModule state to Unit lifecycle, which ADR 0021 rejected for the connectivity signal, and a module may be referenced by ControlModules under several Units or under none. Unknown states the position honestly without inventing a second source of truth. Using the runtime as corroboration remains open.
  • A new state for "probe unavailable". Rejected as a needless enum widening. Unknown means exactly this and is already the documented pre-measurement state.
  • Keep Unknown unannunciated, on the grounds that it is the normal startup state. Rejected. A module sits in Unknown for the seconds its probe takes to start, which is the same window that raised an Offline alarm before this change, so the noise is unchanged. Staying silent afterwards would make a plant-wide loss of I/O visibility the quietest thing on the alarm list.

Note that ADR 0021 rejected an alternative described as "skip the probe for controller-less IOModules (state Unknown)". That is a different proposal. It removed the probe, so the module could never reach Fault and a device dropping mid-batch would never HOLD. Here the probe is unchanged and Fault still HOLDs. What changes is only what the DCS says when the probe itself cannot answer.

Consequences

  • An AlarmDefinition written against StateEquals: Offline no longer fires for a dead or starting probe. That is the intent, and a deployment that wants to act on lost visibility adds a StateEquals: Unknown definition. Definitions that carry exceptionAction: Hold should stay on Fault and Offline, where the DCS has actually observed something about the equipment.
  • kubectl get iomodules shows Unknown in cases that used to print Offline. The state now answers a different question — what the DCS knows, not what it guesses — so a runbook that reads Offline as "check the network" is correct for the first time.
  • The Offline branch of the auto-HOLD narrows to the cases where the Controller or its node is gone. The ISA-88 Clause 7.4 traceability in docs/compliance/isa88.md is unchanged in substance: a device that drops mid-batch still reads Fault from a working probe and still holds the Unit.
  • Unknown becoming reachable means the state can flap Unknown → Online during a probe restart. That is the honest sequence, and it is the one docs/io.md already documented for a newly created module.