ADR 0078: A partial loss of field I/O is annunciated like a total one¶
Status: Accepted Date: 2026-08-21 Issue: #1754 Related: ADR 0075 (the total-loss verdict this extends), ADR 0009 (why a faulted block does not halt the program)
Context¶
A unit lost its field bus mid-batch on the bench and the control plane said
nothing for the whole outage. The runtime knew within five seconds and logged a
dial failure on a loop. The IOModule stayed Online with lastSeen advancing,
the Unit stayed Running with runtimeReady true, no condition transitioned,
and no alarm was raised.
The one annunciation that did arrive came from a step timeout on a guard two steps downstream, which told the operator that a process value had not come down. That was true. It was also the wrong fault, and it was the operator's whole picture.
Three separate things were wrong, and only the first was closed by #1743.
The watchdog asked whether EVERY field driver was down. A unit with two
field buses that loses one kept Running with runtimeReady true, no
condition and no alarm, while half its I/O was gone. The bench happened to cut
the only bus, which is why #1743 catches that particular event and closes
nothing about the general one.
A failed field WRITE produced no alarm at all. Both of the shutdown phase's
writes failed silently before the guard timed out. A block that errors is
skipped and the program keeps regulating every healthy output (ADR 0009). It
reports Degraded with FaultedBlocks populated, and that reaches driver
health, dcs_cm_program_degraded_total, the CM health topic and the HMI. It
reached no Alarm. The physical layer raised
exactly four alarm kinds and none of them was sourced from a degraded program.
The signal that did arrive was an accident of recipe shape. It came from a guard that reads. A longer dwell, or a step whose guard does not read, and the event is silent at the phase level too. The phase-level bound is the remaining dwell plus that guard's own timeout, so it is a property of the recipe rather than of the product.
The sharp edge is that a plant losing SOME of its I/O was in a worse epistemic position than one losing ALL of it. Total loss already Held the batch and raised Critical. Partial loss produced a batch that kept running, writing to the channels that still answered and failing silently on the ones that did not.
Decision¶
Partial loss annunciates and Holds. A degraded program annunciates and does not.
Partial field-driver loss¶
A reading of the runtime's field drivers now carries the disconnected ones by name. Total and partial are then the same measurement read two ways, which is what keeps them from disagreeing.
| Field drivers | Disconnected | Verdict |
|---|---|---|
| none | — | abstain (ADR 0075) |
| some | none | healthy |
| some | some | Hold, High, condition FieldDriversDisconnected |
| some | all | Hold, Critical, condition AllDriversDisconnected |
High sits one tier below the total-loss Critical so the two are
distinguishable in an alarm list. Both alarms name the drivers that went,
because an operator answering one is being told a bus is gone and needs to know
which.
The Hold reaches the whole unit, and that IS reference granularity.
reconcileIOConfig builds a runtime's I/O driver list out of the IOModules
that unit's own ControlModules reference. Every field driver in a runtime is
therefore bound by one of them by construction, and there is no wider set to
narrow down to.
The two extents replace each other. They raise under different
GenerateName prefixes, watchdog-<unit>- and io-loss-<unit>-, and each
clears the other's on the same pass. A raise dedupes against an active alarm on
its own prefix (#1735). One shared prefix would therefore keep the High
annunciation through a loss that had become total, and the operator would be
told the smaller thing was still true. It reads the same way going the other
direction. One bus coming back is not recovery.
A loss that moves WITHIN one extent refreshes the standing alarm. The
dedupe matches on the source and the prefix and never on the message (#1760).
A later pass that is also partial therefore raises nothing while the alarm
stands Active, and the alarm goes on naming the buses it fired on.
RuntimeHealthDegraded is rewritten every pass and does carry the current set.
The Unit's status and the alarm the operator is answering disagreed for the
whole outage. The message alone is rewritten. Identity, acknowledgement and
creationTimestamp belong to the one outage, and the interval a runbook reads
from creationTimestamp to clearedAt has to keep measuring it. Severity is a
pure function of the prefix here, so there is nothing to re-grade and none of
the severity-escalation machinery rejected below is needed. Three field drivers
are required to reach the case at all. With two, a growing set crosses to
total, the prefix swaps, and the paragraph above covers it.
The annunciation runs whether or not the unit is still Running. The Hold
cannot: the state machine is rebuilt each pass from the persisted state and the
watchdog's own Hold leaves Running, so a second Hold command would be
refused. What can change under a unit that is already Held is the extent of the
loss.
dcs_runtime_healthy moves with the ruling. It read "at least one field
driver connected" and now reads "every field driver connected, or none to
lose". A gauge that stayed at 1 through a Held unit with half its I/O gone
would be an alert reading all-clear on a plant that had stopped, which is
#1646's failure shape. It now says what the runtime's own HealthStatus()
says.
The runtime's own gRPC health check is unchanged. It reports SERVING
through a partial loss and NOT_SERVING only after total loss outlasts the
grace period. Restarting the pod does not reconnect a cable, and the split is
deliberate: the control plane annunciates and Holds, and the kubelet is not
asked to do anything about it.
A degraded control program¶
ControlProgram.status.faultedBlocks carries the names the runtime reports,
and ControlProgramAlarmReconciler raises a High equipment alarm naming them
when the set is non-empty, clearing when the scan comes clean or the program is
deleted.
No batch effect. A degraded program is still regulating every healthy output, and halting it would freeze those too (ADR 0009). The alarm is the whole of the change.
The condition is the faulted SET. Reading it directly means the alarm cannot be raised with nothing to name. An empty set on a status the controller has never synced is not a clean scan, so it clears nothing.
It is a reconciler of its own. AlarmGeneratorReconciler.Reconcile
dispatches by trying each kind's Get against the request name, so two objects
of different kinds sharing a name in one namespace shadow each other. A
ControlProgram is named after the control module it regulates, and a dispatch
that cannot see kinds is not the place to add a third one. It shares the
generator's create, re-annunciate and clear path. That is what keeps its alarm
behaving like every other generated alarm.
Alternatives Considered¶
Annunciate partial loss and never Hold. This keeps ADR 0075's abstention
argument intact: a unit can be commissioned against I/O that is not installed
yet, and a false Hold is expensive. It was rejected because the product already
promises the opposite one layer over. IOModuleStateFault raises a High
equipment alarm and notifyAffectedUnits Holds every Running allocated Unit
whose ControlModules reference the module, attributed in the code to ISA-88
Clause 7.4. Choosing "never Hold" here would have meant retiring that, and
leaving the two disagreeing was not on the table. The commissioning case is
answered by the abstention that already exists: a unit with no field drivers
reaches no verdict at either extent.
Hold only when the running phase uses the lost channel. The most precise answer, and the most machinery: it needs the executing phase's live tag set resolved against driver ownership, and the answer changes step to step within one batch. It is deferred and not dismissed. The unit-level rule is a superset, so narrowing it later removes Holds and adds none.
Raise both extents under one prefix and escalate the severity in place. One alarm per condition per unit is tidier to read. It needs a severity-escalation path through the ISA-18.2 state machine that nothing else in the product has, and the dedupe interaction above is a trap that has already caught this code once.
Re-annunciate when the loss grows. ISA-18.2 separates the alarm condition
from acknowledgement, and an operator who has acknowledged a partial loss has
accepted a condition that has not ended. Re-firing on every membership change
would hand one outage a growing stack of annunciations, or take an
acknowledgement back for a fact the operator can already read off the refreshed
message. It was rejected for the reason the product already gives twice, for
the DeviceHealth escalation in AlarmDefinitionReconciler and for a re-graded
server alarm: re-annunciating a standing alarm is how a horn stops meaning
anything. The escalation that does deserve a fresh annunciation is a partial
loss becoming total, and that one raises under the other prefix.
Source the degraded-program alarm from the CM health MQTT topic. The
runtime already publishes it retained, and the gateway already subscribes. It
was rejected because an alarm is a Kubernetes object and the subscriber that
would raise it holds no client, while ControlProgram.status is already synced
from the runtime every ten seconds by a controller that does.
Consequences¶
A partial loss now stops production. That is the ruling. A unit that loses one of several buses Holds its batch, where before it ran on. A deployment with modules wired for I/O that is not installed yet should leave those units without field drivers. That is the abstention ADR 0075 describes, and it is the same advice that already applied to total loss.
DCSRuntimeUnhealthy fires on partial loss. The alert's condition is
unchanged and the gauge under it now moves. A deployment that wants the old
threshold has dcs_runtime_driver_connected per driver to build it from.
Two more alarm kinds reach the operator. io-loss-<unit>- at High from the
unit controller, and alarm-controlprogram-<name>-equipment at High from the
alarm generator. Both clear on their own.
The bench event now annunciates twice and early. The dead bus raises the I/O-loss alarm on the next watchdog poll, and the failing writes raise the degraded-program alarm naming the blocks. Neither waits for a downstream guard to time out, and neither names a process fault.
A degraded program annunciates for every faulted block, including causes that are not I/O. A block that errors for its own reasons raises the same alarm. That is the intended breadth: the alarm says a loop stopped controlling, and it names the block so the reader can tell which kind of failure it is.