ADR 0075: A health verdict counts only drivers that reach the plant¶
Status: Accepted Date: 2026-08-21 Issue: #1743 Related: ADR 0053 (the same distinction drawn one instrument over), ADR 0006 (the runtime health the failover path reads)
Context¶
A unit runtime holds a set of I/O drivers, and four separate things ask the
same question about them. Has this unit lost all of its I/O? The unit
controller's watchdog asks it to decide whether to Hold a running batch. The
runtime's own gRPC health check asks it to decide whether to report
NOT_SERVING. HealthStatus() asks it to summarise the runtime in one word.
dcs_runtime_healthy asks it so that an alert has something to fire on.
All four counted every driver, and all four were wrong in the same way.
Every unit-runtime pod is created with --protocol simulation. That is a
literal in createRuntimePod, and it is the only site in the tree. Every
runtime therefore carries a simulation driver, published in the driver health
table as the entry named default. A simulation driver's Connect sets its
connected flag, and only Disconnect at shutdown clears it. None of the five
fault types touches connectivity.
So default reports connected on every unit, always. Counting it does not
weaken a verdict. It decides one, because an unconditional true folded into an
OR makes the OR unconditional. all drivers down was unreachable on every
deployment this product has ever created. The watchdog Hold never fired. The
60-second NOT_SERVING grace period never expired. HealthStatus() could
never say not_serving. dcs_runtime_healthy was pinned at 1, so
DCSRuntimeUnhealthy could only ever fire on a failed read of the status
API. It could not fire on the total I/O loss it is named for.
The same predicate then read the other way. #1735 gave the watchdog alarm a clearing edge gated on the drivers being back. The predicate was already false. The clear therefore fired against a plant whose I/O was still gone, and it logged that the drivers had reconnected. That statement was untrue at the moment it was written.
The runtime was not the component in error. It reported healthStatus:
"degraded", which was exactly right. Nothing above it read that field.
Two facts settled the shape of the fix. The fact of it was never in question.
First, a simulation IOModule driver carries the identical defect one layer
down. It is the same simulation.Driver, connected on the same terms.
configureSimIOModule had already written this down for a different reason, as
"a simulation driver is connected regardless of whether its profile took"
(#1669). A unit that pairs a simulated module with a real device therefore
cannot report the real device's loss either. Removing only the default entry
from the count would have fixed the shape the issue was filed on and left that
case standing.
Second, a unit with no field drivers at all must abstain. A fully simulated unit has no field connection to lose. Concluding total loss for it would Hold every batch on every demo, training and simulation deployment, permanently, on no evidence. That is ADR 0053's finding in a different instrument. The DCS must not make a claim about a device at the moment it has nothing to make claims with.
Decision¶
A health verdict counts only the drivers whose connectivity is a measurement of something outside this process.
driver.IsFieldProtocol(protocol) is the one definition. It is a statement
about the protocol, and it says nothing about a driver's name or its position
in a table. Today it means every protocol except simulation. The default
driver is included on exactly the same terms as an IOModule driver, so a
deployment that ever gives a unit a real primary driver gets it counted without
a second rule.
Every verdict is then a three-way answer. A boolean cannot carry it:
| Field drivers | Connected | Verdict |
|---|---|---|
| none | — | abstain: serving, healthy, do not act |
| some | at least one | serving, healthy, do not act |
| some | none | total loss: NOT_SERVING, unhealthy, Hold |
Four sites ask the question and consume that answer. checkRuntimeHealth and
updateRuntimeHealthMetrics are in the unit controller. HealthCheck and
HealthStatus are in the adapter.
The driver health table is not filtered. DriverHealthSummary still
publishes the default entry and every simulation IOModule, and
dcs_runtime_driver_connected still carries a series for each. An operator
reading the driver table wants the whole table. Only the verdicts taken over
that table are restricted. That is the same separation #1669 drew between
connected and configError.
An unrecognised protocol counts as field evidence. That is the fail-safe default. The worst it does is let a watchdog conclude loss and Hold a batch. The opposite default would silently re-create this defect for whatever driver was added next.
Alternatives Considered¶
Exclude the default entry and count simulation IOModule drivers. This is
the mechanical reading of the issue, and it is one predicate. It was rejected
on two grounds. It needs a discriminator the status wire does not carry, so the
operator would either match the literal name "default" across a process
boundary or gain a new field for the purpose. It also leaves the same defect
standing wherever a unit pairs a simulated module with a real device, which is
the shape the capture stack runs and the shape #1743 was observed on.
Delete the default driver. The cleanest end state, and the widest change.
a.driver is the fallback for any address with no <module>: prefix
(resolveDriver), it backs the Adapter's own DeviceDriver methods, and
ReplayLastNetwork builds the FB runtime on it. None of those call sites is
nil-safe. (The third of them is gone since #1748, which deleted
ReplayLastNetwork along with the dead state file it read. The alternative
below is unchanged by that: the other two call sites are the load-bearing
ones.) It also changes what the top-level protocol and connected fields
of GET /api/v1/status mean. The vestige is real and worth removing, and
removing it is not a safety fix.
Have the operator consume healthStatus and stop recounting. The runtime
already computes a verdict, so this looked like the layering answer. It does
not work unmodified. HealthStatus() counted the same simulation drivers, which
left not_serving equally unreachable. Its degraded means any driver down,
and Holding on that is a wider contract than the documented one. Once
HealthStatus() is corrected, the operator recounting from the same table it
is already given costs nothing and keeps the two able to disagree visibly.
Gate the Hold behind a chart flag. Rejected as a disabled state that reads as covered on a dashboard, and as a flag that would have to be defaulted on later anyway.
Consequences¶
This arms a Hold path that has never once fired. Correcting the predicate
is a behaviour change on every deployment, which is the rule #1697
established. The survey was therefore run before the predicate was wired, and
kept as TestShippedUnitsWatchdogArming. It derives each shipped Unit's driver
set through resolveIOModules, the reconciler's own function, then drives
checkRuntimeHealth over the status payload that Unit's runtime would publish
with its plant gone. One of the nineteen shipped Units reaches the Hold
verdict. The rest have no field driver and abstain.
The hazard is not novel even for that one. notifyAffectedUnits already issues
Hold to every Running batch-allocated Unit when an IOModule reaches Fault
or Offline, under ISA-88 Clause 7.4. What this adds is a second instrument
for the same response. It reads the runtime's own connection, where the
existing one reads the io-probe sidecar's. That distinction matters
precisely because the probe is not in the control path, and ADR 0053 is the record of it being wrong about a
device that was answering the whole time.
healthStatus changes what it can say. A runtime that has lost all its
field I/O now reports not_serving where it used to report degraded, because
the default driver no longer holds the summary up. The field is published in
GET /api/v1/status and had no in-tree consumer other than that response.
A fully simulated deployment has no watchdog cover, and says so. It had none before either. The difference is that the abstention is now deliberate and documented, where before it was a side effect of a driver that could not disconnect.
The tests that covered these predicates were all built on states the product
cannot reach. Every fixture gave the default driver a value no deployment
produces. It was disconnected in some, and speaking modbus in others. The
predicates read as exercised while the only reachable state went untested,
which is why the defect survived from the driver abstraction that introduced
--protocol simulation. Every fixture now starts from the shipped shape, with
default connected and speaking simulation.