Skip to content

ADR 0071: A watchdog deadline is measured against our own heartbeat

Status: Accepted Date: 2026-08-18 Issue: #1701 Related: ADR 0068 (which this amends), #1683 (where the question was raised and left unanswered), #1700 (what a spurious fire costs to undo)

Context

ADR 0068 gave the IOModule a spec.failSafe.timeout. Two CEL rules govern it. One requires it for clear and the other forbids it for hold. Neither bounds it. timeout: 100ms was admitted, and the WAGO profile accepts that value exactly, because 100 ms is one tick of the watchdog register's own resolution.

The issue that shipped the feature raised this in its own body and closed without answering it. What sharpens it is the same bench work that produced the feature.

The watchdog is fed by any traffic at all. The trigger mask does not select what keeps it alive. Measured on a 750-352, reads whose function code was masked out held the timer open for twenty seconds on a five-second setting. The quantity a timeout has to clear is therefore not the control scan interval. It is the longest gap between the device hearing from anything.

A spurious fire is silent. A cleared 4-20 mA output settles at 4.00 mA. NAMUR NE 43 calls that valid measurement, so nothing downstream alarms. Under latch the coupler refuses process data and the IOModule reaches Fault, which is the only annunciation there is. Under resume the node sits at 4 mA with every surface reading healthy.

Recovery is not free. Getting the outputs back takes more than one write (#1700). A watchdog that fires on a routine gap does not re-establish itself on the next scan.

The obvious floor is not reachable

The issue proposed deriving the floor from the scan interval. That cannot be done, and the reason is the one ADR 0064's fifth amendment already records about IOModule conditions. An IOModule compiles into every referencing Unit's runtime, so the relationship to a scan interval is one-to-many. It travels IOModule to Units to ControlPrograms, and a CEL rule sees one document.

What we found instead

Reading the probe loop settled it. syncFailSafes reads every connected device's watchdog registers on every probe tick, and the tick is 15 seconds. That read is a telegram. So the io-probe feeds the watchdog of every device it serves, whether or not anything else is talking to it.

That inverts the question. Scan traffic can only make the gap between telegrams shorter, and it stops entirely when a program stops, when a Unit goes idle, or when no Unit references the module. The io-probe's cycle is the one heartbeat that never stops. It is therefore the binding quantity. It is also a property of the product, which is what makes it visible to a rule reading one document.

It also says what a short timeout can and cannot mean. A 2s deadline does not buy a two-second response, because our own next read resets it long before it expires. The only behaviour it adds is a trip when one read runs late. The device cannot notice our absence sooner than our own heartbeat.

Decision

A declared fail-safe timeout is refused below three probe cadences at admission. Its margin against the cadence the device actually got is measured and reported at reconcile.

Both, because they answer different questions and neither can do the other's work.

1. The floor is FailSafeTimeoutFloor, 45 seconds, and a CEL rule on IOModuleSpec refuses anything below it. It is FailSafeProbeCadence times FailSafeTimeoutFloorCycles, and neither number is invented here. The cadence is the io-probe's own --probe-interval default, taken from the same constant so the two cannot drift. Three cycles is the fieldbus convention. PROFINET derives a device watchdog from about three update cycles and CIP derives its own from RPI times a multiplier. Three cycles means the io-probe may be two full ticks late before the device decides we are gone.

CEL carries it because the other two rules on this field are CEL, and because a CEL rule then covers every road into the API at once: kubectl, Flux, the gateway, and POST /api/v1/apply. It applies on update as well as create, which is the road a plant would actually take to a 100 ms watchdog.

2. The floor is blunt and says so. It cannot express the real constraint, which is per-plant. What it rules out is the typo and the number picked off the scan rate. Those are the numbers that are wrong for every plant.

3. FailSafeTimeoutMargin is a third condition, reporting the measured margin. The io-probe records how long each device went between its own successive reads and reports the worst over a rolling window. The reconciler compares that to the declared timeout. True names both numbers. False says the watchdog fires on a routine gap, which is not the loss it was armed against. Unknown is the honest answer before two reads have landed.

4. The margin refuses nothing. The device is already armed. Disarming a plant's protection because our own loop went slow would be the wrong repair, and it would assert a decision about the plant on evidence about us.

5. The measurement reads the io-probe's own traffic and nothing else. A read that failed, and a device that was not connected, drop the anchor without recording a gap. A device that stopped answering went unheard because it is unreachable, and connectivity already has its own condition. Charging an outage to the timeout margin would report one fault on evidence of another, which is the #1669 mistake.

6. noMargin is a verdict as well as a condition. ADR 0068 put the verdict in the gateway so the CLI and the browser could not derive these distinctions separately. A module whose declaration landed perfectly and whose deadline is too short reads as applied on the FailSafeApplied condition alone, and that paints it green on every surface. The new verdict outranks applied and is outranked by mismatch, because a deadline that is not on the hardware is not the number to go and fix.

Why the margin is measured and not modelled

The floor rests on the io-probe reading each device once per cadence. The probe loop is serial over every module on its Controller, and a module that is not connected costs a dial timeout. On Modbus that is five seconds by default. Six unreachable couplers on one Controller push the effective cycle past 45 seconds, and a declaration that cleared admission comfortably is then under water. The premise the floor rests on is true of an idle Controller and can stop being true of a loaded one. Shipping the floor without measuring what it assumes would be a gate passing for the wrong reason.

Alternatives Considered

  • A static floor alone. Rejected. It asserts a margin on a cadence the product does not guarantee under load, and the failure is invisible: the declaration is applied, the condition is green, and the node clears its outputs the first time the loop runs long.

  • A reconcile-time check alone, refusing nothing. Rejected. It arrives after the document is accepted, and the numbers it would report first are the ones a floor can refuse outright. It also leaves 100ms a legal declaration, which is the thing the issue was filed about.

  • Derive the floor from the referencing Units' scan intervals. Rejected as unreachable, and once the probe loop was read, as unnecessary. It is one-to-many, a CEL rule sees one document, and scan traffic can only shorten the gap. The binding quantity is the heartbeat that never stops.

  • Make the io-probe feed each watchdog at the declared timeout divided by three, so a plant can have the fast watchdog it asks for. Genuinely better in one respect: the declaration would then be honoured, where today it is merely permitted. Rejected for this issue. It rewrites the probe loop into per-module timers, and it multiplies traffic on exactly the deployments that are already slow. It also does not reach the case it is for. An io-probe pod restart is bounded by the kubelet. A two-second watchdog would therefore clear outputs on every rolling update, and the floor would still have to clear a pod restart. Worth reopening if a plant ever needs a deadline this feature cannot express.

  • Bound the timeout from above as well. Rejected here. The WAGO register ceiling is 6553.5 s and the driver already refuses past it. That is one device family's limit, and the CRD is protocol-agnostic. A ceiling belongs in the profile that has one.

  • Fold the margin into FailSafeApplied. Rejected for ADR 0068's own reason for splitting Declared from Applied. A declaration that was applied perfectly and is mistimed fails differently from one that never reached the hardware, and folding them lets the second hide behind the first.

  • Report the running maximum gap, with no window. Rejected. A running maximum never comes back down. One slow cycle during bring-up would then report a device as marginal for the life of the pod, and a verdict nobody can clear is one nobody reads.

Consequences

  • Code: api/physical/v1alpha1/iomodule_types.go (the constants, the CEL rule, the third condition), cmd/io-probe/main.go (the flag default now reads the constant), internal/ioprobe (readgap.go, the measurement in syncFailSafes, WorstReadGap on the reported state), internal/controller/physical/iomodule_failsafe.go (the condition), internal/gateway (the DTO field and the noMargin verdict), cmd/dcs/internal/cmd/failsafe.go and the Diagnose panel (the surfaces).
  • The shipped example declared timeout: 2s, and docs/io.md taught it. Both are corrected to 60 s, which is the floor plus a fourth cadence. The corpus carried the exact hazard the issue describes, for the whole of the feature's one-day life.
  • An existing IOModule below the floor becomes unpatchable until its timeout is raised. Nothing outside examples/ declared one, and the field is a day old.
  • --probe-interval moving now moves the floor's justification. The flag survives for testing and the operator never sets it. TestProbeIntervalDefaultIsTheDeclaredCadence holds the default against the constant. The margin condition reads the measured cadence, so an overridden flag is reported and never assumed away.
  • Not in scope: an upper bound, a per-channel deadline, and any protocol beyond Modbus. Each waits for a device that can honour one.