Skip to content

ADR 0068: A device fail-safe is declared and never defaulted

Status: Accepted Date: 2026-08-18 Issue: #1683 Related: #1349 (the cold-boot drill that found it), ADR 0009 (which this amends), ADR 0008 (the armed hold), #1689 (the output readback this depends on), ADR 0071 (which amends this, giving the timeout a floor)

Context

During the cold-boot drill on the bench, five computers were cut at the PDU while a batch was executing with a PID loop at 60%. The OT rack takes its own cord, so the field zone stayed powered while every controller went dark. The founder read the AO loop at the field terminals during the dark window: 13.58 mA, which is 60% of the 4-20 mA span. The coupler held its last commanded output with no controller behind it, and would have gone on holding it indefinitely.

Both fail-safe mechanisms we ship are executed by a process that no longer exists at the moment they are needed. ADR 0009's failState is written by WriteSafeOutputs after the scan loop stops. ADR 0008's armed hold is a local timer inside the runtime. Neither survives the machine.

ADR 0009 saw this path. Its coverage table names ungraceful process death and assigns it to "the field-device watchdog", and it explicitly rejected owning that watchdog:

Make the device watchdog a product feature rather than a documented deployment requirement. Rejected: comm-loss fault states are configured on the I/O module itself (vendor-specific), which the product does not own.

This ADR revisits that assignment. What justifies reopening it is that the documented half was never written. Seven places across docs/ tell a deployment to configure a comm-loss fail-safe, including the IEC 62443 SR 3.6 compliance row, and no page names a register, a WBM page, or a recommended timeout. A plant following our documentation to the letter still inherits whatever its coupler shipped with, which is what the drill measured.

What the hardware can actually do

WAGO's manuals for the 750-352 and its successor the 750-362 contain no occurrence of "substitute" or "fail-safe" in 610 pages. There is no substitute-value register block on this family. The only output-affecting mechanism is the Modbus watchdog, it acts on the whole node, and its only action is to set every output to zero.

That is narrower than the industry norm, and the norm is worth stating because integrators arrive with it. PROFINET carries per-device substitute values in the GSDML. CIP carries per-channel Fault Action and Idle Action in the Configuration Assembly, delivered at connection establishment. Modbus standardises none of it. Our one shipped device is the least capable member of the family.

Measured on the live 750-352, three things the manual does not say:

  1. A cleared output settles at 4 mA. Raw zero on a 750-554 is the bottom of the span. NAMUR NE 43 reserves <=3.6 mA and >=21 mA for faults and calls 3.8-20.5 mA valid measurement, so a cleared analog output is a legitimate 0% reading and annunciates nothing downstream.
  2. The process image is overwritten. The old commanded value never returns on its own. This kills the flapping hazard the issue worried about: a cleared output stays cleared even as other traffic restarts the watchdog.
  3. The trigger mask does not select what keeps the watchdog alive. With FC3 excluded from the mask, once-a-second FC3 reads held the watchdog open for twenty seconds on a five-second timer. The mask governs only what may restart it after a timeout.

Decision

spec.failSafe on IOModule declares what the device does with its outputs when it stops hearing from the master. An unset field writes nothing to the device. The product then reports the state the device is actually in.

Five parts.

1. The vocabulary is what a shipped driver can honour. hold and clear, node-wide. substitute and per-channel are absent until a profile can apply them, because offering a capability nothing executes is the mistake ADR 0065's issue records. Adding a word later is the cheap direction.

2. Unset means the product writes nothing. Defaulting would change what a running plant's outputs do on upgrade without anyone asking, and on a 4-20 mA card the change is invisible downstream. This is the compatible answer and it is deliberately not the quiet one.

3. Unset is reported. status.failSafe carries what the device is configured to do, read back from it, and the FailSafeDeclared condition goes False when nothing was declared. An operator cannot otherwise tell a hold somebody chose from a hold a coupler shipped with, and the drill proves those are the same observation from the field. The other families get this property for free, because their configuration rides on a connection. Modbus has no connection to hang it on, so the gap can exist here and has to be named.

4. FailSafeDeclared and FailSafeApplied are two conditions. They fail differently and one would hide the other. Declared-but-not-applied is the dangerous state: a record promising protection the plant does not have reads exactly like a record describing protection it does.

5. Recovery defaults to latch. A cleared output is silent, so the coupler refusing process data is what drives the IOModule to Fault and tells the plant. The alternative leaves a node sitting at 4 mA with every surface reading healthy.

Where it is applied

The io-probe owns this, and the unit runtime does not. A device has one watchdog and needs one writer. An IOModule compiles into every referencing Unit's runtime, and several of those arming the same coupler would fight over its registers. Exactly one io-probe serves each IOModule.

The declaration is written only when it changes. Every telegram resets the watchdog whatever the mask says, so re-arming on a fifteen-second cycle would hold it open forever and protect nothing.

What this measures, stated plainly

Because any traffic feeds the watchdog, it measures whether this product is still running against the device. It does not measure whether control is still writing outputs. It therefore fires when the controller node dies and takes the runtime and the io-probe with it. That is the case the drill produced, and the case ADR 0009 could not reach. It does not fire when the node lives and only the program stops, and it does not need to: ADR 0009's failState covers that path.

Anyone reading the CRD will assume the opposite, so the field documents it.

Alternatives Considered

  • Leave it a documented deployment requirement, which is ADR 0009's answer. Rejected because the documentation was never written. Seven pages say to configure a comm-loss fail-safe and none says how, so a plant following them exactly still inherits its coupler's default. That is what the drill measured.

  • Default an unset field to clear. Rejected. It is the honest reading of the gap, and it changes what every existing modbus IOModule does on upgrade without anyone asking. On a 4-20 mA card the change is also invisible downstream, because a cleared output sits at 4 mA inside the valid band.

  • Default an unset field to hold. Rejected. It changes no behaviour, and it means we actively disarm a watchdog somebody configured by hand at the WBM. A product that silently undoes a plant's own safety configuration is worse than one that leaves it alone.

  • Ship substitute in the enum and refuse it on every current driver. It is the word a process engineer reaches for first, and PROFINET and CIP both carry it. Rejected because no device we address can honour it, so every declaration would fail. Adding a word later costs little. Removing one that six layers have started offering is the migration ADR 0065's issue records.

  • Carry the field per channel now, so the shape is ready. Rejected for the same reason. The WAGO watchdog acts on the whole node. A per-channel spec would therefore have to refuse any non-uniform declaration, and that shape promises something the hardware cannot do.

  • Put the declaration in spec.options. Rejected. It is the existing escape hatch, and a safety-relevant statement living in a free-text string map that nothing validates is the complaint this issue opens with.

  • Apply it from the unit runtime. Rejected on cardinality. An IOModule compiles into every referencing Unit's runtime, and a device has one watchdog. Several runtimes would fight over its registers.

  • Set the trigger mask to FC6 alone, so only control writes feed the watchdog. This was the design at one point, and it would have let the timer mean "control is still writing outputs" while ignoring the io-probe's discovery reads. Refuted by measurement: with FC3 excluded from the mask, once-a-second FC3 reads held the watchdog open for twenty seconds on a five-second timer. The mask governs only what may restart it after a timeout.

  • Re-apply the declaration on every probe tick, for convergence. Rejected. Every telegram resets the watchdog, so a fifteen-second re-arm would hold it open forever. The declaration is written when it changes, and a failed write is retried because it is not recorded as applied.

Consequences

  • Code: api/physical/v1alpha1/iomodule_types.go (spec, status, two conditions, two CEL rules), pkg/driver/failsafe.go (the optional FailSafeConfigurer), pkg/deviceprofile (ModbusFailSafe, ModbusRegWriter), pkg/deviceprofile/wago750/failsafe.go (the registers), pkg/driver/modbus, internal/ioprobe, internal/controller/physical.
  • Two device behaviours are encoded that the manual gets wrong. Each of them silently produces a plant with no protection. The watchdog starts through 0x1003, because a live coupler refuses 0x1007. Disarming needs 0x100A <- 0 as well as the stop register, because 0x100A=1 re-arms on the next telegram.
  • docs/ha-failure-modes.md was wrong and is corrected. It claimed the module "drives its outputs to their configured fault/safe state", and on this hardware there is no configurable fault state.
  • ADR 0009 is amended. Its failState selector still owns the program-halt path. This ADR owns the machine-death path that ADR 0009 assigned away.
  • Amended by ADR 0071, which gives the timeout a floor. This ADR left it a free-text duration, and 100ms was admitted. The question was raised in #1683's own body and answered in #1701. The floor comes from the sentence above about what the timer measures: because any telegram feeds the watchdog, the io-probe's own fifteen-second readback is a heartbeat, and a deadline under that cadence cannot expire on purpose.
  • Not in scope: per-channel fail-safe, substitute values, and any protocol beyond Modbus. Each waits for a device that can honour it.