Skip to content

ADR 0042: Network io-probe placement is declared per IOModule as a nodeSelector, and the shared probe partitions into one pod per distinct selector

Status: Accepted Date: 2026-08-08 Issue: #1329

Context

ADR 0021 monitors a controller-less network IOModule with a namespace-shared io-probe pod carrying no nodeSelector, on the written rationale that a network device is reachable from any node. On a zoned OT network that rationale is false. The hardware bench's field switch has no uplink to the IT switch (deploy-bench decision 2026-08-04), so field devices are reachable only through the device nodes' field bonds — today only ipc-1. A probe pod scheduled on a control-plane node has no route to the field subnet, and the IOModule reads Fault forever with nothing wrong except pod placement. (A pod that cannot schedule at all reads Unknown since ADR 0053; this paragraph said Offline/Fault when it was written.) Any deployment with IEC 62443 zoning — which our own docs recommend — reproduces this.

The commissioning workaround (#271) registered the bench coupler with controllerRef: ipc-1, pinning the per-Controller probe to the one node with field reach. That couples the record to one machine: when ipc-2's field bonds land the coupler is reachable from both IPCs, naming either is wrong-by-omission, and a failover drill that kills ipc-1 takes the probe down even though ipc-2 still reaches the device.

Decision

A controller-less network IOModule may declare spec.probePlacement.nodeSelector. The IOModule reconciler groups the namespace's controller-less network IOModules by that selector and runs one shared probe pod per distinct selector, each carrying its group's selector; modules with no placement share the unconstrained fixed-name pod exactly as before.

  • The declaration is a selector over node labels, not a node name. The labels are deployment-layer vocabulary: the layer that joins nodes (ADR 0004) knows which of them carry field-network interfaces, and labels them (e.g. dcs.io/field-reach: ot). A record naming a reach label instead of a hostname stays correct when a second node gains field reach, and the scheduler moves the probe when its current node dies — the redundancy the workaround lacked, without any probe fan-out or state merging.
  • Grouping keeps the ADR 0021 sharing economics. Modules with the same selector share one pod and one ConfigMap, named network-io-probe-<hash> where the hash digests the normalized selector. The empty selector keeps the fixed network-io-probe name, so flat networks see no change at all — same pod name, same unconstrained placement, and an unset field on every existing record.
  • A placement edit re-homes, never mutates. Because the group name hashes the selector, editing a module's placement lands it in a differently-named group; the reconciler prunes every probe Pod and ConfigMap (matched by the network-probe labels) that no current group claims. Pod nodeSelector is immutable, so this sidesteps in-place mutation entirely; a pod whose selector nonetheless disagrees with its group (hash collision, hand edit) is deleted for recreation.
  • The refusal condition matches the consume condition. Placement is read only on the controller-less network path — a controller-bound probe is pinned to its Controller's node — so a CEL rule refuses probePlacement on any IOModule that names a controllerRef (transitively covering simulation, which requires one). The field can never sit inert on a record (the ADR 0040 discipline).

Alternatives Considered

  • Zone-derived placement. Have the site/zone model decide where probes run. Rejected for now: ADR 0031's taint excludes workloads from device nodes, it does not express which field networks a node reaches, and no other object models reach today. Building that model for this fix would be speculative; the selector composes with it later — when a zone-derived reach label exists, the placement names it.
  • Probe-per-eligible-node with merged state. Run the probe on every matching node and merge verdicts. Rejected as redundancy machinery the selector already substitutes for: a selector matched by several nodes reschedules the single probe when one dies, which is the availability the bench needs. Multi-vantage reachability (device visible from node A but not node B) is a diagnosis surface nobody has asked for, and it would complicate the Clause 7.4 fault signal with quorum questions.
  • Keep the controllerRef workaround. Rejected by the issue itself: it couples the record to one machine, misstates the topology once two nodes carry field reach, and turns a single-node failure into a false device fault.
  • Placement on the Site or namespace instead of the IOModule. One selector for all of a site's controller-less probes. Rejected: a site can front devices on differently-reachable networks at once (an OT zone and a lab VLAN), and the module is where protocol and address — the other locality facts — already live.

Consequences

  • The bench coupler is re-registered controller-less with a probePlacement naming a field-reach label the deploy layer applies to ipc-1 (and to ipc-2 when its bonds land) — tracked with the #271 commissioning work. The #942 drill then costs the probe a reschedule, not the device a false Fault.
  • The CRD change is additive and backward compatible: unset probePlacement is byte-for-byte today's behavior.
  • The reconciler now sweeps stale probe groups by label. Probes created before this change carry the same labels and prune identically, so upgrades strand nothing.
  • status.state for a placed module reads through its own group's pod; a group whose pod is still scheduling reads Offline, unchanged from the fixed-name probe's bootstrap behavior.
  • The declared-selector shape leaves multi-vantage probing open: a future need can widen probePlacement without another schema break.
  • Amended (#1480): the resolved pod name is published on IOModule.status.probePodName. Naming a group by hashing its selector means the serving pod can no longer be derived from the IOModule spec alone by anything outside this reconciler, and the gateway's dcs io read/dcs io write path was still deriving it from the fixed ADR 0021 name — so a placed module was reported as having no probe pod while its probe ran, and the fixed-name pod is not even present once no placement-free module claims it. The reconciler that creates the pod now records its name, readers take it from there, and what the hash is computed over stops being a second package's problem. This mirrors Controller.status.ioProbePodName, which the same path already trusted for controller-bound modules.