Skip to content

ADR 0031: Device nodes are tainted unit-runtime-only — the deployment layer applies dcs.io/role=device-node:NoSchedule, and only the control-zone pods the physical operator creates tolerate it

Status: Accepted Date: 2026-08-03 Issue: #1167

Context

A DCS cluster mixes two classes of node. Platform nodes host the operations tier: operators, the gateway, the MQTT broker, the historian and its PostgreSQL database. Device nodes are controller compute, the machines bound to Controller resources and running unit-runtime pods that execute the deterministic FB scan against field I/O. In the ISA-95/Purdue frame the historian is a Level-3 operations function while unit-runtime is Level-1/2 control, and our IEC 62443 traceability claims zone segmentation with a per-zone service topology (docs/compliance/iec62443-4-2.md, CR 5.1). ADR 0026 already rejected collapsing the zone boundary to a VLAN tag.

Nothing enforced that split at the scheduling layer. On the bench (2026-08-03, after the #1147 control-plane re-taint) the scheduler placed the historian's CNPG database and the MQTT broker on a device node, and the node-local storage class made the placement sticky. That is a compliance posture problem twice over. Network segmentation between zones is moot when both zones share a kernel: a compromise or fault of the L3 database host is a compromise of the control host. It is also a resource-contention problem, because bulk-storage compaction and query load competing with deterministic control scans is exactly the essential-function degradation SR 7.1/7.2 exist to prevent.

The founder's position, adopted by #1167: running the historian on controller compute "should never be possible, not merely discouraged."

Two mechanical gaps let it happen. First, no mechanism marked device nodes as unit-runtime-only. Second, the chart could not express the fix even manually. historian.nodeSelector and mqtt.nodeSelector existed, but no component had a tolerations value, and the CNPG Cluster template rendered no spec.affinity at all, so on any cluster whose platform nodes carry taints the services could not be pinned anywhere.

Decision

The zone model gets a scheduling half, split along the ADR 0004 line (node lifecycle belongs to the deployment layer):

  1. The deployment layer taints device nodes dcs.io/role=device-node:NoSchedule when it joins them, in the same layer that owns node join (ADR 0004). This mirrors the inverse taint a hardened cluster already carries on its control plane.
  2. The product hard-codes the toleration into exactly the two pod kinds that belong in the control zone. The physical operator stamps unit-runtime and io-probe pods with a toleration for that taint (internal/controller/physical/device_node_toleration.go). The toleration carries an empty effect, so a deployment that hardens the taint to NoExecute does not evict running control pods. Nothing else in the product tolerates it. Exclusion is the default, and no values knob exists that would let a platform service opt itself onto controller compute.
  3. The chart makes every platform pod steerable. tolerations values (alongside the existing nodeSelector) cover the four operators, the gateway, MQTT (Deployment and HA StatefulSet), the docs site, the audit-flux-bridge, and the historian family, where historian.nodeSelector/historian.tolerations govern the collector Deployment, the prune CronJob, and the audit-archival CronJob together. The CNPG Cluster gains historian.database.cnpg.affinity, a passthrough to CNPG's AffinityConfiguration. The database is the stickiest pod in the chart (node-local storage pins it to whichever node it first lands on), which makes it the component that most needs deliberate placement.

The taint key deliberately reuses the dcs.io/role vocabulary while remaining a distinct mechanism from the dcs.io/role node label the Controller reconciler converges from Controller.spec.nodeLabels. The label describes what a node is. The taint excludes what a node may run.

Alternatives Considered

  • Documentation plus nodeSelectors, no taint. Pin every platform service to platform nodes via values and call it done. Rejected: a selector is an instruction to the pods that carry it, and the failure mode is silent. Any service added later, or any consumer omitting one value, lands on controller compute again. Exclusion has to live on the node so that forgetting is safe.
  • Product-applied taint. Have the physical operator taint the node when it binds a Controller. Rejected on the ADR 0004 boundary: node lifecycle belongs to the deployment layer, and an operator that taints nodes on bind would fight deployments that stage workloads before enrollment, surprise single-node dev clusters (kind, sales-demo, and docs-shots would all need the toleration everywhere), and turn a compliance posture into an unconditional product behavior. The product's half is making the posture possible and safe: the toleration ships hard-coded while the taint stays deployment policy.
  • Scheduling gates via admission (a ValidatingAdmissionPolicy denying platform pods on device nodes). Heavier machinery for the same outcome. It would also have to enumerate "platform pods" by label, inverting the default the taint gives for free (new services excluded unless opted in).
  • Reusing node-role.kubernetes.io/* taints. Device nodes are workers, and overloading the upstream control-plane vocabulary would collide with what cluster tooling infers from it. A dcs.io-scoped key states whose policy it is.

Consequences

  • The historian-on-controller-compute class of placement disappears on any deployment that applies the taint. Platform pods without the toleration are unschedulable on device nodes by default, which is the founder's "never possible" rather than "discouraged".
  • Deploy repos adopt the taint at node join and pin platform services with the new values. cndcs-deploy-bench retires its HelmRelease postRenderers patches (the interim carrying this posture on chart 0.2.3) once a chart release with these values ships.
  • Single-node and untainted clusters see no behavior change. The toleration is inert without the taint, every new value defaults empty, and rendered manifests are unchanged unless a value is set.
  • A unit-runtime or io-probe pod always carries the toleration, even on clusters that never taint anything. That is visible pod-spec noise, accepted in exchange for the guarantee that enabling the posture never requires a product change or restart choreography.
  • DaemonSet-style infrastructure is outside this decision, but it is not automatically safe. A DaemonSet has no exemption from taints: the scheduler places its pods normally, so an untolerated taint keeps them off a device node exactly as it would a Deployment. CNI and kube-proxy keep working because they carry a blanket operator: Exists, which is a property of those charts rather than of DaemonSets. Node-scoped observability agents generally do not. kube-prometheus-stack ships node-exporter with a control-plane toleration only, so adopting this taint silently drops node metrics for every device node — the series describing controller compute, which is what a failover drill or a soak run is measuring. Deployments must give such agents an explicit toleration for this taint.
  • The layer that owns node join cannot always apply the taint declaratively, so decision 1 is a statement about ownership rather than about mechanism. A kubelet may set taints only at registration (the NodeRestriction admission plugin), so a distribution's node-taint field may be unable to taint a node that has already joined. Talos machine.nodeTaints is the worked example: it renders a NodeTaintSpec and its apply controller is refused with node "<n>" is not allowed to modify taints, while kubectl get node simply shows no taint. Deployments should apply the taint with an administrative credential and verify it on the Node object.
  • The taint is not self-healing. It lives on the Node object, so it survives reboots, upgrades, and cold boots, and is lost only when the Node object is deleted and recreated — a node rebuild or re-enrolment. Re-joining a device node therefore re-applies it, and an untainted device node presents as healthy.
  • CR 5.1's "per-zone service topology" claim now has a host-layer enforcement mechanism to cite, closing the gap the bench incident exposed.