Skip to content

ADR 0032: Server degradation annunciates as an ISA-18.2 alarm raised by an operator in every site namespace

Status: Accepted Date: 2026-08-03 Issue: #1170

Context

#1168 gave the Kubernetes nodes running the control software a surface in the product. GET /api/v1/servers reads the Node objects, classifies them at read time (control-plane, controller-bound, unclassified worker), and the System app renders an inventory with readiness, abnormal-condition chips, node-bound PVs, and a cluster summary carrying a derived etcd quorum. There is no parallel CRD, which is what keeps the surface from drifting away from cluster truth.

The surface is read-only, and that is the gap it left. A control-plane node going NotReady is visible to whoever is already looking at the Servers view. Nobody watches an inventory page. On the bench (2026-08-03) a control-plane reboot held quorum the whole way through, and the fact that the cluster spent that window one failure away from a read-only API was recoverable only by going and looking. Dead RTC cells and node-pinned local-path PVs had the same property. For a customer this UI is the only management surface the hardware hardware has, so an infrastructure fault that never reaches the operator is a fault the product has chosen not to report.

Every other plant-relevant abnormal condition in this system annunciates. The alarm tier already carries an Alarm CR through the HMI sidebar cue and topbar badge (ADR 0029), the alarms list and history, and a retained per-source MQTT event that external annunciators subscribe to (ADR 0014). Server degradation is the one category of abnormal condition sitting outside it.

Three constraints shape where the annunciation can come from.

The ISA-88 modeling boundary. ADR 0004 makes a Controller control-zone compute adopted by the deployment layer; a server is site infrastructure and is deliberately absent from the physical model. #1168 held that line and this must too, which rules out attributing a node fault to a Unit or a ControlModule so that it inherits an existing alarm path.

Cluster health already exists and is a different indication. ADR 0023 established the .health-indicator rollup over control-plane workloads and unit runtimes as the single status light in the top bar. It answers "is the plant's control software running". It is a state indicator that an operator reads, and it carries no acknowledgement, no history, no shelving, and no external annunciation.

Alarms are namespaced and served per site. Alarm is a namespaced CRD and the gateway serves it only as /api/v1/sites/{site}/alarms. A cluster-level condition has no site of its own, and the Kubernetes cluster hosts every site configured on it.

Decision

A watch-driven reconciler in the physical operator derives server conditions from Node state and raises them as ISA-18.2 System alarms in every site namespace.

ServerAlarmReconciler (internal/controller/alarm/server_alarm_controller.go) watches Node and Site, maps every event to one reconcile key, re-evaluates the whole cluster, and converges each site namespace against the derived condition set.

The rules.

Condition Severity Message carries
Node's Ready condition is not True (control-plane node) High The status, the reason, and the live etcd quorum derivation including the margin
Node's Ready condition is not True (any other node) Medium The status and the reason
MemoryPressure / DiskPressure / PIDPressure is True Low Which pressures are active
Ready control-plane nodes below quorum size Critical Ready members, total members, quorum size

A node with no Ready condition at all has never posted status, which is the same operational fact as an explicit Unknown, and is treated as not Ready. Cordoning is an operator's deliberate act and raises nothing; #1168 already flags it in the inventory.

Quorum annunciates once, when it is lost. The margin rides on the control-plane node's own alarm. The bench case asks to see the margin shrink, and a High alarm reading 2 of 3 members Ready, quorum size 2, margin 0 says that on the alarm the operator is already looking at. A separate margin alarm would double-annunciate one root event, which is the ISA-18.2 failure mode alarm rationalization exists to prevent. Losing quorum outright is a different event with a different consequence, so it gets its own Critical alarm.

Severity is graded by consequence. A control-plane node carries API availability and the etcd margin. Another node's workloads reschedule, and where they cannot (a device node pinned by ADR 0031's taint), the units on it already annunciate at High through the Unit runtime rule. The pressures are early warnings whose consequence, if it persists, arrives as an eviction or as the node itself leaving Ready, so they sit below both.

One condition produces one Alarm CR per site namespace. Every site configured on the cluster runs on the degraded hardware, so the fan-out annunciates the condition on each site's console, and each console acknowledges what it saw. It also means the HMI, the topbar badge, the alarm list, the history and the MQTT event all carry server alarms with no new API surface, no new poll, and no new UI, which is what keeps the HMI out of scope here as #1170 asked.

The source stays outside the physical model. Node alarms carry source: Server/<node>, the quorum alarm carries source: Cluster/etcd-quorum. Neither names a Unit or a ControlModule. Server is the same identity #1168 made canonically addressable as #/r/Server/{name}.

Alarms this reconciler owns are labelled and converged as a set. alarm.dcs.io/server-managed=true marks them, so a condition that vanishes with the object that produced it (a node removed from the cluster, a control-plane resized) is still found and cleared. A standing alarm's message and severity are refreshed while it stands, because both carry live derivations.

A returning condition re-annunciates from either cleared state. An alarm sitting in ClearedUnacknowledged when its condition returns goes back to ActiveUnacknowledged. A node that flaps must annunciate every time it fails, and the alternative silently drops the second failure.

Virtual nodes are excluded. dcs.io/virtual-node marks the simulation fixtures the Controller reconciler creates and keeps Ready. Their readiness is an artifact of this project's own code, so annunciating it would report a simulation detail as a plant condition.

Alternatives Considered

Serve server alarms from the gateway, synthesized at read time from the same Node list #1168 already reads. No controller, no CRs, and consistent with the surface it extends. Rejected because a synthesized alarm has no identity to acknowledge, no state to shelve, no history to query and nothing to publish to MQTT. It would be a second alarm-shaped thing that does not behave like an alarm, and every consumer would need to learn about it.

Raise the alarms in one designated namespace and give the gateway a /api/v1/servers/alarms route. One CR per condition, no fan-out, and a home that matches the cluster-level place the Servers section occupies in the tree. Rejected on reach. It needs a new route, a new poll in the badge tick, HMI work to merge two alarm sources into one working set, and a decision about how site scoping applies to a cluster-level list. That is a larger change than the problem warrants, and it delivers annunciation to fewer surfaces than the fan-out does on day one. The route stays available if multi-site clusters ever make the duplication a real cost.

Fold node conditions into the ADR 0023 cluster-health rollup instead. Cheapest of all, and node readiness genuinely belongs in a health rollup. Rejected as the primary mechanism because it is deliberately not an alarm: ADR 0023 keeps the health dot as a state indication precisely so it does not compete with the alarm tier, and a Critical infrastructure event needs acknowledgement, history and external annunciation. Adding node conditions to the rollup remains a reasonable separate improvement.

Model servers as CRs so they can carry AlarmDefinition rules like any other source. Rejected twice over: it re-introduces the parallel CRD #1168 avoided, and it puts server infrastructure inside the ISA-88 physical model that ADR 0004 keeps it out of.

Consequences

  • A control-plane node leaving Ready now annunciates within one watch event on every site's HMI, alarm list, badge and MQTT topic, and the etcd margin is visible in the alarm text rather than only in the Servers view.
  • A cluster with N sites carries N Alarm CRs per server condition. Each is acknowledged and shelved independently, so acknowledging on one site does not silence another site's console.
  • Server alarms are shelvable like any other alarm, so a known-degraded node during planned maintenance can be suppressed through the ADR 0014 path without engineering involvement.
  • The etcd quorum derivation now exists in two places: the gateway's read-time surface and this reconciler. Both are the same stacked-etcd assumption stated in the same terms, and both state in the payload that the value is derived from control-plane node readiness. Neither speaks to etcd.
  • The physical operator's alarm surface now depends on cluster-scoped Node and Site reads. Both were already granted to it.
  • The equipment alarm generator still re-activates only from ClearedAcknowledged, so a flapping IOModule or Unit drops its second occurrence where a flapping node does not. That divergence is filed as #1181, and this ADR fixes the behaviour where it is being written. (Resolved 2026-08-05 by #1181: the equipment and AlarmDefinition generators now share this ADR's rule.)
  • A cluster whose control-plane nodes are not labelled (some managed distributions hide them) derives no quorum and raises no quorum alarm. Node readiness rules still apply to every node that is visible.