Skip to content

ADR 0037: A server alarm is interpreted on the console it annunciates to, and the HMI gains no infrastructure surface

Status: Accepted Date: 2026-08-05 Issue: #1240

Context

#1168 built the Servers surface for the engineering UI and put the HMI out of scope. ADR 0032 then raised server degradation as an ISA-18.2 alarm in every site namespace, and the fan-out carried it onto the HMI for free: the sidebar cue, the topbar badge, the alarm list, the alarm detail panel and the retained MQTT event all took server alarms with no new API surface and no new poll.

The two decisions were each correct and together they left a gap. A plant-floor operator is now annunciated a condition their console cannot explain. What reaches them reads:

Server plant-w2 is not Ready (status False, reason KubeletNotReady)

That is a kubectl sentence on the surface built for the person #1168 described as not speaking kubectl. The engineering UI holds the rest of the story, and the operator does not have the engineering UI.

An operator looking at that alarm has three questions.

  1. Is the control system underneath me degraded?
  2. Are my units affected?
  3. What am I supposed to do about it?

Some of the answer is already on their console. The ADR 0023 health indicator sits in the HMI top bar on every screen, backed by the read-tier system:health action that every role carries. A unit whose runtime pod stops being ready raises its own System alarm at High through the equipment alarm generator, in plant language, naming the unit. Feed freshness (ADR 0028) marks the values on a frozen display as stale. So the operator is not blind to the effects of a dead node. What they cannot do is connect the effects to the cause, or tell a server alarm that threatens their batch from one that does not.

ISA-18.2 frames the defect precisely. An annunciated alarm must carry a defined operator response, and the response must be one that operator can carry out. Annunciating a condition whose response is unstated teaches the operator that the category is noise, which is the same failure mode ADR 0032 invoked when it refused to raise a second alarm for the shrinking etcd margin. The missing surface is not a node table. The missing thing is the response.

Decision

The HMI gains no infrastructure surface. Server degradation is answered by the health indication already on every HMI screen, and by the alarm itself, which states its consequence, its per-site impact and its operator response before any engineering detail.

The HMI gains no server inventory, no node detail and no infrastructure action

The asset tree is an engineering surface (ADR 0033), and every action on it is an engineering action (ADR 0034). An operator escalates a hardware fault. Kubelet versions, volume bindings, cordon state and quorum arithmetic are inputs to a decision the operator does not make, and putting them on the HMI would furnish the appearance of a decision without the authority to take it.

The health indication gains a servers dimension

GET /api/v1/system/health now grades the cluster's servers alongside its core services and its unit runtimes, using the one node-health derivation #1227 already moved server-side, so the dot on the operator's screen and the alarm on their console are graded by the same rules. ADR 0032 named this as a reasonable separate improvement, and it is the whole of the "minimal system health readout" the HMI needs: one chip, already present, already read-tier, on every screen.

The rollup grades a lost etcd quorum as critical and any server that is not healthy as degraded. A single server leaving Ready is a redundant element failing, and the workload it carried either reschedules or annunciates through its unit's own rule. Losing quorum is the cluster-level event that stops state changes taking effect, which is why it alone is graded critical.

The chip stays a state indication. It carries no acknowledgement, no history and no external annunciation, exactly as ADR 0023 requires, so nothing here competes with the alarm tier or annunciates a condition twice.

Every server alarm states its consequence, its impact and its response

A server alarm message is assembled in a fixed order:

  1. Consequence. What has happened, in terms of what the plant loses. This leads because the alarm list truncates a long message and the first sentence is what the operator reads at a glance.
  2. Impact on this site. Which unit runtimes in this site namespace were running on that server. ADR 0032's fan-out already writes one Alarm per site, so each site's copy names that site's own units, and a site with none on the server says so. That answer is worth as much as the positive one: it is how an operator learns the alarm does not threaten their batch.
  3. Response. Escalate to whoever maintains the control system hardware, stated on every alarm, because escalation is the defined operator response and an operator console cannot correct a server fault.
  4. Engineering detail. The readiness status, the reason string and the quorum derivation, kept last so the engineer looking at the same alarm loses nothing.

Where the impact cannot be determined, the alarm says so. A message that reported no affected units because the query failed would be read as reassurance, which is the rule #1230 established for the pre-flight panel and it applies with more force to an alarm.

Who to call is deployment data, and the product does not invent it

The alarm names the class of responder. A site's escalation contact belongs to that site's alarm response procedure, which is customer data the product has no business guessing. A configured per-site contact remains available as a later addition if a deployment asks for one.

Alternatives Considered

Give the HMI a minimal system health page: the node list trimmed to health and nothing else. Rejected on three counts. It is the Servers view with columns removed, so it re-poses every scoping question #1168 answered and then diverges from it. It needs either a new route or the servers:read action, which a locked-down operator role may deliberately not carry. And it answers "which server" for a reader who has no use for the answer, while the question they can act on is whether their units are affected, which a node list does not address at all.

Write down that the operator escalates, and change nothing else. This is the option the issue offers as defensible, and it is half right. Escalation genuinely is the correct response, and the ruling above adopts it. It is wrong about where the writing goes. A response recorded only in a manual leaves the operator inferring it from a kubelet reason string at the moment they are holding an unacknowledged alarm, and ISA-18.2 asks for the response to travel with the alarm.

Suppress server alarms from the HMI and keep them engineering-only. Rejected. It hides a plant-relevant condition from the person watching the plant, which is the outcome ADR 0032 exists to prevent. A lost quorum also changes what the operator can do, because state changes stop taking effect, so it must reach them.

Put the server cause on the unit's own runtime alarm instead, and leave the server alarm as it is. Attractive, because it lands the infrastructure fact on the alarm the operator already acts on. Rejected as the primary mechanism because it covers only the case where a unit runtime is lost. A control-plane failure, a resource pressure warning and a lost quorum all reach the operator with no unit alarm beside them, and those are the cases where the operator most needs telling that the fault is not theirs to fix. It remains a reasonable addition on top of this decision.

Fold the affected-unit list into a structured field on the Alarm CRD rather than into the message. Rejected for now. It would need API surface, a DTO field, HMI rendering and a policy for every other alarm source that has no such list. The message reaches every consumer that already carries alarms, including the retained MQTT event an external annunciator reads, at no schema cost.

Consequences

  • The health chip on every HMI screen now moves for a hardware condition. A plant running a chronically pressured node will see a persistently degraded chip. That reading is true, and the remedy is to fix the node.
  • Server alarm messages get materially longer. The alarm list truncates the row and carries the full text in the tooltip and the detail panel, so the leading consequence sentence is load-bearing and the rules table in ADR 0032 now governs severity alone.
  • ServerAlarmReconciler reads Unit objects in each site namespace and watches for runtime-binding changes, so a failover updates the impact sentence on a standing alarm rather than leaving a stale claim on it.
  • A cluster where the reconciler cannot list units keeps annunciating, with the impact stated as undetermined.
  • The system health summary names a degraded server to every read-tier role. Nothing is newly exposed: ADR 0032 already puts the server's name on an alarm in front of every operator on every site.
  • Server health now has one derivation with three readers, which are the Servers view, the health rollup and the pre-flight panel. The alarm reconciler keeps its own evaluation because it grades ISA-18.2 severities rather than a health label, and both are stated against the same rules table.