Skip to content

ADR 0029: Alarm annunciation is poll-backed, with the WebSocket as an accelerator

Status: Accepted Date: 2026-07-24 Issue: #1072

Context

The HMI annunciates alarms through two cues driven off one working set, HMIState.activeAlarms:

  • HMIAlarm.updateNavIcon() — the sidebar alarm cue, lit while any alarm in the set is unacknowledged.
  • HMIDisplay.updateAlarmIndicators() — the unit-card borders, unit-dashboard outlines, faceplate borders and faceplate alarm strip.

That set was written by exactly one thing: HMIAlarm.onAlarmMessage, dispatched from the WebSocket alarm push. loadInitialAlarms seeded it once per site change and nothing refreshed it afterwards — not even a WebSocket reconnect, which re-subscribes but never re-reads.

So a dropped broker link froze annunciation at the instant the socket died. An alarm firing during the outage did not light the cue; an alarm clearing during the outage kept its cue lit; and a reconnect did not repair either, because the pushes that would have carried the news were the ones that were lost. Nothing recovered until the operator happened to change site.

ADR 0028 made this visible by removing the accident that had been half-covering it. The WebSocket fault chrome — the Feed lost readout and the full-width banner — is now suppressed on the batch and alarm views, correctly: those views paint from their own HTTP poll, so a banner asserting that on-screen values are last-known would be false where it appeared. But that banner was also, incidentally, the only local warning an operator got that annunciation had stopped. ADR 0028 recorded the gap in its own Consequences and deferred it.

The mitigations that existed were weaker than they looked. The alarms list view polls, so it stays current — but only while it is the mounted view. Cluster health reports the broker as Degraded — but that is a plant-software indication, not an alarm one, and ADR 0023 deliberately keeps the two apart. And the topbar alarm badge, named in #1072 as the surviving HTTP-backed count, was not counting anything at all: all three shells tested alarm.status.state === 'Active' against a DTO whose state is a flat top-level field carrying an ISA-18.2 name (AlarmDTO, types_alarm.go). Nothing ever matched. The badge had never appeared on any page, in any shell, for any alarm.

Decision

Alarm annunciation is backed by the periodic REST poll. The WebSocket push is an accelerator on top of it, not the source of truth.

Concretely:

The working set is reconciled from the alarm list. HMIAlarm.reconcileFromList(site, items) rebuilds HMIState.activeAlarms from the same REST DTO loadInitialAlarms already parsed, applying the same filters: active states only, shelved alarms excluded from annunciation per ADR 0014. It replaces rather than merges, so an alarm that cleared while the socket was down disappears — the failure mode a merge could never repair.

It rides the poll that is already on the wire. The cross-cutting 15s tick that refreshes the topbar alarm badge fetches each site's alarm list already; the HMI's current site hands its list to the reconcile via hmiNoteAlarmPoll. No new timer, no new request. The other two shells load the same badge code and never touch HMI state, because the call site is a typeof guard and the hook is defined in hmi-alarm.js.

Annunciation under broker loss is 15-second, and that is the designed behaviour. Not the push latency, and stated as such rather than left implicit. An alarm cue that updates on a bounded, known period is a system with a degradation mode; a cue that silently stops is not.

An ack the operator has just made outranks the poll for a bounded window. acknowledge() clears the cue optimistically the moment the POST succeeds, while the Alarm CR takes a reconcile to catch up. A poll tick landing in between reports the alarm as still unacknowledged and would relight the sidebar cue for one period. _localAcks holds the local answer until the server agrees or 60s elapse; after that, an alarm still reading unacknowledged is a genuine re-annunciation and must be shown.

A poll that cannot complete says so, on the badge. The badge held the last count and swallowed its error. It now holds the last count and marks itself stale — after two consecutive failures, so a single blip does not flash topbar chrome — carried as a dashed border rather than a colour, with the reason in the title. This is the one surface that needs to speak, because the poll failing is the one case where annunciation is frozen again.

The badge counts what is actually annunciating. isAnnunciatingAlarm() is one shared predicate over the real DTO: both active states count, an acknowledged-but-still-present alarm included; cleared and currently-shelved alarms do not. One copy, in utils.js, for all three shells.

Alternatives Considered

A scoped "annunciation degraded" indicator on poll-driven views (option 1 as filed). This keeps annunciation WebSocket-only and adds honest chrome saying so. Rejected as the primary fix: it makes the operator responsible for compensating for a cue that has stopped, which is the wrong division of labour for the alarm tier. It is also strictly worse than making the cue keep working — and once the cue keeps working, the indicator has nothing true left to say that cluster health does not already say. The narrow version of it survives as the badge's stale marker, which fires on the case that genuinely is degraded.

Move alarm annunciation onto a path that survives the split (option 3 as filed) — the same structural question ADR 0028 deferred for batch and execution updates. Rejected on scope for the same reason: it means new WebSocket channels and server-side fan-out to fix a client-side single-source-of-truth mistake. Nothing here forecloses it. If annunciation later moves to a socket that survives broker loss, this poll becomes redundant rather than wrong.

Seed the working set on WebSocket reconnect instead of polling. Much smaller change, and it repairs the post-outage state. Rejected because it does nothing during the outage, which is the window the operator is actually in, and because it makes correctness depend on a reconnect event firing — the reconnect path is exactly what is untrustworthy when the broker is down.

Give annunciation its own poll rather than riding the badge's. Simpler coupling and independent of state.sites being populated. Rejected for the duplicate request: the badge tick already fetches the current site's alarm list on the same cadence, and two requests for the same list is the kind of thing that gets noticed and "optimised" back into one later, by someone who has to rediscover why there were two.

Make the poll faster than 15s under broker loss. Rejected as premature: it trades a known, stated cadence for an adaptive one whose behaviour depends on correctly detecting the outage, and the detection is the part that was wrong in the first place. The cadence can be revisited with operational evidence.

Consequences

  • Annunciation degrades in cadence rather than stopping. During a broker outage the sidebar cue and faceplate indicators update on the 15s tick.
  • A missed or dropped push self-heals within one poll period, so annunciation no longer depends on the socket having been continuously healthy since the last site change.
  • The topbar alarm badge works. It has never displayed a count before this change, on any page — which also means every screenshot and clip in the docs and marketing corpus was captured without it. Placements that show an alarm condition may now show a badge they did not show when they were shot.
  • The badge can report that it does not know, instead of reporting zero.
  • The two paths can disagree briefly: a WS push lands immediately, and the next poll tick confirms it. Since the poll replaces the set wholesale, the poll wins any disagreement — except on a just-made ack, which the operator wins for 60 seconds.
  • updateAlarmBadge exists once instead of three times. The three shells keep their own pollAlarmBadge timers, which is where they legitimately differ.
  • Nothing about feed freshness changes. hmiNoteAlarmPoll deliberately does not stamp HMIState.lastPollAt: the badge tick runs on every view including the process display, and ADR 0028 turns on it never vouching for a screen it does not paint.