ADR 0014: Alarm shelving suppresses annunciation via the retained alarm event¶

Status: Accepted Date: 2026-06-27 Issue: #675

Context¶

ISA-18.2 shelving is a temporary, operator-authorized, time-limited, audited suppression of a currently-annunciating nuisance alarm (a chattering or standing alarm), so it stops cluttering the active alarm list and masking new alarms — without engineering involvement. Its defining effect is annunciation suppression: while shelved, the alarm should not raise the operator's attention anywhere, not just in one view.

As built (see #675), CNDCS shelving sets Status.ShelvedUntil on the Alarm CR, audits it, and auto-returns on expiry — but it publishes nothing about the shelf to MQTT. The shelve's only observable effect is that one HMI sub-tab (alarms.js "Active") hides the row. Every external consumer keeps annunciating, and the same alarm is still visible (dimmed) in the "All" tab. The suppression — the entire point — is missing.

The alarm event transport, today:

Topic: dcs/<namespace>/equipment/<kind>/<name>/alarm (pkg/mqtt/topics.go TopicEquipmentAlarm).
QoS 1, retained = true (internal/controller/alarm/mqtt.go). The retained flag means a late-joining subscriber immediately receives the last alarm event for that source — so whatever we put in the payload is the current truth a reconnecting annunciator sees.
Payload: name, source, type, severity, state, message, plus optional acknowledgedBy/At, clearedAt, timestamp. No shelve information.
Publishers: only the alarm controllers (alarm_controller.go, alarmdefinition_controller.go, alarm_generator_controller.go) call publishAlarmEvent. The gateway never publishes alarm events — it only mutates CR status.
Consumers: in-repo, the gateway subscribes (internal/gateway/tagbus.go handleAlarmMessage) and fans events out to the HMI over WS/SSE. Out of repo, any annunciator / SCADA bridge / notifier may subscribe to the alarm topic. We can enforce the contract on the gateway; for external consumers we can only publish the data and document the contract.

A relevant asymmetry among the three suppression mechanisms (#675): spec.enabled = false (out-of-service) and ArmingGate (suppressed-by-design) both suppress at evaluation time — they auto-clear any active alarm, so no "active-but-suppressed" alarm is ever emitted. Shelving is the only mechanism that leaves an alarm active while suppressing its annunciation. That is why this ADR is about shelving specifically, not a generic suppression flag.

The decision this ADR settles is item 2 of #675: how does a consumer learn an alarm is shelved, and what is it obligated to do?

Decision¶

Extend the existing retained alarm event with the shelve state, and publish it on the shelve transition over the same per-source topic. Suppression is a documented consumer contract carried by that payload — not a new topic.

Concretely:

Payload, additive: add shelved (bool) and shelvedUntil (RFC3339) to publishAlarmEvent. shelved is the field consumers key on; shelvedUntil is informational (countdown / display). Absent/false ⇒ not shelved. The change is backward-compatible — existing fields are untouched.
Publish on the shelve transition, from the controller. The gateway/CLI continue to only set Status.ShelvedUntil (no MQTT in the gateway — keep the single-publisher invariant). The AlarmReconciler, which today merely RequeueAfters while shelved, must publish once on entering the shelved state (newly-set ShelvedUntil), in addition to its existing publish on expiry/unshelve. Retained delivery then makes the shelf the current truth for any subscriber, including late joiners.
Consumer contract (documented, enforced in-repo):

A consumer that performs annunciation (horn, flashing, push notification, escalation, SCADA alarm raise) MUST treat shelved: true as "do not annunciate" for the duration. The alarm MUST still remain visible in any alarm-summary / list surface, marked as shelved with its shelvedUntil — shelving suppresses annunciation, not existence.

The gateway HMI is brought into conformance: shelved alarms are suppressed from active annunciation consistently (not merely hidden from one sub-tab) and shown as shelved in the summary with their remaining time.

Field name is shelved, not generic suppressed. The other two suppression tiers don't emit active-but-suppressed alarms (they auto-clear at evaluation), so a generic flag would have only one producer today and would over-promise. If a future ADR makes out-of-service keep the alarm active-but-suppressed, that is the moment to generalize to suppressed + suppressedReason — called out here as the extension path, not built now.

This ADR covers transport and the consumer contract only. The shelve state gate (State == ActiveAcknowledged) and the auto-return behavior are decided in #675 and are not re-litigated here.

Alternatives Considered¶

Dedicated suppression topic (e.g. .../alarm/suppression) carrying shelve state separately. Rejected: doubles the topic surface and forces every consumer to subscribe to and correlate two retained streams per source to know whether to annunciate one alarm. The shelve fact belongs with the alarm it modifies; one retained message per source is simpler and race-free.
Generic suppressed + suppressedReason (shelved | out-of-service | by-design) now. Rejected for now: the other two reasons never produce an active alarm to suppress (they auto-clear), so the generalization has a single real producer today and invites consumers to code against states we don't emit. Kept as the documented extension path.
Publish the shelve from the gateway handler instead of the controller. Rejected: it breaks the single-publisher invariant (all alarm MQTT comes from the controllers), splitting alarm-event ownership across two processes and risking divergent payloads. The gateway sets status; the reconciler reacts.
Leave transport as-is; do suppression purely in the HMI (status quo, just more views honoring it). Rejected: it cannot silence external annunciators, so it never delivers ISA-18.2 shelving — exactly the gap #675 raises.
Remove shelving entirely. A live option in #675, but out of scope for this ADR, which assumes the decision to keep and implement shelving. If shelving is removed, this ADR is withdrawn.

Consequences¶

Code that moves:
internal/controller/alarm/mqtt.go — add shelved / shelvedUntil to the payload.
internal/controller/alarm/alarm_controller.go — publish once on entering the shelved state, not just on expiry.
internal/gateway/tagbus.go handleAlarmMessage + the WS/SSE message and HMI — carry and honor shelved so suppression is consistent across the UI, not a single-tab filter.
docs/api-reference.md if the alarm event schema is documented there (ADR 0003 contract); docs/monitoring-metrics.md is unaffected.
Compliance: docs/compliance/isa88.md (and any ISA-18.2 mapping) updates from "shelving = single-view hide" to "shelving = annunciation suppression via retained alarm event," which is the first genuinely ISA-18.2-conformant shelve behavior in the product.
Operational: external annunciators must honor shelved to benefit. We publish the data and document the contract; we cannot enforce third-party behavior. The retained flag means reconnecting consumers get correct shelve state without a replay mechanism.
Followups unlocked: a later ADR can generalize to a suppressed reason enum if out-of-service/by-design ever emit active-but-suppressed alarms; shelve escalation/notifier integration can now key off a real signal.
Reversibility: high. The payload additions are additive and ignorable; reverting means consumers fall back to today's behavior (treat as not-shelved). No schema migration, no CRD change (the ShelvedUntil field already exists).

Amendment (2026-08-11, #1442): the retained flag speaks only for MQTT subscribers¶

The Consequences above end with "the retained flag means reconnecting consumers get correct shelve state without a replay mechanism." That is true of a consumer subscribed to the alarm topic, and it was false of a consumer on the WS/SSE surface this same ADR names as the fan-out path. The gateway receives the retained event once, at its own broker connect, and broadcasts it to whoever is connected at that instant. A SCADA that connected an hour later got nothing.

The shelved alarm is the case the gap costs most. A shelf produces exactly one transition and then silence, so a consumer that missed the frame has no second chance to learn the alarm is suppressed, and the ADR exists precisely so that it does not annunciate one.

The decision is unchanged. What changes is the claim about recovery: the hub now caches alarm and state frames the way it already cached tag values and control module health, and replays the matching ones when a subscription is registered, marked replay: true. A resolved alarm is replayed for a bounded window and then forgotten, because a return-to-normal that was also acknowledged ends the alarm and a consumer that missed the end still has to be told about it.

The REST snapshot is brought into the same shape. AlarmDTO gained a shelved boolean computed against the current time, and the WS/SSE alarm frame gained shelvedUntil, so a consumer recovering over either transport reads the same two fields. Reading the deadline alone was the trap: a lapsed shelf leaves shelvedUntil populated until the reconciler clears it, and a consumer treating that as suppression silences an alarm that is annunciating.

Amendment (2026-08-11, #1470): a deleted alarm publishes an empty retained payload¶

The Decision above makes the retained event the current truth for a source, and the single-publisher invariant makes the alarm controllers the only thing that can write it. Neither of them covered an alarm ceasing to exist. publishAlarmEvent runs on transitions, deletion is not a transition, and the retained flag means publishing nothing is not silence — the deleted alarm's last event went on standing on the topic as the current truth for every consumer that connected afterwards, forever. An Alarm is deleted when its AlarmDefinition is removed and garbage collection takes the alarms with it, or when an engineer deletes an orphan through the gateway. #1442 closed the half the gateway performs itself by evicting the alarm from the hub's replay cache. It never saw the other half, and it could not have cleared the broker's retained payload if it had.

The alarm gains a finalizer, and the deletion path publishes an empty payload to the source's topic. That is the MQTT retained-clear, and it is the tombstone convention the CM-health topic's consumers already recognise: the gateway and the historian both ignore a zero-length payload rather than reading it as a malformed event.

The design question this amendment answers is the one the topic shape forces. Retention is per source, not per alarm. A unit with two alarms shares one retained topic, so a tombstone published for one of them erases the other's recoverable state along with it — and publishing nothing when other alarms remain is no better, because the standing retained event may well be the deleted alarm's and the controller cannot read the broker to find out. Three options were on the table.

Wait for the source's last alarm. Rejected: it leaves the common case wrong. Deleting one alarm of several leaves whatever the deleted alarm last published standing, which is exactly the bug, narrowed rather than fixed.
Carry the source's whole alarm set in the retained payload. Rejected: it changes the shape of every alarm event, for every consumer, to fix an event that is published on deletion. The payload is a documented integrator contract (docs/mqtt-telemetry.md), and it would go from one alarm to a set on every transition.
Add an alarm-name segment to the topic. Rejected outright: it breaks every existing subscriber, in-repo and out.

Clear the source, then republish the alarms that survive it. The tombstone is per source because that is what the topic can address, and the republish is what makes the clear safe: the survivors go back on the topic immediately, walked least-consequential first so that the alarm a late subscriber most needs — active over cleared, unacknowledged over acknowledged, then severity — is published last and wins retention. A source whose last alarm was deleted keeps no retained message at all.

The empty message in front of the republishes is not redundant. The gateway hub caches alarm frames per alarm (the #1442 amendment above), so it holds alarms this topic can no longer name; the clear tells it to forget the source's alarms and the republishes tell it which ones are actually left. A consumer that holds only what the topic retains loses nothing by the same sequence.

Two consequences worth stating. A deletion now re-publishes surviving alarms, so a consumer may see an alarm it already knows about arrive again — the same shape as the idempotent re-publish a shelved alarm's reconcile already performs, and deletions are administrative rather than routine. And the finalizer means a broker that will not accept the tombstone holds the deletion: for two minutes, after which the finalizer is released and the stale retained event is logged. A finalizer that never releases does not block one alarm, it blocks the namespace, and the operator who deleted an AlarmDefinition has no way to see that a broker outage is what is holding it.