Skip to content

ADR 0079: The retained alarm event carries the alarm as it stands

Status: Accepted Date: 2026-08-22 Issue: #1764 Related: ADR 0014 (the retained event as the suppression transport), ADR 0078 (the partial-loss alarms this was found under), ADR 0077 (a reader whose writer went away)

Context

dcs/{ns}/equipment/{kind}/{name}/alarm is retained. Its last message is therefore not a record of something that happened. It is the current truth about that equipment for every consumer that connects afterwards. The historian stores one row per message and reads them back as Alarm History. pkg/omfegress forwards them. The gateway's alarm cache feeds the HMI's live annunciation set.

AlarmReconciler published that event on four edges. The status.state == "" initialisation, a shelve expiry, a republish while shelved, and an acknowledge. Nothing else in the controller published at all.

Six sites outside internal/controller/alarm raise alarms. Three sit in unit_pod.go, and one each in unit_failover.go, batch_controller.go and phase_controller.go. Each of them Creates an Alarm carrying a spec and an empty status. The initialisation edge stamps ActiveUnacknowledged on it and publishes it exactly once. Every later change to those alarms is written straight onto the object by the raiser:

  • clearUnitAlarmsByPrefix ends one by writing Cleared* and ClearedAt onto its status.
  • annunciateDriverLoss rewrites a standing alarm's spec.Message when the set of dead buses grows underneath it (#1760). It does so deliberately without re-annunciating.

Both wake the AlarmReconciler. Both fell past all four branches and returned having published nothing. A probe swapping the publish seam and driving Reconcile over an io-loss- alarm recorded it:

after message drift, publishes = []
after clear,         publishes = []

The HMI self-heals, which is why nobody saw it. HMIAlarms.reconcileFromList replaces its whole working set from the 15-second REST poll (ADR 0029). That list reads the API. The historian self-heals from nothing. It recorded the raise and never the return to normal, so Alarm History showed an alarm that started and never ended. An external subscriber connecting after the outage replayed an ActiveUnacknowledged alarm that had cleared.

The alarms concerned are the loudest ones the product raises. Watchdog Holds, partial I/O loss, lease expiry, runtime crashes and failover.

Decision

The retained event carries the alarm as it currently stands, and the AlarmReconciler is what makes that true. It is the only controller watching Alarm. Every change to one reaches it, whichever package wrote it. What it lacked was a way to tell an alarm the broker has been told about from one it has not.

status.publishedRevision is that marker. It is a SHA-256 over the alarm event payload with two fields removed. The fingerprint is computed from buildAlarmPayload itself. A field added to the payload is therefore fingerprinted the moment it is added. AlarmReconciler.Reconcile ends by publishing when, and only when, the marker disagrees with the alarm in front of it.

One marker covers spec and status together. metadata.generation would have answered the message drift and nothing else. A clear is written to the status subresource and bumps no generation at all. Two markers would have meant two rules for one question.

The marker is written after the broker accepts the publish. A marker set in front of a publish that then fails is a claim that fails towards all-clear. The retained event stays stale, and the one thing that could notice believes it is current. Written afterwards, a marker that is missing or behind always means the event is owed. The reconcile returns the publish error, so controller-runtime's backoff retries it. Nothing else would wake that alarm. The object is already at rest, and the revision it is stale against is its own.

The three generators inside the package record what they publish. The recording lives inside publishAlarmEvent and publishAnnunciation, the two helpers they already call. That is where the unit resolution lives too (#1549), for the same reason: one place decides, and a new publish site cannot forget. Without it, every transition they publish would be published a second time by the catch-all, and the historian would store the duplicate row.

Two payload fields are excluded, and both are declared. timestamp moves on every publish. A fingerprint carrying it would never repeat and nothing would converge. unit is resolved out of the plant model, and it is not a field of the alarm. No controller watching Alarm wakes when a source's owning unit changes. Carrying it would put a plant-model Get on the reconcile of every alarm, to detect something nothing can notice anyway.

Alternatives Considered

Export the publish and call it from each raiser. Smaller: four functions become exported and six sites gain a call. Rejected because it makes "who publishes an alarm event" a question with four answers across four packages. It is also the shape that fails one site at a time. A seventh raise site lands green with no publish, exactly as the six did. The alarm controller owns alarm publishing by design. The fix that keeps that ownership is the one that cannot be forgotten.

status.observedGeneration alone. It covers the message drift and misses the clear, which is the half that matters most. An alarm that never ends is worse than an alarm whose text is stale. A partial answer is worse than none here, because publishing a drift while the clear still publishes nothing refreshes an Active event that is never retracted.

Publish unconditionally on every reconcile. Idempotent for a consumer holding the last retained message, and one stored row per reconcile for the historian. The shelved branch was already doing this. It is now gated too.

Let the catch-all republish the generators' transitions. Correct on the wire and wrong in the store. MQTT retention makes the second publish invisible to a cache, and the historian keeps both rows.

Consequences

An alarm ended by any of the six external sites now publishes its clear. Alarm History records the return to normal and pkg/omfegress forwards it. A subscriber connecting afterwards is told the alarm ended, in place of replaying it as active. A standing alarm whose message moves publishes the current message without re-annunciating, which is what #1760 decided and could not deliver past the API.

A failed publish is now retried. It used to be swallowed at every site, so a broker blip during a clear lost that clear permanently. The reconcile returns the error and the alarm is still marked unpublished. The next attempt sends it.

A shelved alarm no longer publishes once per reconcile. The shelve announcement still goes out. The repeats behind it were historian rows for an event that had not happened.

Every alarm standing when the operator is upgraded publishes once more, because its marker is empty and its revision is not. That is one row per standing alarm, once. It leaves the retained topic in agreement with the cluster, which is the state the fix is for.

An alarm transition costs one more status write. The publish and the record of it cannot be one write without putting the claim in front of the act.

make test holds five properties, each proven by breaking what it guards. The clear written from outside the package publishes, the message drift publishes, an alarm at rest does not, a failed publish is owed and retried, and a generator's own transition publishes exactly once. TestRevisionMovesWithEveryFieldThePayloadCarries walks the Alarm's spec and status fields from the type itself. A field it has no mutation strategy for fails the test, and is never skipped. The fingerprint is derived from the payload today, and that test is there for the hand-written field list that would replace it.