ADR 0036: Planned maintenance shelves the server alarms it is expected to cause, on a clock of its own¶

Status: Accepted Date: 2026-08-04 Issue: #1236

Context¶

ADR 0032 made server degradation annunciate: a node whose Ready condition leaves True raises an ISA-18.2 alarm in every site namespace, and it reaches the operator through the HMI cue, the topbar badge, the alarm list and the retained MQTT event. ADR 0034 then gave the product a way to take a node out of service on purpose, through a cluster-scoped NodeMaintenance object the physical operator executes.

Put together, the two produce a nuisance alarm by construction. A drain followed by a reboot takes the node out of Ready, every time, because that is what the operator asked for. ADR 0032 correctly rules that cordoning raises nothing, since a cordon changes no node condition, but a reboot changes several. Each planned maintenance therefore ends with a burst of alarms describing exactly the thing the operator just authorized.

ISA-18.2 is unambiguous about where that leads. An alarm that fires on every planned act teaches operators that the category is noise, and the next real server fault arrives in a channel nobody reads. That would quietly undo what #1170 delivered, which is the reason this is worth an ADR rather than a filter.

The suppression mechanism is not in question. ADR 0014 already settled it: shelving sets status.shelvedUntil, the retained alarm event carries shelved: true, and every annunciating consumer is contractually obliged to go quiet while an alarm-summary surface keeps showing it. Shelving is also the only one of the three suppression mechanisms that leaves an alarm active while suppressing annunciation, which is exactly the semantics a maintenance window wants: the node really is not Ready, and the operator really does not need to be told.

What has to be decided is the scope, the clock, and what happens when a maintenance does not end.

Decision¶

A maintenance window shelves the ADR 0032 server alarms of the node under maintenance, for a fixed period measured from status.startedAt, and an overrun annunciates in its own right.

Scope: the node under maintenance, and nothing else¶

Only conditions whose source is Server/<node under maintenance> are shelved. Three categories are deliberately left annunciating.

Another server failing during the window. A second node leaving Ready while the first is being serviced is not a planned event, and it is the moment redundancy is thinnest.

The Cluster/etcd-quorum alarm. Losing quorum while a node is deliberately out of service is the worst possible moment to be quiet about it. The maintenance guard refuses a request that would lose quorum, so reaching this state at all means something else went wrong too.

Every equipment alarm in the plant. A process fault that merely coincides with a maintenance window is still a process fault. This is narrower than #1236 first assumed, because #1235 refuses to drain a node hosting live unit runtimes at all: they have to be failed over through the ADR 0006 fencing path first. A drain therefore never evicts a running unit, and the eviction burst that would have justified shelving equipment alarms does not exist. Shelving them anyway would buy nothing and hide a real fault.

Clock: a fixed window from the moment the maintenance started¶

The shelve runs for MaintenanceAlarmShelveWindow (two hours) from status.startedAt, and expires on that clock whether or not the node comes back and whether or not anybody closes the object.

A shelve that lasts "until return to service" is the obvious design and it is wrong, because the return path is precisely what fails when a maintenance goes badly. A machine that never comes back, an operator called away mid-procedure, an object abandoned in Drained: each leaves a node permanently un-annunciated by a suppression nobody remembers authorizing. ISA-18.2 shelving is time-limited by definition for this reason.

Expiry needs no new machinery. The AlarmReconciler already requeues at shelvedUntil and clears it, and this is the same path an operator's own shelve takes. Two hours covers a reboot, a disk swap or a memory replacement with room to spare.

Return to service before the deadline releases the shelve early, so an alarm that is still standing when the maintenance closes goes back to annunciating immediately rather than staying quiet for the rest of the window.

Ownership: release only what the maintenance applied¶

The shelve carries alarm.dcs.io/maintenance-shelve: <maintenance name> on the alarm. It is the same ownership rule ADR 0034 uses for cordonedByUs: an operator's own hand shelve on a server alarm survives the maintenance untouched, and a maintenance never shortens one that reaches further than its window.

The marker is also what makes the shelve apply exactly once per alarm and maintenance. Without it, the converge pass would re-shelve at the next resync whatever the expiry had just released, and the fixed clock would be decorative. It equally means an operator who unshelves by hand inside the window is not fought on the next pass.

An overrun annunciates¶

A maintenance still open past its window raises server-<node>-maintenance-overrun, a Medium System alarm sourced Server/<node>, naming the intent, the phase, when it started and when the window closed.

Silence is the wrong answer to an overrun, and the expiry alone is not an answer. A cordon-only or drain-only maintenance leaves the node perfectly Ready, so no ADR 0032 rule fires and the lapsing shelve produces nothing at all. A chassis can sit cordoned and out of service for days with every indication green. The overrun is the annunciable fact, and on the bench it is the likelier failure than a node that never boots.

It is graded Medium on every node, control-plane or not. The overrun itself is a loss of attention rather than a loss of control; whatever the node's absence actually costs annunciates through its own rule the moment the window lapses, graded there by consequence. The overrun alarm is exempt from the very window it reports on, and it is acknowledged and shelved through the normal operator path like any other alarm, which is the right answer for a node legitimately awaiting a part for a week.

The maintenance object says what it is suppressing¶

NodeMaintenance publishes AlarmsShelved: True/WindowOpen while the window is open and carrying the expiry instant, False/WindowExpired once it closes on a maintenance that is still open, and False/Released on any terminal phase. The reconcile requeues at the deadline so the readout cannot sit at "shelved until 14:00" an hour after 14:00.

The window itself is derived from status.startedAt and the phase rather than stored, through helpers on the API type that both reconcilers call, so the tier applying the shelve and the object reporting it cannot drift. No CRD field is added.

Server alarm rules stay fixed¶

Shelving changes no rule and authors nothing. The rules table in docs/alarm-management.md remains code, unavailable as an AlarmDefinition, and a maintenance can only suppress annunciation of what those fixed rules already produce, for a bounded time, on one named node. The overrun alarm is a new fixed rule in the same table, added by this ADR in the same place every other server rule lives.

The #1181 divergence¶

ADR 0032 filed #1181: the equipment alarm generator re-activates a returning condition only from ClearedAcknowledged, while the server generator re-activates from either cleared state.

Shelving does not cross that ground, and the reason is worth writing down: an ADR 0014 shelve never changes an alarm's state. It suppresses annunciation and leaves the state model alone, so shelving and un-shelving add no new re-activation question and need no second answer.

The divergence does bear on whether shelving is correct, and it settles which side is right. Inside a maintenance window a node can recover and fail again; the alarm clears to ClearedUnacknowledged and its condition returns. Under the server generator's rule the second failure re-activates, stays suppressed while the window is open, and is audible the moment it closes. Under the equipment generator's rule the second failure would be dropped, and an operator reaching the end of a maintenance would find a cleared alarm and conclude nothing had happened. Re-activation from either cleared state is therefore the product behaviour, as ADR 0032 already ruled.

Making the equipment generator match stays #1181's work, because it lands in alarm_generator_controller.go, the AlarmDefinition reconciler and the state diagram in docs/alarm-management.md, and none of those is reachable from a shelve. #1181 landed on 2026-08-05, and all three generators now decide re-activation through one shared isActiveAlarmState predicate.

Alternatives Considered¶

Shelve every alarm in the affected site while a maintenance is open. Simple, and it certainly stops the burst. Rejected: it silences the plant during a window when the operator's attention is already on the infrastructure, and the first real process fault to coincide with a maintenance would be discovered afterwards. Scope is the whole safety argument here.

Suppress at evaluation instead: skip the ADR 0032 rules for a node under maintenance so no alarm is raised at all. Cheaper than shelving, with no CR churn. Rejected because the alarm's absence would be indistinguishable from health. Shelving keeps the alarm in the list, marked and counting down, which is what ADR 0014 requires and what an operator needs to know a node is down on purpose. It would also have been a second suppression mechanism sitting beside the one #675 built.

Hold the shelve until return to service. The intuitive scope, and it never needs an overrun rule. Rejected: it makes the suppression depend on the exact path that fails when a maintenance goes wrong, and an indefinite shelve is not shelving. The fixed clock costs an operator on a long job one acknowledgement.

A per-maintenance shelve duration on the spec. More flexible, and a plant with genuinely long procedures could set it. Rejected for now: it invites the number to be raised until the shelve is effectively indefinite, which is the failure this ADR is preventing. The overrun alarm is the pressure valve, and it is acknowledgeable. If real deployments show two hours is wrong, the constant moves, and a spec field remains available afterwards.

Store the window as a status field on NodeMaintenance. Then the gateway could read it directly. Rejected as a CRD change buying nothing the derivation does not already give: both reconcilers compute it from status.startedAt through one shared helper, and the condition message carries the instant for anything reading the object.

Let the maintenance reconciler write the shelve onto the alarms itself. It owns the window, so it looks like the natural writer. Rejected: the server alarm reconciler owns those CRs, already fans every condition out to every site namespace, and is the only component positioned to shelve an alarm at the moment it is raised. A shelve applied by a second writer would always arrive at least one reconcile after the alarm, which is one reconcile after the horn.

Consequences¶

A planned drain and reboot no longer annunciates on any console. The alarm is raised, marked shelved with its countdown, and visible in the alarm list and history where it belongs.
The retained MQTT event carries shelved: true from the moment the alarm is created, so external annunciators honouring the ADR 0014 contract are silent too, not just the HMI.
A maintenance that overruns produces one Medium alarm per site namespace and the node's own conditions come back audible. A long legitimate maintenance therefore costs the operator an acknowledgement, by design.
The ServerAlarmReconciler now reads NodeMaintenance and watches it. Both reconcilers already run in the physical operator, which already holds the read.
The reconciler's backstop resync moves forward to the earliest shelve deadline still ahead, so an overrun annunciates at the instant the window closes rather than up to five minutes later.
Alarm churn is bounded: the shelve is applied once per alarm and maintenance, and the overrun message carries fixed instants rather than an elapsed time, so a standing overrun does not rewrite and republish itself on every pass.
A maintenance parked in a waiting phase now commits its status on each reconcile where it previously returned without writing, which is what lets the AlarmsShelved readout age out of its window on time.
ISA-18.2 conformance improves in the direction #675 and ADR 0014 started: the product now shelves a predicted nuisance before it annunciates, with an audited authorization, a bounded duration and an automatic return.