Phase-Scoped Alarm Arming

Status: Implemented (#232) Issue: #231 Scope: alarm.dcs.io/v1alpha1 AlarmDefinition and the alarm-operator

Problem

Some equipment alarms are only meaningful while a specific phase is running. Today, every AlarmDefinition evaluates its condition continuously against a single source (an IOModule, Unit, or ControlModule). There is no way to say "this alarm only applies while equipment is being driven into a non-idle state by a known phase."

The motivating example is fd1-vacuum-loss (examples/riverbend/25-alarms.yaml). It fires when the filter-dryer pressure sensor reads near atmospheric (>= 913 mbar). That is the correct condition during a vacuum-drying phase. It is also the normal idle state, since an unused vessel relaxes to ambient by design. The alarm therefore fires continuously whenever the vessel is not actively under vacuum, which is exactly the kind of nuisance alarm ISA-18.2 considers a top-tier defect.

Operator-toggled disable (spec.enabled) is the wrong tool: an operator would have to remember to enable it before every drying phase and disable it after, defeating both the purpose of automation and the intent of the out-of-service flag.

Background

DeltaV phase alarms

In Emerson DeltaV, alarms can be declared in a Phase Class. The phase logic runtime activates them when the phase instance enters its execution state and removes them when the phase ends, so the alarm only exists in the operator alarm summary while it is conceptually relevant. The list of alarms is part of the phase class definition, and instances are created and removed in lock-step with the phase lifecycle. From the operator's point of view this looks like "the alarm is part of the phase," not "the alarm is part of the equipment."

ISA-18.2 / IEC 62682 — three orthogonal axes of "not firing"

ISA-18.2 distinguishes three reasons an alarm may not fire when its underlying condition is met:

ISA-18.2 concept	Today	Driver	Persistence
Out-of-service	AlarmDefinition.spec.enabled=false	Engineering authority	Persistent until re-enabled
Shelved	Alarm.status.shelvedUntil	Operator request	Time-bounded
Suppressed-by-design / state-based	(absent)	Process or equipment state	Automatic

The third category ("suppressed by design, state-based") is what issue #231 actually needs. ISA-18.2 (clause 12.2 in the 2016 edition) treats it as a distinct class precisely because conflating it with the other two breaks operator and engineering intent.

ISA-88 phase classes vs. phase instances

ISA-88 (Clause 5.3 of Part 1) separates procedural elements (the class) from their runtime activations (the instance). In this codebase that is PhaseTemplate (class, persistent) versus Phase (instance, transient, created per batch run).

Any phase-scoped alarm design must respect that split:

• The alarm declaration must live with the class, because that is what is reusable and authored once.
• The arming decision must be evaluated against the instance state, because that is what is transient and process-meaningful.

Candidate implementations

A. armingGate field on AlarmDefinition (recommended)

Add an optional spec.armingGate to AlarmDefinition. The alarm-operator watches Phase resources and treats the underlying condition as suppressed whenever the gate is non-empty and no matching phase is in one of the allowed states.

apiVersion: alarm.dcs.io/v1alpha1
kind: AlarmDefinition
metadata:
  name: fd1-vacuum-loss
  namespace: site-riverbend
spec:
  source:
    kind: ControlModule
    name: fd1-pressure-sensor
  condition:
    type: TagHigh
    tagName: PV
    threshold: "913"
    deadband: "5"
    debounceSeconds: 10
  armingGate:
    phaseTemplateRef: vacuum-dry
    unitRef: FD1
    states: [Running, Holding, Restarting]
  type: Equipment
  severity: High
  message: "Filter-dryer vacuum loss — pump fault or seal leak"
  priority: 2
  exceptionAction: Hold

Semantics: an Alarm CR is created (or kept active) iff all of the following hold:

1. spec.enabled != false (engineering hasn't taken it out of service).
2. Alarm.status.shelvedUntil is unset or expired (operator hasn't shelved it).
3. armingGate is empty, or at least one Phase in the same namespace has spec.unitRef == armingGate.unitRef (when set), references the named phaseTemplateRef, and is currently in one of armingGate.states.
4. The condition itself evaluates true.

A new status.armed boolean and a Suppressed status condition record why the alarm is not currently armed, so operators can distinguish out-of-service from "waiting for a vacuum-dry phase."

Pros - One CRD, one controller, one writer for Alarm lifecycle. - Orthogonal to enabled and shelvedUntil — preserves the three distinct ISA-18.2 axes. - Backward compatible: empty armingGate is the current behavior. - No CRD churn — AlarmDefinition is created once at engineering time and survives across many phase executions. - Operator workbench (alarm list, tuning UI) keeps a single source of truth per alarm.

Cons - The alarm-operator gains a watch on Phase, which adds reconcile fan-out on phase state changes. Bounded and acceptable. - The alarm declaration sits in the alarm namespace, away from the phase template. Mitigated by adding a back-reference field on PhaseTemplate for documentation and validation (see Future work).

B. PhaseTemplate-owned PhaseAlarmDefinition CRD

Introduce a separate CRD declared inline on PhaseTemplate.spec.alarms. The procedural-operator instantiates an alarm per phase activation, owned by the Phase so it is garbage-collected when the phase ends.

Pros - Closest to DeltaV's surface syntax, and alarms travel with the phase class. - Lifecycle is naturally bounded by the phase instance.

Cons - CRD churn: one alarm CR created and deleted per phase execution. Hundreds of batches per day at tens of phases each means thousands of CR writes/day for a feature that is mostly idle. Pressure on etcd, audit log noise. - Alarm tuning becomes hard. Threshold or severity changes must be made on the template, propagate at next instantiation only, and never affect a currently running phase. That is the wrong default for safety-relevant parameters. - Operator workbench fragments — the live alarm list mixes definitions from arbitrary templates that may exist for only seconds. - Ownership chain becomes deep: Batch → Procedure → UnitProcedure → Operation → Phase → Alarm. Cascade-delete works but is fragile.

C. Procedural-operator patches spec.enabled

PhaseTemplate references existing AlarmDefinition names, and the procedural-operator patches spec.enabled = true on phase activation and = false on completion.

Pros - Reuses the existing enabled field, with minimal CRD change.

Cons - Semantic conflation, fatal. Out-of-service is an engineering action that should require explicit re-enable. Auto-toggling it from a phase controller silently re-arms alarms that were intentionally taken out for maintenance. - Multi-writer hazard. Two phases referencing the same alarm fight over the field, and whoever finishes last wins. Reconciliation order is undefined. - Crash exposure. If procedural-operator dies between "Phase Running" and "Phase Complete," the alarm is left armed forever. - Write amplification on every phase state transition.

This is the option that maps most directly onto the DeltaV recollection, but the conflation of out-of-service with state-based suppression is the exact failure mode ISA-18.2 calls out as a top-tier alarm management defect. We should not adopt it.

Recommendation

Implement Option A. The armingGate field on AlarmDefinition, evaluated by the alarm-operator, gives us:

• ISA-18.2-correct separation of disabled / shelved / suppressed-by-design.
• ISA-88-correct mapping of class-level declaration to instance-level state.
• Kubernetes-idiomatic single-writer reconciliation with no CR churn.

CRD shape

// AlarmArmingGate suppresses alarm evaluation unless a matching phase
// instance is in one of the listed ISA-88 states. Empty gate means the
// alarm is always armed (current behavior).
type AlarmArmingGate struct {
    // PhaseTemplateRef matches Phase resources whose spec.phaseTemplateRef
    // (or, transitionally, spec.templateRef) equals this name.
    PhaseTemplateRef string `json:"phaseTemplateRef"`

    // UnitRef optionally restricts matches to phases running on a specific
    // Unit. Required when the alarm source could be driven by phases on
    // multiple units; recommended in all cases for clarity.
    // +optional
    UnitRef string `json:"unitRef,omitempty"`

    // States lists the ISA-88 procedural states in which the alarm is
    // armed. Defaults to ["Running"]. Other valid values: Holding, Held,
    // Restarting, Stopping, Aborting. Idle, Complete, Stopped, Aborted
    // are forbidden — by definition an "in idle" state is not actively
    // driving the equipment.
    // +optional
    States []string `json:"states,omitempty"`
}

type AlarmDefinitionSpec struct {
    // ... existing fields ...

    // ArmingGate optionally suppresses alarm evaluation unless a phase
    // matching the gate is currently in an active state. When unset, the
    // alarm is always armed (current behavior). Orthogonal to spec.enabled
    // and to operator shelving.
    // +optional
    ArmingGate *AlarmArmingGate `json:"armingGate,omitempty"`
}

type AlarmDefinitionStatus struct {
    // ... existing fields ...

    // Armed reflects whether the gate currently permits evaluation. True
    // when no gate is set; otherwise true iff at least one matching Phase
    // is in an allowed state. Updated on every reconcile.
    // +optional
    Armed *bool `json:"armed,omitempty"`

    // GatedBy names the Phase that currently arms the alarm, when armed
    // by gate. Empty when no gate is set or when not armed.
    // +optional
    GatedBy string `json:"gatedBy,omitempty"`
}

A new status condition Suppressed carries the reason (Disabled / Shelved / NotArmed) so the HMI and CLI can present a single, unambiguous explanation.

Owning operator

The alarm-operator owns the gate evaluation:

• Adds Watches(&proceduralv1alpha1.Phase{}, ...) to AlarmDefinitionReconciler.SetupWithManager. The handler enqueues every AlarmDefinition in the same namespace whose armingGate.unitRef matches the phase's spec.unitRef and whose armingGate.phaseTemplateRef matches the phase's template ref. A label selector on the Phase watch keeps fan-out bounded.
• On reconcile, after the existing gating checks (enabled, supported condition type), evaluates the gate by listing phases matching the unitRef + phaseTemplateRef selector and checking states. Skips the rest of the reconcile when not armed and either auto-clears any active alarm or leaves it untouched (see Open question 3 below).
• Surfaces status.armed, status.gatedBy, and the Suppressed condition.

The procedural-operator is unchanged. There is no cross-controller write path, so there is no multi-writer hazard.

Migration story

Existing alarm authors

The change is purely additive. Every existing AlarmDefinition continues to work unchanged because armingGate defaults to nil ("always armed"), which is the current behavior.

Existing example alarms

File	Alarms	Action
examples/riverbend/25-alarms.yaml	23	Audit each; add armingGate to those that only matter during a phase
examples/newark-plant/20-alarmdefinitions.yaml	18	Same audit
examples/riverbend/16-alarms.yaml, examples/newark-plant/16-alarms.yaml	(Alarm CRs only, no definitions)	No change

The audit identifies two classes:

1. Always-armed (no gate): equipment-health alarms — IOModule Offline/Fault, Unit Faulted, ControlModule offline, sensor out-of-range that should never legitimately occur. The majority of existing alarms fall here. Leave unchanged.
2. Phase-gated: process alarms that describe deviations from a setpoint the phase is currently commanding. Examples: vacuum loss during vacuum-dry, jacket over-temperature during heat-up, agitator stall during mix. Add armingGate referencing the relevant PhaseTemplate.

For the reference plant, expect roughly 5-8 alarms per site to be reclassified as phase-gated. The remaining ~30 are always-armed equipment-health alarms.

CRD schema rollout

Standard kubebuilder additive change. CRD regeneration is non-breaking (new optional field). Migration is a kubectl apply of the regenerated CRD followed by edits to individual AlarmDefinition CRs.

Worked example: fd1-vacuum-loss

Today (broken):

spec:
  source: { kind: ControlModule, name: fd1-pressure-sensor }
  condition: { type: TagHigh, tagName: PV, threshold: "913", deadband: "5", debounceSeconds: 10 }
  type: Equipment
  severity: High
  message: "Filter-dryer vacuum loss — pump fault or seal leak"

Behavior: condition is true whenever the vessel is at atmospheric, which is its normal idle resting state. Alarm fires continuously when no batch is running. Operator either learns to ignore it (defeating the entire alarm system) or has to remember to disable and re-enable it around batches.

After (fixed via Option A):

spec:
  source: { kind: ControlModule, name: fd1-pressure-sensor }
  condition: { type: TagHigh, tagName: PV, threshold: "913", deadband: "5", debounceSeconds: 10 }
  armingGate:
    phaseTemplateRef: vacuum-dry
    unitRef: FD1
    states: [Running, Holding, Restarting]
  type: Equipment
  severity: High
  message: "Filter-dryer vacuum loss — pump fault or seal leak"

Behavior:

• Vessel idle, no phase: gate evaluates to "no matching Phase in allowed state" → status.armed=false, Suppressed=NotArmed. No alarm.
• A Phase with phaseTemplateRef: vacuum-dry and unitRef: FD1 enters Running. The Phase watch enqueues the alarm. Reconciler observes the phase, sets status.armed=true, status.gatedBy=<phase-name>. From this moment the existing TagHigh + debounce + deadband logic governs whether the alarm fires.
• Pump fails mid-phase: pressure climbs, debounce expires, alarm fires with severity High. Existing exceptionAction: Hold puts the batch on hold.
• Phase completes (or transitions to Aborted/Stopped/Complete). Phase watch enqueues the alarm. Reconciler observes no matching phase, sets status.armed=false, and (per Open question 3) auto-clears any alarm that is still active so it does not stick around past the process state that defined it.

Open questions to resolve at implementation time

1. Gate-target shape: extend to OperationTemplate / UnitProcedureTemplate? Start phase-only. The field is a struct (a string would foreclose this), so adding kind later is non-breaking.
2. Multi-state gate semantics. Spec defaults to ["Running"]. Holding and Restarting are interesting because the equipment is still actively driven. Held is borderline, since equipment may have relaxed back. Idle, Complete, Stopped, Aborted are forbidden by validation.
3. Behavior on transition from armed → not-armed while alarm is active. Resolved: auto-clear, symmetric with spec.enabled=false. An initial build chose "preserve" for audit fidelity. The ActiveUnack → ClearedUnack transition still forces operator acknowledgement, so the historical event isn't lost: the activation record is in the audit trail, and the ClearedUnack state keeps the alarm visible until acked. Preserving it as Active instead lied about the alarm's current status: we'd stopped evaluating it but the HMI still claimed a real ongoing fault, which wasted operator attention chasing stale alarms. Re-arming on the next matching phase creates a new Alarm if the condition recurs.
4. Validation of gate targets at admission. A webhook can verify that phaseTemplateRef exists and unitRef resolves to a real Unit. Out of scope for v1. Emit a status condition GateUnresolved instead, so the alarm is never armed against a typo'd reference.

Future work

• Back-reference on PhaseTemplate: add spec.relatedAlarmDefinitionRefs as documentation. Optional sub-controller cross-checks that each referenced alarm has an armingGate pointing back at this template. Surfaces orphans and dangling references.
• HMI surfacing: the alarm faceplate should display the current Suppressed reason ("Out of service" / "Shelved until X" / "Waiting for vacuum-dry phase on FD1") so operators understand silence rather than guessing.
• ISA-18.2 KPI bookkeeping: state-based suppression should not count toward "operator suppression rate" or "shelved-alarm" KPIs. Track separately in the historian.