Skip to content

ADR-0002: Alarms reaction-time stance — BPCS poll cadence, phase guards for fast interlocks

Status: Accepted - Date: 2026-05-11 - Issue: #350 - Related: ADR-0001, #344 (terminology — BPCS vs SIS)

Amended by ADR 0007 (2026-06-12, #567): the interlock designation in this ADR is superseded. The phase SFC guard is not the primary interlock — it evaluates only while a phase executes, and it runs in the procedural operator on the control plane, so it does not survive a control-plane partition. Device-level interlocks now live in the output function blocks (DO/AO ILCK input), evaluated every scan on the edge; the phase guard's role is re-scoped to procedural safe-state sequencing. The alarm reaction-time table and the tagEvalInterval stance below remain in force.

Context

AlarmDefinition is the system's ISA-18.2 / 21 CFR Part 11 alarm record. The AlarmDefinitionReconciler (internal/controller/alarm/alarmdefinition_controller.go) evaluates two condition families:

  • State-driven (StateEquals against IOModule, Unit, ControlModule): watch-driven. The reconciler fires within one reconcile cycle of the source object's status update — typically well under one second.
  • Tag-driven (TagHigh, TagLow, TagHighHigh, TagLowLow, TagDeviation, TagTrue, TagFalse): polled, because live tag values are read out of the unit runtime and do not produce Kubernetes watch events. The poll cadence is the package-level constant tagEvalInterval = 10 * time.Second.

A tag-driven alarm therefore takes poll-phase (0–10 s) + debounceSeconds to fire end-to-end. With the common debounce range of 3–30 s, real-world reaction time lands in the 10–60 s band. That is appropriate for "audit-traceable annunciation" (the role alarms play under ISA-18.2 and 21 CFR Part 11) and inappropriate for hazardous interlocks that need to intervene within a control-loop scan.

The system already provides a deterministic, one-scan path for fast interlocks: an SFC transition guard at priority: -1 in the phase template's chart, leading to a safe-hold step (the "Pattern 3" worked example in alarms-and-interlocks.md). The transition is re-evaluated every scan cycle (100–200 ms) and its response is local to the phase's equipment. Issue #344 formalised this division of labour:

Phase SFC transition guard = primary, deterministic interlock. AlarmDefinition exceptionAction = secondary, audit-traceable companion.

What is missing is an explicit, citable statement of the reaction-time guarantee for each path, so that recipe authors, alarm-rationalisation reviewers, and pharma-validation auditors can find it without reading controller source. ADR-0001 dissolved the Asset layer; ADR-0002 closes the analogous gap in the alarming layer.

The IEC 61511 "BPCS not SIS" framing already lives prominently in docs/architecture.md and docs/library/alarms-and-interlocks.md. This ADR builds on that frame: even within BPCS, we owe operators a documented worst-case reaction-time bound for each alarm path, and we owe them an unambiguous "use this path for fast interlocks" pointer.

Decision

Cloud-Native DCS commits to the following alarm reaction-time characterisation. Reaction time is part of the AlarmDefinition contract and is documented as such; it is not a tunable.

Path Reaction time (worst case) Determinism Use for
Phase SFC transition guard (priority: -1) One scan cycle, 100–200 ms Deterministic, re-evaluated every scan Fast interlocks. Hazardous transitions whose safe action must complete inside one control-loop scan.
AlarmDefinition with StateEquals < 1 s (watch-driven) Deterministic; fires within one reconcile of the source status update State-based annunciation (IOModule fault, Unit offline).
AlarmDefinition with tag-based condition 10 s poll + debounceSeconds (typically 10–60 s end-to-end) Polled; bounded by tagEvalInterval = 10 s Process-value annunciation, audit trail, batch-level exceptionAction.

tagEvalInterval stays at 10 s. The controller's reconciler-loop poll cadence is the documented reaction-time floor for tag-based alarms, not an implementation accident. Recipe authors and alarm-rationalisation reviewers MUST treat tag-based AlarmDefinitions as slow-loop annunciation, not as fast interlocks.

Authors whose process needs a sub-second response to a tag threshold MUST implement it as a priority: -1 SFC transition guard in the relevant phase template (Pattern 3 in docs/library/alarms-and-interlocks.md), optionally paired with a parallel AlarmDefinition whose threshold is slightly more permissive so the alarm fires only when the guard's safe-hold did not recover. That layered pattern is already the recommended shape and is now also the only sanctioned way to get a fast process-value interlock out of this system.

The BPCS-not-SIS disclaimer in docs/architecture.md and docs/library/alarms-and-interlocks.md continues to apply. Safety-rated trips remain out of scope for either path.

Framing reconciled by ADR 0011

ADR 0011 reframed that disclaimer: CNDCS alarm and phase-guard paths are BPCS-layer protection functions that can be credited as independent protection layers (IEC 61511 Clause 9.2), and whether a hazard also requires an independent SIS is a per-process PHA/LOPA decision. This ADR's reaction-time bounds are unaffected — they describe the BPCS paths, which carry no SIL rating.

Alternatives Considered

  • Event-driven tag evaluation (~100 ms reaction) — subscribe to tag changes through OPC UA MonitoredItem, Modbus delta-on-poll, and GPIO edge events; replace the AlarmDefinitionReconciler requeue with a channel-driven evaluator goroutine. Not chosen because it adds a large cross-driver surface (each protocol needs a Subscribe path; Modbus has no native event channel and would still poll under the hood), the benefit duplicates what the SFC guard path already provides deterministically, and the operational story ("which path do I use for fast interlocks?") becomes ambiguous once both paths claim sub-second reaction times. Worth reconsidering only if a use case appears that cannot be expressed as a phase guard — e.g. a fast cross-unit interlock whose safe action is not localisable to one phase.
  • Drop tagEvalInterval from 10 s to 1 s (1–5 s reaction) — a one-line change in alarmdefinition_controller.go. Not chosen because it produces a 10× reconcile-rate amplification across every AlarmDefinition in the cluster for a reaction-time improvement that still does not reach the SFC guard's scan-cycle bound. It would make the alarm path look fast enough to substitute for a phase guard while still being one or more control-loop scans behind, which is the worst of both worlds for an authoring story. The CPU and etcd-write cost on the reference cluster droplet (already capacity-watched; see the 2026-04-30 demo-instance incident record, now in the internal cndcs-operations repo) is non-trivial.
  • Per-AlarmDefinition tunable poll interval — surface spec.evaluationIntervalSeconds on the CRD so authors can speed up hot alarms individually. Not chosen because it offers tuning without removing the policy ambiguity (authors would still try to express fast interlocks as 1-s polled alarms instead of using phase guards), and CRD-shape changes pay a migration cost we do not yet need.

Consequences

  • Code: no change. tagEvalInterval stays at 10 s; the controller's reconcile loop is unchanged. The constant is now policy, not an implementation detail.
  • CRD shape: no change. AlarmDefinition keeps its current fields.
  • Docs (this PR):
  • docs/alarm-management.md gains a "Reaction time" section that states the table above and cross-links to Pattern 3.
  • docs/api-reference.md Alarm Definitions section gains a timing paragraph that quotes the 10-s poll cadence and points at the ADR.
  • docs/library/alarms-and-interlocks.md already carries the side-by-side table from #344; gains a citation back to this ADR.
  • docs/architecture.md gains a one-sentence cross-link to the ADR near the existing BPCS-not-SIS warning.
  • Compliance: the 21 CFR Part 11 and ISA-18.2 audit posture is unchanged — alarms continue to be the system's record-of-truth for annunciated process events. The ADR makes the reaction-time bound citable so that alarm-rationalisation reviews under ISA-18.2 §5.4 can reference it directly.
  • Followups unlocked: #351 (interlock integration tests) can reference this ADR for its assertions about phase-guard timing vs AlarmDefinition timing without having to re-derive the contract. #349 (hazardous-transition validation) gains a clear "fast interlock = phase guard" target for its webhook.
  • Reversibility: if a future use case forces an event-driven tag path, a successor ADR can supersede this one. The doc surface is the only thing this ADR locks in; no code or schema migration would be needed to revisit it.