Monitoring and Metrics¶

Cloud-Native DCS exposes Prometheus metrics from all operators, the historian, and runtimes. Metrics are registered via the controller-runtime metrics registry and served on the /metrics endpoint.

Metrics Endpoint¶

Each operator exposes metrics on port 8080 at /metrics. The historian serves /metrics both on its API port (8092, TLS-covered when configured) and on a dedicated plaintext metrics port (8094, in-cluster only). The OMF egress serves /metrics on 8096, in-cluster only, and has no API port. The Helm chart configures ServiceMonitor resources for Prometheus Operator scraping.

Equipment Metrics¶

dcs_equipment_state¶

Type: Gauge (0 or 1)

Current ISA-88 state of equipment. Only the current state has value 1.

Labels: kind, namespace, name, state

dcs_equipment_state{kind="Unit",namespace="site-newark-plant",name="reactor-1",state="Running"} 1
dcs_equipment_state{kind="Unit",namespace="site-newark-plant",name="reactor-1",state="Idle"} 0

dcs_equipment_transitions_total¶

Type: Counter

Total state transitions by equipment.

Labels: kind, namespace, name, from_state, to_state

Batch Metrics¶

dcs_batch_phase¶

Type: Gauge (0 or 1)

Current phase of batch execution.

Labels: namespace, name, batch_id, phase

dcs_batch_duration_seconds¶

Type: Histogram

Duration of batch execution. Observed when batch reaches a terminal state.

Labels: namespace, name, master_recipe

Buckets: 1m, 2m, 4m, 8m, 16m, 32m, 64m, ~2.1h, ~4.3h, ~8.5h (10 exponential buckets, base 2, starting at 60s)

dcs_batches_total¶

Type: Counter

Total batches by outcome.

Labels: namespace, outcome (complete, stopped, failed, aborted)

dcs_batch_allocation_ranking_failures_total¶

Type: Counter

Total allocation-queue ranking failures. Incremented when the batch controller cannot list contending batches to compute queue order. The batch is requeued with backoff, which preserves queue order where degrading to first-come-first-served would lose it. A rising rate indicates API-server flakiness affecting allocation fairness.

Labels: namespace

Procedural Metrics¶

dcs_procedural_state¶

Type: Gauge (0 or 1)

Current ISA-88 state of procedural elements (Procedure, UnitProcedure, Operation, Phase).

Labels: kind, namespace, name, state

dcs_phase_execution_duration_seconds¶

Type: Histogram

Duration of phase execution. Observed when phase reaches a terminal state.

Labels: namespace, name, unit

Buckets: 1s, 2s, 4s, ... 2048s (~34m) (12 exponential buckets, base 2, starting at 1s)

dcs_phase_engine_stalls_total¶

Type: Counter

Phase ActionChart runs whose SFC engine kept scanning while the chart stopped advancing: no transition condition evaluated and no step changed for the stall grace. Each increment is one stall episode. The same episode also raises an alarm-phase-<phase>-engine-stall alarm and one AuditRecord.

This does not count a phase that is simply dwelling: a timed charge or hold re-evaluates its guard on every scan, which is what the metric measures. A non-zero rate means charts parked in a step action that never returns.

It does not count a chart wedged on a tag it cannot read. That failure is ended by the engine's own escalation after two minutes, three minutes before this metric's grace elapses, and it surfaces as a self-held phase whose message names the failing transition. Reaching for this counter to find unreachable tags finds nothing, however many there are.

Labels: namespace, phase

dcs_phase_control_gap_seconds¶

Type: Histogram

The length of an interval in which no scan of a running phase's action chart ran, observed once per episode, when the phase resumes and measures it. A non-zero observation means the control plane was absent while the process was live.

A step's dwell is wall clock from its activation, so before ADR 0067 the interval counted as hold time. An eleven-minute power loss satisfied a 120-second hold in two scans and the batch ran to Complete unattended. The verdict label is what the resuming run did about it, and it takes one of Hold, Extend, Fail or Count.

Alert on any observation. The verdict label says whether the lot stopped for a human, and the Count verdict is the one to watch, because it means a dwell was declared to survive the outage and nothing paused to check.

Labels: namespace, phase, verdict

Buckets: 60s, 120s, 240s, ... 30720s (~8.5h) (10 exponential buckets, base 2, starting at 60s)

dcs_procedural_budget_exceeded_total¶

Type: Counter

Operations, unit procedures and procedures held because their own spec.timeoutSeconds elapsed. Each increment is one element held. The hold is forwarded to whichever child was running, and an operator decides whether to Restart or Abort.

The clock is wall-clock from status.startTime and keeps running while the element is Held or Paused, so a long investigation hold counts against the budget. That is deliberate: the budget bounds how long material may sit in the element. How long the equipment was busy is a different quantity, and the budget never measures it. It also means an element restarted after a lengthy hold can trip again almost immediately.

A phase blowing its own budget does not land here: that path fails the chart and self-holds the phase, and shows up as a phase execution alarm.

Labels: kind (Operation, UnitProcedure, Procedure), namespace, name

Recipe Metrics¶

dcs_recipes_count¶

Type: Gauge

Number of recipes by kind and phase.

Labels: kind, phase

Runtime Metrics¶

dcs_runtime_reads_total¶

Type: Counter

Total device driver read operations, by outcome. This includes the hardware diagnostic reads behind a channel's diagnosticAddress, and those are the case the error outcome exists for. A diagnostic read that fails leaves the tag at the quality the driver reported, so the wire-break check stops running while the tag still reads Good.

A read that fails on the scan path has no caller to return an error to. The value cache keeps the sample it already had, and the HMI renders that as a stale timestamp. outcome="error" is what says the reads are still being attempted and still being refused.

Labels: namespace, unit, outcome (ok, error)

dcs_runtime_writes_total¶

Type: Counter

Total device driver write operations, by outcome.

Labels: namespace, unit, outcome (ok, error)

dcs_runtime_mqtt_publishes_total¶

Type: Counter

Every message the unit runtime attempted to send, by what it was about and what became of it.

outcome is the reason to alert on this. published reached the broker. queued is buffered for replay while the broker is unreachable, which the queue depth and replay counters below then track. dropped is gone: either the payload would not marshal, or this runtime has no persistent queue to hold it, which is what a runtime started without a data directory does to every message it cannot send.

kind is bounded to the five things a runtime publishes. It is deliberately not the topic: a runtime value topic carries the tag path, so a topic label would be one time series per tag per unit.

Labels: namespace, unit, kind (value, status, hold_event, safe_stop, cm_health), outcome (published, queued, dropped)

Example PromQL:

# Telemetry the runtime lost outright
sum by (namespace, unit) (rate(dcs_runtime_mqtt_publishes_total{outcome="dropped"}[5m])) > 0

dcs_runtime_mqtt_queue_depth¶

Type: Gauge

Number of MQTT messages currently queued for replay while the broker is unreachable.

Labels: namespace, unit

dcs_runtime_mqtt_queue_replayed_total¶

Type: Counter

Total MQTT messages replayed from the queue after broker reconnection.

Labels: namespace, unit

dcs_runtime_mqtt_queue_evicted_total¶

Type: Counter

Total MQTT messages evicted from the replay queue due to the size limit. A growing value means telemetry was lost during a broker outage. It rises only during a rotation. A rotation happens when the queue is already at maxQueueSize and a further message arrives, so any non-zero rate here is data the plant will not get back.

Labels: namespace, unit

dcs_runtime_scan_cycle_jitter_seconds¶

Type: Histogram

Absolute deviation of the observed inter-tick interval from the configured FB network scan period, measured at the moment the scan goroutine consumes the tick. Buckets resolve from 100µs to 2.5s, so single-digit-millisecond jitter on a 100–500ms scan period lands mid-range with headroom on both sides. This is the #943 soak's primary timing metric. The first tick of a run is not observed (it has no predecessor).

Labels: namespace, unit, network

dcs_runtime_scan_cycle_duration_seconds¶

Type: Histogram

Execution duration of one FB network scan cycle: every block's Execute in declared order plus output collection. Compare against the configured scan period: a p99 approaching the period predicts overruns.

Labels: namespace, unit, network

dcs_runtime_scan_cycle_overruns_total¶

Type: Counter

Scan cycles whose execution duration exceeded the configured scan period. While a cycle overruns, ticker ticks are dropped, so the network is effectively scanning slower than configured. A non-zero rate is a capacity signal: the scan period is too tight for the block network on this hardware.

Labels: namespace, unit, network

dcs_runtime_healthy¶

Type: Gauge (0 or 1)

Whether the unit runtime has at least one connected driver (1 = healthy, 0 = unhealthy). This is the signal the unit controller watchdog evaluates.

A runtime the controller cannot read reports 0 as well. The controller reads the status API once per reconcile, and that read can fail in four ways: an unreachable pod, a refused status code, an unreadable body, and an unparseable one. Each of those leaves driver health unknown, which is not the same as healthy. Before #1638 they left the gauge at its last reading. A runtime that went away then kept reporting 1 for as long as the operator process ran, and an alert on this metric could not fire. A pod with no IP assigned yet is the one exception. That is a scheduling state and not a failed read, so it leaves the gauge alone.

Labels: namespace, unit

dcs_runtime_driver_connected¶

Type: Gauge (0 or 1)

Per-driver connectivity status (1 = connected, 0 = disconnected), from the same watchdog poll that sets dcs_runtime_healthy, broken out per named driver. This is the metric to alert on for a lost field connection.

Labels: namespace, unit, driver_name, protocol

dcs_runtime_driver_config_valid¶

Type: Gauge (0 or 1)

Whether the runtime accepted this driver's configuration (1 = applied, 0 = refused), from the same watchdog poll that sets dcs_runtime_driver_connected. The two answer different questions. A simulation module whose profile the runtime refused stays connected and answers every read, so dcs_runtime_driver_connected reads 1 for it while the module runs no behaviours at all and holds every channel at its initial value. Alert on this one for a plant whose tags have stopped moving.

Every driver the watchdog sees is published on every poll, clean ones included. The series therefore exists before it ever goes to 0. The refusal message itself is not a label. It is served by GET /api/v1/diagnostics and shown in the Diagnose panel's I/O Drivers table, which is where an engineer reads which behaviour was rejected.

Labels: namespace, unit, driver_name, protocol

dcs_runtime_watchdog_holds_total¶

Type: Counter

Total times the unit controller watchdog triggered a Hold because the runtime lost driver connectivity.

Labels: namespace, unit

dcs_runtime_lease_renewals_total¶

Type: Counter

Total control-lease renewals accepted by the unit runtime. Only emitted for units with availability mode Failover (ADR 0006). The physical operator renews the lease at a third of the configured lease duration, so a healthy unit shows a steady renewal rate.

Labels: namespace, unit

dcs_runtime_lease_expirations_total¶

Type: Counter

Total times the unit runtime's control lease expired and the runtime self-fenced (stopped FB output writes). Each increment corresponds to a lost-contact episode between the physical operator and the runtime. Alert on any increase.

Labels: namespace, unit

dcs_runtime_fenced¶

Type: Gauge

Whether the unit runtime is currently fenced. 1 means fenced: FB output writes dropped and mutating API routes refused. 0 means controlling. Failover-mode runtimes start fenced until their first lease grant, so a brief 1 at pod startup is normal. A sustained 1 means the operator cannot reach the runtime or the unit has been failed over.

Labels: namespace, unit

dcs_runtime_fenced_writes_dropped_total¶

Type: Counter

Total FB output writes dropped at the driver layer because the runtime was fenced. A rising rate while dcs_runtime_fenced is 1 shows the FB network is still scanning (reads continue) with outputs held.

Labels: namespace, unit

Edge-Local Holding Metrics¶

These expose the ADR 0008 edge-local holding logic: a control-plane partition triggers an armed safe-state SFC chart that the unit runtime runs locally, driving a deliberate sequenced safe state. Freezing outputs at their last value is the failure mode this exists to replace.

dcs_runtime_hold_armed¶

Type: Gauge

Whether the runtime currently has an armed edge-local hold program (1 = armed, 0 = none). A unit with a safeStateChart is armed with that baseline whenever no phase is active. A running phase arms its holdingChart in preference. 0 means a partition would fall back to frozen-at-last-value.

Labels: namespace, unit

dcs_runtime_hold_triggered_total¶

Type: Counter

Total times the runtime ran its armed hold program. The reason label distinguishes autonomy-watchdog (heartbeat loss past holdGraceSeconds) from failover-lease-expiry (control lease lapsed).

Labels: namespace, unit, reason

dcs_runtime_self_held¶

Type: Gauge

Whether the runtime is in an edge-local self-held state (1 = self-held, 0 = not). Set when the hold program runs on a partition and cleared on an ISA-88 Restart after reconnect. A sustained 1 means the edge drove and is holding a safe state without the control plane.

Labels: namespace, unit

dcs_runtime_hold_sequence_seconds¶

Type: Histogram

Wall-clock duration of a hold sequence from trigger to settled safe state. In Failover mode the re-bind margin must exceed this so the partitioned node finishes its safe-state actions and self-fences before a standby takes over.

Labels: namespace, unit

Driver Reconnection Metrics¶

These three cover the network protocol drivers only: modbus, opcua and ethernetip. Those are the ones wrapped in the reconnecting driver, so a simulation unit mints no series here at all. All three are created at zero when a driver is wrapped, and a rate alert therefore has a baseline to rise from before anything has failed.

dcs_driver_reconnect_attempts_total¶

Type: Counter

Reconnection attempts that reached the field. A call turned away by the backoff window or by the max-retries ceiling never tried, so it is not counted here: attempts always equals successes plus failures.

Labels: namespace, unit, protocol

dcs_driver_reconnect_successes_total¶

Type: Counter

Total successful driver reconnections.

Labels: namespace, unit, protocol

dcs_driver_reconnect_failures_total¶

Type: Counter

Total failed driver reconnection attempts. A rising rate with no matching successes indicates a device or network that stays down.

Labels: namespace, unit, protocol

Control Module Observability Metrics¶

These expose ISA-88 Control Module continuous-observability health. A Control Module's FB network is designed to run continuously. When it isn't, the runtime falls back to publishing raw driver reads so operators never see a silent -- on the HMI.

dcs_cm_fb_network_running¶

Type: Gauge (0 or 1)

Whether a Control Module's FB network is currently scanning. 1 when state is Running or Degraded (the scan is executing and regulating the healthy outputs). 0 when Down, CompileError, or Unknown.

Labels: namespace, unit, cm

dcs_cm_program_degraded_total¶

Type: Counter

Total times a Control Module's FB program entered the Degraded state: one or more blocks faulted on a scan and were skipped while the program kept running and driving every healthy output (ADR 0009). A growing value means an output's driver write (or other block) is repeatedly erroring. The faulted output holds its last value while the rest of the program continues. Pair with the CM health topic's Degraded state to see which blocks faulted.

Labels: namespace, unit, cm

dcs_cm_raw_fallback_publishes_total¶

Type: Counter

Total raw-I/O fallback values published for a CM because its FB network is stopped. A non-zero and growing value indicates the CM's engineering- scaled path is unavailable. Operators are seeing driver-direct values.

Labels: namespace, unit, cm

dcs_cm_raw_fallback_skipped_total¶

Type: Counter

Total raw-fallback attempts skipped, labeled by reason: - no_raw_address — variable-only / computed tag has no physical address (intentional, since publishing would fabricate data) - resolve_error — address resolution or driver routing failed - read_error — driver read failed

Labels: namespace, unit, cm, reason

Alarm Metrics¶

dcs_alarms_active¶

Type: Gauge

Number of currently active alarms.

Labels: namespace, type, severity

dcs_alarms_total¶

Type: Counter

Total alarms raised (lifetime).

Labels: namespace, type, severity

dcs_alarm_annunciation_latency_seconds¶

Type: Histogram

Time from the generating reconciler observing an alarm condition met to the broker accepting the retained AlarmEvent publish (QoS 1), which is the transport every annunciation consumer subscribes to. Observed only on transitions into ActiveUnacknowledged. Clears, acknowledgements, and shelve traffic are not annunciations and observe nothing, as does a failed publish. Consumers add their own transport on top of this number: the HMI its alarm poll interval (ADR 0029), external subscribers their broker round trip. The origin label names the generating path: state (IOModule/Unit state generator), definition (AlarmDefinition condition evaluation, excluding any configured debounce), or server (the node condition sweep).

Labels: namespace, origin

SyncBarrier Metrics¶

dcs_syncbarrier_state¶

Type: Gauge (0 or 1)

Current state of sync barrier.

Labels: namespace, name, state

dcs_syncbarrier_wait_duration_seconds¶

Type: Histogram

Duration phases waited at sync barrier.

Labels: namespace, name

Buckets: 1s, 2s, 4s, ... 512s (~8.5m) (10 exponential buckets, base 2, starting at 1s)

dcs_syncbarrier_timeouts_total¶

Type: Counter

Total sync barrier timeouts.

Labels: namespace

CoordinationBlock Metrics¶

dcs_coordinationblock_state¶

Type: Gauge (0 or 1)

Current state of a coordination block. Only the current state has value 1.

Labels: namespace, name, mode, state

dcs_coordination_signal_latency_seconds¶

Type: Histogram

Time from a signal being raised to all consumers being notified.

Labels: namespace, name

Buckets: 0.5s, 1s, 2s, ... 256s (~4.3m) (10 exponential buckets, base 2, starting at 0.5s)

dcs_coordination_timeouts_total¶

Type: Counter

Total coordination block timeouts.

Labels: namespace, mode

Unit Allocation Metrics¶

Both gauges are recomputed by the physical operator on every Unit reconcile, across the whole namespace. A process cell that empties therefore drops back to zero, and never holds its last reading. A census the operator cannot take leaves the previous values standing and logs that it did. A frozen number is a worse answer than a stale one only when nothing says so.

They do not partition the fleet. A unit that is Running without having been allocated (an ad-hoc phase) and a unit that is Held are neither allocated nor available, and the gap between the two numbers is what a capacity view is looking at.

dcs_units_allocated¶

Type: Gauge

Number of units currently held by a batch (status.allocatedTo is set).

Labels: namespace, process_cell

dcs_units_available¶

Type: Gauge

Number of units the allocator would consider a candidate right now, before any capability or property requirement narrows the field. This is the allocator's own test, so the gauge cannot drift from what allocation does. A unit qualifies when it is Idle or has no state yet, is free of any status.allocatedTo, and carries no other batch's in-flight claim annotation.

Labels: namespace, process_cell

Audit Metrics¶

dcs_audit_records_created_total¶

Type: Counter

Total audit records created.

Labels: action, target_kind, result, component

dcs_audit_record_failures_total¶

Type: Counter

Total audit records that failed to persist after the recorder's bounded retry. On best-effort paths (generic controllers, the audit-flux-bridge) the record is dropped. On the change-control approval/apply path the failure withholds the state transition, since advancing un-auditably is the one outcome the path refuses. A 21 CFR Part 11 audit trail must not silently lose records, so any non-zero rate warrants an alert. It means an action occurred (best-effort paths) or is stalled (approval/apply) without its compliance artifact.

Labels: action, target_kind, component

# Any audit-record write failure in the last 15 minutes
increase(dcs_audit_record_failures_total[15m]) > 0

TLS Certificate Metrics¶

dcs_cert_expiry_seconds¶

Type: Gauge

Unix timestamp of the NotAfter field of each TLS certificate loaded by a component (server cert/CA and MQTT cert/CA). Subtract time() in PromQL to get seconds until expiry. Keeping the sample static between polls defers clock arithmetic to Prometheus. Exposed by the unit runtime, io-probe, historian, and gateway.

Labels: component, host, file

# Certificates expiring within 14 days
dcs_cert_expiry_seconds - time() < 14 * 24 * 3600

Reconciliation Metrics¶

dcs_reconcile_total¶

Type: Counter

Total reconciliations by controller and result.

Labels: controller, result (success, error, requeue, requeue_after, invalid_command, stale_view)

stale_view counts passes abandoned because the informer cache served a state the object had already left. The pass writes nothing. The write that moved the state is itself a watch event, so a fresh pass follows on its own. A low background rate is the cache doing its job. A sustained one says reconciles are arriving faster than the cache settles, which is the condition #1615 was found under.

dcs_reconcile_duration_seconds¶

Type: Histogram

Duration of reconciliation loops.

Labels: controller

dcs_device_health_unknown_total¶

Type: Counter

NE 107 device-health polls that resolved to Unknown, by reason (ADR 0043): ReadFailed (the io-probe could not reach or read the declared address), NotReported (the device answered with no value), UnrecognizedValue (a numeric outside the pinned DeviceHealthEnumeration). A healthy fleet holds this flat. A rising ReadFailed slope is a probe/reachability problem, and the device itself is the wrong suspect.

Labels: namespace, reason

Historian Metrics¶

dcs_historian_ingested_total¶

Type: Counter

Total records ingested by type.

Labels: type (tags, alarms, states, cm_health)

dcs_historian_flush_duration_seconds¶

Type: Histogram

Time spent flushing buffered data to the database.

Labels: type (tags, alarms, states, cm_health)

dcs_historian_buffer_size¶

Type: Gauge

Current number of buffered records awaiting flush.

Labels: type (tags, alarms, states, cm_health)

dcs_historian_flush_failures_total¶

Type: Counter

Flushes whose database insert failed. The batch is requeued in front of the buffer and retried on an exponential backoff (doubling from the flush interval, capped at 30s). A transient database blip therefore loses nothing. dcs_historian_ingested_total stalls while this counter advances, then catches up on recovery.

Labels: type (tags, alarms, states, cm_health)

dcs_historian_dropped_total¶

Type: Counter

Records dropped because the retention cap (10× the configured buffer size, per type) was exceeded during a storage outage. This is the one remaining drop path, and it takes the oldest records first. The cap is enforced wherever a buffer grows: on a failed flush requeueing its batch, and on a message arriving while flushes are gated by the retry backoff. The #943 soak's "no dropped points" criterion is this counter staying at zero for the full window.

The historian's log reports only that a buffer reached the cap and later drained below it, and while the cap holds it repeats at most once per 30 seconds per type (#1607). The per-record account is this counter.

Labels: type (tags, alarms, states, cm_health)

dcs_historian_requeued_total¶

Type: Counter

Records put back in front of the buffer after their batch's insert failed. A record retried across N failed flushes counts N times. The counter therefore measures retry churn during an outage, and unique records at risk is a different quantity. dcs_historian_dropped_total is the loss record.

Labels: type (tags, alarms, states, cm_health)

dcs_historian_ingest_lag_seconds¶

Type: Histogram

Age of the oldest sample in each successfully flushed batch, from sample timestamp to insert completion. Healthy lag sits just above the flush interval. A rising trend means ingest is falling behind sustained input even while throughput counters still look busy.

Labels: type (tags, alarms, states, cm_health)

dcs_historian_query_duration_seconds¶

Type: Histogram

Query endpoint latency.

Labels: endpoint (tags, alarms, states, batch_tags, batch_alarms, batch_states, audit, audit_manifests, audit_manifest_records)

OMF Egress Metrics¶

The OMF egress carries plant state northbound to a customer's OMF endpoint (PI Web API, Edge Data Store, or AVEVA Data Hub). It reads the same dcs/ topic tree the historian ingests, holds what it has read in a bounded in-memory queue, and posts it in order.

Two things shape what these metrics have to answer. The endpoint being slow or down is the normal case, so a failing post is not by itself a fault. And PI reads an out-of-order arrival as a data quality problem. The component therefore holds a failed batch at its queue position, where later records cannot overtake it. A rising dcs_omf_egress_oldest_pending_seconds with dcs_omf_egress_endpoint_up at 0 is an outage being ridden out correctly. The same shape with dcs_omf_egress_dropped_total advancing is data that will not arrive.

dcs_omf_egress_endpoint_up¶

Type: Gauge

1 when the last post to this endpoint succeeded, 0 when it failed. OMF is stateless HTTP, so there is no held connection to report and this is the outcome of the most recent attempt.

Labels: endpoint (the configured OMF URL)

dcs_omf_egress_records_total¶

Type: Counter

Records read off the dcs/ topic tree and queued for egress. This is the input side. Compare it against dcs_omf_egress_posted_total to see whether the endpoint is keeping up.

Labels: kind (tag, state, alarm, cm_health)

dcs_omf_egress_posted_total¶

Type: Counter

OMF messages the endpoint accepted.

Labels: messagetype (type, container, data)

dcs_omf_egress_post_duration_seconds¶

Type: Histogram

Time spent posting one OMF message, from request to response. A latency that climbs while dcs_omf_egress_endpoint_up stays at 1 is the endpoint under load, and the queue depth is where that shows up next.

Labels: messagetype (type, container, data)

dcs_omf_egress_retries_total¶

Type: Counter

Post attempts that failed retryably and will be attempted again from the same queue position. Retryable means a transport error, a 408, a 429, or any 5xx: the endpoint being unreachable or overloaded. Attempts are gated by an exponential backoff doubling from the flush interval, so this counter rises slowly during an outage, on the backoff's schedule and no faster. Nothing is lost while it advances. dcs_omf_egress_dropped_total is the loss record.

Labels: messagetype (type, container, data)

dcs_omf_egress_batch_splits_total¶

Type: Counter

Batches that were cut into smaller ones to fit the maximum OMF message body, which the specification sets at 192 KB. The component batches by record count, and the bytes that count produces depend on how long a site's equipment names are, because every OMF message carries canonical paths in full. A batch measured over the cap is split and projected again, and this counter is what says the configured batch size is larger than this plant's naming allows. Nothing is lost when it advances.

Labels: trigger --

preflight -- the projected batch measured over our own bound before anything was posted. This is the ordinary case, and a steady rate of it is the signal to lower the configured batch size. Nothing is at fault.
endpoint -- the endpoint refused a message that our bound had passed. This one says the bound itself is wrong: our 192 KB is read from the specification and has never been checked against a real PI Web API, which is what check D3 of hack/pi-omf-conformance/README.md measures. A non-zero count here is the evidence that it needs lowering. Here the evidence arrives from a customer site, ahead of any conformance run.

dcs_omf_egress_dropped_total¶

Type: Counter

Records that will never reach the endpoint. This is the only counter that records loss, and it exists because a silent drop is indistinguishable from a quiet plant: the trend in PI goes flat either way.

Labels: reason --

capacity -- the queue hit its retention cap while the endpoint was unreachable, and the oldest records went first. Raise the cap or shorten the outage.
rejected -- the endpoint refused the batch in a way no retry changes: a 4xx other than 408 or 429, or a 207 naming messages it would not take. Holding the queue behind it would stop egress for the whole plant over one batch, so it is dropped and counted here.
unprojectable -- the projection could not build OMF messages from the batch at all.
oversize -- one record on its own projects to an OMF message larger than the body cap, and a batch of one cannot be split any further. Unlike every other reason here this one is about a declaration, with no load in it: it will not clear when the plant quietens or the endpoint recovers, and it recurs for as long as that equipment publishes. The fix is the declaration that produced it, usually a name or a description far longer than the rest of the site's.
shutdown -- the process exited with records still queued. The queue is memory-resident, so a restart resumes from the live feed and leaves a gap.

The egress log reports only that the queue reached the cap and later drained below it, and while the cap holds it repeats at most once per retry-backoff interval (#1609). The per-record account is this counter.

dcs_omf_egress_queue_depth¶

Type: Gauge

Records queued for projection and egress. A depth pinned at the configured retention cap means dcs_omf_egress_dropped_total{reason="capacity"} is advancing.

dcs_omf_egress_oldest_pending_seconds¶

Type: Gauge

Age of the oldest queued record, or 0 when the queue is empty. This is how far behind the endpoint has fallen, and it is the number to alert on: it rises during an outage and returns to near the flush interval on recovery.

dcs_omf_egress_refused_total¶

Type: Counter

Declarations the projection will not send, because PI cannot carry the name they project onto. A refusal is a declaration for an engineer to correct. The component and the endpoint are both healthy when it fires.

PI's naming rules are narrower than ours, and one of our name sources is unvalidated: a ControlModule tag name is a plain string with no pattern on it, while every equipment name is DNS-1123 and cannot collide with the rejected set. ADR 0056 § 4 refuses such a declaration outright, because rewriting it (substituting a character, truncating a long name) collapses two distinct declarations onto one PI Point and neither side is visible afterwards. The refusal is loud on the day the declaration is written. The substitution stays silent until two tags fight over one point.

Each refusal is also logged once, at error level, naming the canonical path and the declaration at fault. Once is the operative word, because a refused tag publishes at its scan rate, and a per-sample log line would flood.

Labels: reason --

illegal_character -- the name carries one of * ? ; { } [ ] | \ ` ' " or a control character. The log line names which.
name_too_long -- the projected Container id or element name is over 200 characters. A configured Container id prefix counts toward that limit, so a long prefix can refuse a stream that would otherwise have projected.
name_collision -- two declarations under one parent land on one name once PI's case-insensitivity is applied. A tag colliding with one of the State, Health or DeviceHealth context attributes loses, since those carry the same meaning at every site. Two tags colliding with each other are both withheld, because neither is more entitled to the name. An equipment resource whose metadata.name folds onto a context element its parent carries loses the same way: a control module or equipment module named Batch or Alarms under a unit, or named Alarms under an equipment module.
empty_name -- a name segment is empty, or is empty once the whitespace around it is trimmed, and addresses nothing either way.
surrounding_whitespace -- the name carries leading or trailing whitespace. PI trims it, so the name lands on the same object as the same name without it. Only the padded declaration is refused. Unlike a case collision, one of the two is plainly the name that was intended.
reserved_prefix -- the name begins with the double underscore PI reserves for its own system names, which OMF's predefined messages also use.

dcs_omf_egress_refused_streams¶

Type: Gauge

Distinct declarations currently refused. Non-zero means a declared tag or element is absent from PI and will stay absent until the declaration is corrected. This is the one to alert on: the counter above says a refusal happened, and this says the condition is still current.

dcs_omf_egress_skipped_total¶

Type: Counter

Records that produced no OMF at all. A skip is not loss the way dcs_omf_egress_dropped_total is: it is the projection declining to guess, or honouring a site that has said this equipment does not leave the cluster.

Labels: reason --

not_published -- the Site CR does not declare northbound publication, or declares it while excluding this unit. Publication is opt-in per site and defaults to off (ADR 0044 § 6). A fresh deployment therefore publishes nothing until a site says otherwise, and every record it reads lands here.
undeclared_tag -- the value is not a declared tag. The runtime publishes raw driver addresses on the same topic, and ADR 0044 § 2 keeps what is not declared off the northbound surface. A steady rate here is normal on a plant with raw I/O. It is not a sign of anything wrong.
unresolved -- the equipment is not in the plant model, or a hop in its declared containment chain is missing. A CR deleted while its retained MQTT message still stands is the ordinary cause. A rate that climbs and stays up means a broken chain: an Area a ProcessCell names but that does not exist, for instance.
unsupported_source -- the record addresses something that is not a node of the equipment tree, such as an alarm raised on an IOModule. An Operation and a Procedure land here too: both publish their state to the equipment-state topic, both sit above the unit in the procedural model, and neither names one unit to be context on. A Phase or a UnitProcedure becomes its unit's ActiveElement context and lands here only when its payload names no unit.
value_type_mismatch -- the value on the wire does not fit the OMF type the tag's declared dataType projects onto. The endpoint rejects a whole message carrying a string where the type declares a number, and a rejected batch is dropped entire. Skipping the one mistyped tag spares every good record travelling with it. Anything here is a dataType declaration that disagrees with what the runtime publishes.
no_value -- the payload carried nothing to write.

MQTT Connectivity Metrics¶

dcs_mqtt_connected¶

Type: Gauge (0 or 1)

Whether the component's connection to the MQTT broker is up (1 = connected, 0 = disconnected or still retrying the initial connect).

The connection is managed in the background with capped exponential backoff. Components therefore start and stay ready even when the broker is down (cold-start ordering, broker outage). A sustained 0 means the component is running degraded (equipment/alarm events are not published and the HMI real-time feed is stale) and recovers automatically once the broker is reachable. This gauge must never be wired into readiness probes: failing readiness while the broker is down would just recreate the outage.

Labels: component (physical-operator, procedural-operator, batch-operator, dcs-gateway)

Gateway Metrics¶

The gateway exposes /metrics on port 61051 using the same controller-runtime registry as the other components. The Helm chart deploys a dedicated gateway ServiceMonitor for it (the operator ServiceMonitors select only their own metrics Services and never covered the gateway), and a PodMonitor that scrapes unit-runtime pods across all site namespaces. Both are gated by monitoring.enabled and monitoring.serviceMonitor.enabled. The historian serves the same registry twice: on its API port (covered by TLS when configured) and on a dedicated plaintext metricsPort (default 8094, in-cluster only) that its own ServiceMonitor scrapes, since a stock Prometheus scrape cannot reach the mTLS copy.

dcs_gateway_request_duration_seconds¶

Type: Histogram

Duration of HTTP requests served by the gateway, observed just outside the panic-recovery middleware so error responses (including recovered panics) are recorded with their final status code.

Labels: route, method, status

route is the /api/v1/<resource> first segment from an allowlist (e.g. batches, sites, audit, historian, ws). Unknown paths, static assets, /metrics, and /healthz fold into other so the time-series cardinality stays bounded regardless of path parameters like {site} or {name}.
method is the HTTP method (GET, POST, PUT, DELETE, OPTIONS).
status is the numeric response status code as a string.

Buckets: 1ms, 2ms, 4ms, ... 16s (15 exponential buckets, base 2)

Example PromQL:

# p99 request latency across the gateway
histogram_quantile(0.99, rate(dcs_gateway_request_duration_seconds_bucket[5m]))

# p95 latency broken out per resource family
histogram_quantile(0.95,
  sum by (route, le) (rate(dcs_gateway_request_duration_seconds_bucket[5m])))

# Request rate by status class
sum by (status) (rate(dcs_gateway_request_duration_seconds_count[1m]))

This histogram is the signal backing the "Gateway API latency" scale threshold in capacity-planning.md.

dcs_gateway_stream_frames_dropped_total¶

Type: Counter

Frames the gateway could not hand to a live WebSocket or SSE consumer because that connection's send buffer was full. Each connection holds 256 slots, and the gateway sheds a frame that finds no room. Blocking the whole fan-out behind one slow consumer is the alternative it refuses.

The consumer is told what it lost, in the sequence numbers it receives and in a gap frame, and this counter is the same fact from the plant's side. A rising rate means some consumer is slower than the fan-out, and what it is missing is not only tag values: those are republished every 200 ms and correct themselves within a tick, while alarm and state frames publish on transition only and are never restated. See scada-integration.md.

Unlabelled. The question it answers is whether the gateway is shedding at all. A per-connection label would be unbounded, since a SCADA holds one connection for months while a browser opens a new one on every reload.

Example PromQL:

# Any consumer falling behind at all
rate(dcs_gateway_stream_frames_dropped_total[5m]) > 0

dcs_gateway_stream_connections_refused_total¶

Type: Counter

Labels: transport (websocket, sse)

Live-stream connects refused because the source address already held the configured maximum (gateway.streamLimits.maxConnectionsPerIP, ten by default, counting both transports together).

This is how a plant learns it has outgrown the ceiling it is running. A redundant SCADA pair NATed to one egress address reaches ten quickly, and so does a control room fronted by one address. Nothing looks wrong from the outside, because the connections that did get through keep delivering. The refused client is told, in a 429 and a Retry-After, but nobody should have to read a SCADA's logs to find out that its redundant half never connected.

Labelled by transport only. Which of the two a consumer chose is the part worth knowing when sizing the ceiling. The source address would be an unbounded label, and it is already in the gateway log line beside each refusal. See scada-integration.md.

Example PromQL:

# Anything being turned away at all
sum by (transport) (rate(dcs_gateway_stream_connections_refused_total[5m])) > 0

Alerting Suggestions¶

Recommended Prometheus alert rules for pharmaceutical environments:

groups:
  - name: dcs-alerts
    rules:
      - alert: CriticalAlarmActive
        expr: dcs_alarms_active{severity="Critical"} > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Critical alarm active in {{ $labels.namespace }}"

      - alert: RuntimeDisconnected
        expr: dcs_runtime_driver_connected == 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Runtime {{ $labels.unit }} lost driver {{ $labels.driver_name }}"

      - alert: RuntimeDriverConfigRefused
        expr: dcs_runtime_driver_config_valid == 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Runtime {{ $labels.unit }} refused the configuration of driver {{ $labels.driver_name }}"
          description: "The driver is connected and its configuration was thrown away. Read GET /api/v1/diagnostics on the unit runtime, or the Diagnose panel's I/O Drivers table, for the message."

      - alert: ReconcileErrorRate
        expr: rate(dcs_reconcile_total{result="error"}[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.controller }} reconcile error rate elevated"

      - alert: HistorianBufferBacklog
        expr: dcs_historian_buffer_size > 5000
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Historian {{ $labels.type }} buffer backlog"

      - alert: BatchFailed
        expr: increase(dcs_batches_total{outcome="failed"}[1h]) > 0
        labels:
          severity: warning
        annotations:
          summary: "Batch failed in {{ $labels.namespace }}"

Grafana Dashboards¶

Pre-built reference dashboards ship in grafana/dashboards/. They consume the Prometheus metrics documented above and are a starting point: copy them into your own Grafana instance and adapt panels and thresholds to your deployment.

Dashboard	File	Focus
Plant Overview	`plant-overview.json`	Equipment totals, running units, active batches, state transitions
Unit Detail	`unit-detail.json`	One unit's state, allocation, runtime connectivity, device read/write rates
Batch Monitor	`batch-monitor.json`	A running batch: phase, state, duration, allocated units, phase progress
Batch History	`batch-history.json`	Completed batches: outcomes, duration distribution, recent list
Phase Timeline	`phase-timeline.json`	Phase states over time, durations, active phases
Alarm Console	`alarm-console.json`	Active alarms and warnings, alarm rate, reconcile errors
Audit Compliance	`audit-compliance.json`	21 CFR Part 11 monitoring (detailed below)

The Helm chart does not install dashboards automatically, since that would bind the chart to a specific Grafana instance and namespace. The recommended workflow:

helm install the DCS chart, which creates the ServiceMonitors and PrometheusRules. On a cluster without the Prometheus Operator CRDs the chart skips them, and a helm upgrade after step 0 picks them up (#1345).
Import the dashboards via the Grafana UI (Dashboards > Import), provision them through your existing Grafana dashboard provisioning config, or deploy them as ConfigMaps with the Grafana sidecar.
Review each panel's thresholds and customise for your capacity plan.

Audit Compliance Dashboard¶

grafana/dashboards/audit-compliance.json -- 21 CFR Part 11 compliance monitoring with panels organized into rows:

Summary: Total records, approvals, rejections, failures, signature errors, reconcile error rate (24h stats)
Audit Record Creation: Rate by action type and result over time
Actor Activity: Records by component (pie chart), rate by component
Electronic Signatures: Approval/rejection timeline, signatures by signer
Record Modifications: Top modified resource kinds, state transitions
Compliance Alerts: Pipeline health, unsigned changes, sequence gaps, webhook rejections
System Health: Audit record creation latency, reconcile duration

Some panels require metrics that are not yet instrumented (noted in the dashboard's "Metric Prerequisites" panel). These panels show "No data" until the metrics are implemented.

Coverage Gaps¶

The shipped set does not yet cover:

Fleet-wide runtime connectivity: a status map of every runtime at once (Unit Detail shows one unit at a time)
Historian ingestion rate: counter rate by type
Per-controller reconcile error rate: beyond the single panels on Alarm Console and Audit Compliance

Distributed Tracing¶

All operators, the gateway, and unit runtimes support OpenTelemetry distributed tracing via OTLP gRPC export. When enabled, traces flow across the full request chain: gateway HTTP request → operator reconcile → runtime HTTP call.

Enabling Tracing¶

Set the Helm values:

observability:
  tracing:
    enabled: true
    endpoint: "tempo:4317"  # or jaeger-collector:4317

This injects OTEL_EXPORTER_OTLP_ENDPOINT into all operator and gateway pods. The unit runtime inherits the endpoint from the physical-operator, which passes it to runtime pod specs automatically.

When the endpoint is empty or tracing is disabled, the tracer provider is a no-op with zero overhead.

Instrumented Components¶

Component	Spans
Gateway	HTTP request spans (auto via `otelhttp`), excludes SSE/WebSocket
Gateway → Runtime	Outgoing proxy calls propagate `traceparent` header
Unit reconciler	`Reconcile/Unit` with resource name/namespace
Phase reconciler	`Reconcile/Phase` with resource name/namespace
Operation reconciler	`Reconcile/Operation` with resource name/namespace
Batch reconciler	`Reconcile/Batch` with resource name/namespace
ControlProgram reconciler	`Reconcile/ControlProgram` with resource name/namespace
Unit runtime	HTTP handler spans (auto via `otelhttp`)

Trace Context Propagation¶

W3C Trace Context (traceparent header) is used for propagation. The gateway and operators wrap outgoing HTTP clients with otelhttp.NewTransport(), so runtime spans appear as children of the originating gateway or operator span.

Troubleshooting -- Using metrics to diagnose issues
Alarm Management -- Alarm metrics details
Historian -- Historian-specific metrics
Architecture -- System overview