MQTT Telemetry

MQTT Telemetry is the DCS's outbound, publish-only telemetry feed: the system publishes real-time tag values, alarm events, state transitions, and batch-lifecycle messages to an MQTT broker, and external consumers (dashboards, historians, analytics pipelines) subscribe. Nothing flows back in through this path. Commands and configuration use the REST APIs. It uses MQTT v5, and the client is built on Eclipse Paho's autopaho library for managed connections with automatic reconnection.

Client Configuration

The MQTT client is configured via the Helm chart or environment variables on individual components (runtimes, physical/batch/procedural operators, historian).

Parameter	Description
Broker URL	tcp://broker:1883 or tls://broker:8883
Client ID	Unique per component (e.g., dcs-historian, runtime-reactor-1)
Username/Password	Optional MQTT authentication credentials
CA File	CA certificate for broker verification
Cert File	Client certificate for mutual TLS
Key File	Client private key for mutual TLS
Clean Start	Whether to discard previous session state

TLS Configuration

TLS is automatically enabled when any TLS-related field (CA, cert, key) is set. The client enforces TLS 1.2 as the minimum version.

• Server verification only: Set CAFile to the broker's CA certificate
• Mutual TLS: Set CAFile, CertFile, and KeyFile
• Development only: InsecureSkipVerify skips server certificate verification (never use in production)

Reconnection

The autopaho library handles automatic reconnection with exponential backoff (capped at 30 seconds). All topic subscriptions are re-established on reconnect.

The same mechanism covers the initial connection: the operators and the gateway start their broker connection in the background and keep retrying until it succeeds, so a broker that comes up after them (cold-start ordering, single-node reboot) is picked up automatically without a pod restart. While disconnected the components run degraded. Publishes fail fast and are logged, subscriptions are deferred to the first successful connect, and the dcs_mqtt_connected gauge reads 0 (see Monitoring & Metrics).

Topic Structure

All topics follow the prefix dcs/ and use a hierarchical namespace. The topics actually published today are:

dcs/
  {namespace}/
    equipment/{kind}/{name}/
      state              ISA-88 state of equipment and procedural elements
      alarm              Alarm events (ISA-18.2)
    equipment/controlmodule/{cmName}/
      health             Control Module FB-network health (retained)
    runtime/{unitName}/
      status             Runtime connection state (retained)
      value/{nodeID}     Individual FB output values
      hold-event         ADR 0008 edge-local hold lifecycle
      safe-stop          Terminal-stop safe-state write-out
    batch/{batchID}/
      state              Batch lifecycle state changes (retained)

Note that procedural elements (Procedure, UnitProcedure, Operation, Phase) publish their ISA-88 state to the equipment state topic with the element kind in the {kind} segment. There are no separate per-phase or per-procedure topic subtrees on the wire.

The two {kind} segments do not carry the same set of values, which is worth knowing before you point a subscriber at the tree:

• state is published for Unit and for the four procedural kinds above, and for nothing else.
• alarm is published for the alarm's source kind: Unit, IOModule or ControlModule. An IOModule or ControlModule branch therefore holds alarm topics and no state topic.

That is also why a browsing client shows a lowercase controlmodule branch next to a capitalised ControlModule one. They are different topics rather than a naming inconsistency: controlmodule is the fixed literal segment of the Control Module health topic, while ControlModule is a resource kind substituted into {kind}.

Topic Functions

The system provides helper functions (pkg/mqtt/topics.go) for constructing topics. All components use these helpers to ensure consistent naming. Some helpers are reserved for future use and have no publisher today. Nothing appears on those topics.

Function	Topic Pattern	Published by
TopicEquipmentState	dcs/{ns}/equipment/{kind}/{name}/state	physical-operator (Unit), procedural-operator (Procedure, UnitProcedure, Operation, Phase)
TopicEquipmentAlarm	dcs/{ns}/equipment/{kind}/{name}/alarm	alarm controller
TopicControlModuleHealth	dcs/{ns}/equipment/controlmodule/{cm}/health	unit runtime — the module's FB-network health, a different signal from its instrument's NE 107 self-diagnosis (see Device health for a maintenance system)
TopicRuntimeStatus	dcs/{ns}/runtime/{unit}/status	unit runtime
TopicRuntimeValue	dcs/{ns}/runtime/{unit}/value/{nodeID}	unit runtime
TopicRuntimeHoldEvent	dcs/{ns}/runtime/{unit}/hold-event	unit runtime — ADR 0008 edge-local hold-lifecycle events (armed / triggered / action / settled / self-fenced / released), store-and-forward buffered and materialized into AuditRecords on replay
TopicRuntimeSafeStop	dcs/{ns}/runtime/{unit}/safe-stop	unit runtime — the record that a terminal stop drove the field to its configured safe values (ADR 0009 amendment, #1283), buffered and materialized into an AuditRecord the same way
TopicBatchState	dcs/{ns}/batch/{batchID}/state	batch-operator
TopicEquipmentCommand	dcs/{ns}/equipment/{kind}/{name}/command	reserved — no current publisher; commands flow through the REST API
TopicEquipmentTelemetry	dcs/{ns}/unit/{unit}/telemetry	reserved — no current publisher
TopicBatchEvent	dcs/{ns}/batch/{batchID}/event	reserved — no current publisher
TopicPhaseState	dcs/{ns}/phase/{phase}/state	reserved — Phase states publish to the equipment state topic instead
TopicProcedureState	dcs/{ns}/procedure/{proc}/state	reserved — Procedure states publish to the equipment state topic instead

Seen from an ordinary MQTT client

Everything above is a claim about what is on the wire, and the wire is the one place it can be checked. Below, MQTT Explorer (a third-party client, connected to the broker with no credentials and nothing configured for it) subscribes to dcs/# on a running plant and draws the tree itself.

A third-party MQTT client on the live topic tree: control-module health, alarms, runtime status and batch state are retained, so they are present the moment it connects. The client marks the batch lifecycle message RETAINED itself. That message carries the units the batch holds, which is the binding every other record is attributed through. A tag payload is the runtime's driver.TagValue marshaled as-is, capitalised keys and all.

Publishers

Unit Runtime

The runtime publishes real-time FB output values:

• Topic: dcs/{ns}/runtime/{unit}/value/{nodeID}
• Payload: {"Address": "reactor-1.temp", "Value": 23.5, "Quality": "Good", "Timestamp": "2026-07-02T10:00:00Z"} The wire format is the runtime's driver.TagValue struct marshaled as-is, so the keys are capitalized (Value, Quality, Good/Bad/ Uncertain quality strings, RFC 3339 timestamp)
• Frequency: Every 200 ms (a fixed publish interval, independent of the FB scan cycle, with no configuration knob)

It also publishes its own connection status:

• Topic: dcs/{ns}/runtime/{unit}/status
• QoS: 1 (at-least-once)
• Retained: true — a client that connects mid-batch learns immediately whether the runtime is up, with no wait for the next transition
• Payload: the unit and namespace, the driver protocol, status (online, fenced or offline), the lease availabilityMode and epoch, and the ADR 0008 edge-hold fields selfHeld and holdProgram
• Published: on connect, on every fencing transition, and once more on the way down carrying status: offline

And, for every Control Module it hosts, the FB-network health:

• Topic: dcs/{ns}/equipment/controlmodule/{cm}/health
• QoS: 1 (at-least-once)
• Retained: true (latest health is always available)
• Published: on every health transition (Running, Down, CompileError, Unknown). Consumed by the gateway to surface per-CM health on the HMI, and by the historian for the health timeline
• Which unit it came from: unit names the runtime's own unit. The topic is addressed by control module, so without this field nothing on the wire says which unit the record belongs to. A subscriber holding no Kubernetes client then cannot attribute it to a batch. See Which batch a record belongs to

Alarm Controller

Publishes on every alarm state transition:

• Topic: dcs/{ns}/equipment/{kind}/{name}/alarm
• QoS: 1 (at-least-once)
• Retained: true (latest alarm state is always available)
• Payload: JSON with alarm name, source, type, severity, state, message, timestamp, and the shelve-suppression fields shelved (bool) and shelvedUntil (RFC3339, present only when shelved is true)
• Which unit it came from: unit names the unit the alarm source sits under, resolved by the operator. A Unit source is its own unit and a ControlModule source resolves through its declared parent. An IOModule source carries no unit, because one I/O module can serve control modules under several units and naming one of them would be a guess (see Which batch a record belongs to)
• What kind of alarm it is: conditionType names the AlarmDefinition condition that raised it (StateEquals, TagHigh, DeviceHealth, and the rest of the enumeration), and value carries whatever the condition matched on: a tag reading for a threshold condition, an NE 107 word for a device-health one. Both are absent when the alarm has neither: type is Equipment or System and says nothing about the condition, and an alarm raised by the state generator or the server sweep answers to no AlarmDefinition condition at all. A consumer keys on these two fields. Matching the name or the message is the derivation ADR-0016 forbids.
• Shelve contract (ADR 0014): a consumer that annunciates (horn, flashing, push notification, escalation, SCADA alarm raise) MUST treat shelved: true as "do not annunciate" for the duration. The alarm MUST still appear in any alarm-summary / list surface, marked shelved. Shelving suppresses annunciation alone. The retained event carries the current shelve state, so a reconnecting consumer sees it immediately.

It also publishes when an alarm is deleted, which is the one end of an alarm's life that has no state transition to carry it. An alarm is deleted when its AlarmDefinition is removed, or by an engineer cleaning up an orphan through the API:

• Payload: empty (zero bytes), which clears the retained message on the source's topic. A consumer MUST treat it as "forget what you are holding for this equipment". A malformed-event branch is the wrong place for it.
• Followed by: one ordinary alarm event per alarm that still exists on that source, republished least-consequential first, so the alarm a late subscriber most needs to see is the one left retained. A source whose last alarm was deleted is left with no retained message at all.

The clear is per source, because retention is: one topic carries the last event published for the equipment, whichever of its alarms that was. A tombstone naming one alarm is not something the topic can express, which is why the republish exists. It is what puts the surviving alarms back.

Device health for a maintenance system

A maintenance organization running a CMMS wants a work order when an instrument reports MaintenanceRequired. The alarm topic is the feed for that, and the convention is three things.

Author the policy. The platform mints no health alarm on its own. A deployment declares an AlarmDefinition with the DeviceHealth condition and lists the NE 107 states it wants to hear about in healthIn (see Annunciate instrument health for the form). Which states raise a work order is a site decision, and writing it as a definition keeps it a reviewable line of alarm policy. A setting inside the integration would hide it from that review.

Match on conditionType, read the state from value. A device-health alarm arrives on the same per-source topic as every other alarm for that equipment. conditionType: DeviceHealth is what separates it from a level or deviation alarm, and value holds the NE 107 word the instrument reported: Failure, CheckFunction, OffSpec, MaintenanceRequired or Unknown. Both fields are declared, which is the point: nothing about the alarm's name or message is part of this contract, and reading meaning out of either is what ADR-0016 forbids.

{
  "name": "alarm-def-ft-101-health",
  "source": { "kind": "ControlModule", "name": "ft-101" },
  "type": "Equipment",
  "severity": "Medium",
  "state": "ActiveUnacknowledged",
  "message": "FT-101 requests service",
  "conditionType": "DeviceHealth",
  "value": "MaintenanceRequired",
  "shelved": false,
  "timestamp": "2026-08-12T14:02:11Z"
}

An instrument that degrades further without recovering (MaintenanceRequired to Failure, with both words on the same healthIn list) publishes again with the new value. The alarm never cleared. This is not a fresh annunciation, and consumers that raise a horn should not raise it twice. The republish exists so a consumer holding the retained event is not left reading the word the alarm first fired on.

Poll REST for the ones that are fine. The alarm feed reports the instruments that crossed a line, and a maintenance system usually also wants the current health of every instrument, healthy ones included. GET /api/v1/sites/{site}/controlmodules serves deviceHealth and deviceHealthReason on each module that declares a health address, alongside the rest of the module record. Absence is not a state: a module that declares no spec.deviceHealth carries no value, and one whose poll failed reads Unknown with the reason.

One naming trap. dcs/{ns}/equipment/controlmodule/{cm}/health is a different signal entirely: the health of the control module's own function-block network (Running, Degraded, Down, CompileError), published by the unit runtime. It says whether our logic is executing. The instrument's own condition travels on the alarm topic above.

Physical Operator — Unit Controller

Publishes ISA-88 state transitions for Unit equipment:

• Topic: dcs/{ns}/equipment/Unit/{name}/state
• QoS: 1 (at-least-once)
• Payload: {"state": "Running", "previous": "Idle", "timestamp": "..."}
• Operator: physical-operator (requires MQTT_BROKER_URL)
• When it publishes: whenever the stored state changes, whatever moved it. A commanded transition and one the state machine settles on its own are the same event here. An abort completing is the second kind, and so is the Hold a Failover-mode unit takes when its control lease expires
• What previous means: the state this one replaced, which is the state that was stored. It is not always an ISA-88 edge. A single reconcile can cross several states and store only the last of them, so an abort settling on an idle unit publishes Aborting to Idle without an Aborted message between them. That is what the resource recorded, and the stream reports the resource

Procedural Operator — Procedure / UnitProcedure / Operation / Phase

Publishes ISA-88 state changes for all procedural elements:

• Topics: dcs/{ns}/equipment/{kind}/{name}/state where kind is Procedure, UnitProcedure, Operation, or Phase
• QoS: 1 (at-least-once)
• Payload: {"state": "Running", "previous": "Idle", "timestamp": "...", "batchID": "FERM-2026-0421", "unit": "fermenter-01"}
• Which batch it belongs to: batchID is the batch that owns this element. A procedural element is instantiated from a control recipe and labelled with its batch. Every other record on this topic tree has to be attributed through the equipment it came from. This one names its batch outright. The gateway's ad-hoc phase path creates an element outside any batch, and such an element carries no batchID at all. That is the correct answer for it
• Which unit it runs on: unit is the element's spec.unitRef. Only a Phase and a UnitProcedure declare one, and the key is absent on an Operation and on a Procedure, which sit above the unit and name none. The topic addresses the procedural element, so nothing else on the wire says which equipment is executing it. A subscriber needs that to place the element as the unit's active element
• When it publishes: whenever the stored state changes, whatever moved it. A commanded transition and one the element settles on its own are the same event here. Most of what these four kinds do is the second kind: a phase advancing through the steps of its action chart, a phase self-holding after an execution error, a phase settling Held because its unit runtime went away, the stuck-phase watchdog force-aborting, and an operation, unit procedure or procedure following its children to a terminal state. Until #1616 only the commanded transitions were published, so the as-run history of a batch was missing the transitions that say something went wrong
• What previous means: the state this one replaced, which is the state that was stored. The same rule as the Unit above, and for the same reason. It is not always an ISA-88 edge
• Operator: procedural-operator (requires MQTT_BROKER_URL)

Batch Operator — Batch Controller

Publishes batch lifecycle events used by the historian to tag records with batch IDs:

• Topic: dcs/{ns}/batch/{batchID}/state — the topic segment is the batch's spec.batchID. The Batch resource name never appears in it
• QoS: 1 (at-least-once)
• Retained: true — a subscriber that starts mid-batch learns the current binding on connect. This topic is the only thing on the wire that says which batch holds which equipment. The next transition can be twelve hours away on a fermentation, so a subscriber that waited for one stored and forwarded every record in between with no batch at all
• Payload: {"state": "running", "timestamp": "...", "units": ["fermenter-01"]} or {"state": "complete", "timestamp": "..."}
• States published: the lowercased ISA-88 state on every commanded transition (e.g. holding, aborting, restarting), plus lifecycle markers published when the controller settles a phase: running (on allocation or restart), held, complete, stopped, aborted
• Which equipment it holds: units is the batch's status.allocatedUnits, published on every transition. It is the binding a subscriber cannot obtain for itself, and it is what makes per-unit attribution possible. A batch that holds nothing omits the field entirely. A message with no units has made no claim about equipment, so a subscriber neither binds from it nor releases what an earlier transition bound. Two transitions legitimately carry none: a terminal one, because the controller has released the equipment by then, and a commanded one published before allocation has run
• Which message a reconnecting subscriber sees: whichever transition happened last, which is not necessarily running. A batch held overnight leaves held standing. Every non-terminal state carries the units and binds from them, because a held batch still owns its equipment
• Published on deletion: an empty payload (zero bytes), which clears the retained message when the Batch resource is deleted. A consumer MUST treat it as "this batch is gone, release whatever it held". Without it one retained message would accumulate per batch the plant has ever run, since the topic is addressed by batch id. It is best-effort: a broker that is down at the moment of deletion leaves a message the next batch of that id overwrites
• Operator: batch-operator (requires MQTT_BROKER_URL)

Which batch a record belongs to

The historian and the OMF egress both stamp a batch id on every record they store or forward. That id is what a batch report selects on. Getting it wrong corrupts the record a batch report is assembled from.

Neither attributes a record from a Kubernetes read. The historian holds no Kubernetes client at all. The OMF egress stamps the id at the moment the message arrives. Its queue rides out an endpoint outage, and a record projected hours later would otherwise carry whatever is allocated by then. Everything either one knows at that moment arrived on these topics. Each record is therefore attributed by a chain the publishers complete between them:

Record	How its batch is decided
Tag value	The unit is the topic's own {unit} segment
Unit state	The unit is the topic's own {name} segment
Procedural state	The payload's batchID, which names the batch exactly. Its unit names the equipment executing it
Alarm	The payload's unit, resolved by the alarm operator
Control-module health	The payload's unit, which is the publishing runtime's own
Batch lifecycle	Not a record — it carries units, the binding all of the above resolve through

A record whose unit cannot be named carries no batch. That is deliberate: an unattributed record reads as unattributed and stays recoverable from the batch record, while one carrying a batch that did not produce it reads as fact.

The binding is per unit. A namespace is a site and a site holds every unit in it. Two batches running at once on two units of one plant is therefore ordinary operation. Before #1549 both components held one batch slot per namespace, which meant every record from both units carried whichever batch started last. The first completion to arrive then cleared the slot for the batch still running. Neither failure logged anything, because from a namespace-keyed map's point of view nothing conflicted.

Subscribers

Historian

Subscribes to these wildcard topics to ingest all telemetry data:

dcs/+/runtime/+/value/#                  → tag values
dcs/+/equipment/+/+/alarm                → alarm events
dcs/+/equipment/+/+/state                → state transitions
dcs/+/equipment/controlmodule/+/health   → control module health
dcs/+/batch/+/state                      → batch context tracking

See Historian for details on ingestion and storage.

Security Profile

The Helm chart exposes mqtt.securityProfile to control broker security:

Profile	TLS	Auth	ACL	Use Case
production (default)	Port 8883, cert-manager TLS	Per-role credentials	Topic-scoped ACLs	Deployed systems
dev	Disabled (port 1883)	Anonymous allowed	None	Local development

When the production profile is active:

• Plaintext port 1883 is not exposed
• Anonymous connections are rejected
• Each MQTT client authenticates with a role-specific username/password
• Topic access is enforced by Mosquitto ACL rules
• Connection events and auth failures are logged for audit

Certificates are provisioned via cert-manager. When mtls.enabled is true, the MQTT certificate automatically uses the same CA issuer as inter-component mTLS. Alternatively, set mqtt.tls.certSecret to use a pre-existing Secret.

ACL Matrix

Role	Username	Read	Write
Gateway	dcs-gateway	dcs/#	dcs/#
Operators	dcs-operator	dcs/#	dcs/#
Runtimes	dcs-runtime	dcs/+/equipment/+/+/command	dcs/+/runtime/#, dcs/+/unit/+/telemetry
Historian	dcs-historian	dcs/#	(none)
OMF egress	dcs-omf-egress	dcs/#	(none)

Each role also has a second account name, its base name with a -b suffix, carrying identical rules. That is the credential slot a password rotation moves through, and it exists only while a rotation is running. See ADR 0061 and the rotation runbook.

Helm Configuration

mqtt:
  securityProfile: production  # or "dev"
  tls:
    enabled: true
    certManager:
      enabled: true
      issuerRef:
        name: ""     # empty = auto-use mTLS CA issuer
        kind: Issuer
  auth:
    enabled: true
    users:
      gateway:
        password: ""   # auto-generated if empty
      operator:
        password: ""
      runtime:
        password: ""
      historian:
        password: ""

Broker Requirements

Any MQTT v5 compatible broker works. The Helm chart ships Mosquitto 2.x by default.

• Mosquitto: Lightweight, suitable for single-site deployments
• HiveMQ: Enterprise-grade, supports clustering
• EMQX: High-performance, built-in dashboard

The broker should be deployed in the same Kubernetes cluster as the DCS components. The Helm chart supports configuring the broker URL for all components.

Graceful Degradation

All MQTT publishing is optional. If the MQTT client is not configured or the broker is unreachable:

• Controllers operate normally without publishing events
• The historian will not receive data until the broker is available
• Alarm state transitions are still recorded in the Kubernetes API
• The HMI shows a "No MQTT" connection indicator. It recovers live when the gateway's background reconnect succeeds

Broker unavailability never fails pod readiness. Components stay up, report dcs_mqtt_connected = 0, and resume publishing automatically once the broker is reachable.

Related Documentation

• Alarm Management -- Alarm MQTT events
• Historian -- MQTT ingestion and time-series storage
• Architecture -- System overview and runtime design
• Security Operations -- TLS configuration