Architecture

Cloud-Native DCS is a Distributed Control System that runs on Kubernetes. Every domain object (equipment hierarchy, recipes, batches, control logic, alarms, audit records) is a Kubernetes Custom Resource, reconciled by operators, and exposed through a REST API and browser-based Web UI. The Web UI is split into three role-aligned endpoints, /system (engineering), /hmi (operations), and /data (production information), documented in Terminology.

BPCS-layer protection — an independent SIS is a PHA/LOPA decision

Cloud-Native DCS is a Basic Process Control System (BPCS) in IEC 61511 terms. It is not a Safety Instrumented System (SIS) and is not rated for SIL 1, SIL 2, SIL 3, or SIL 4 functions under IEC 61508 / IEC 61511. Its device interlocks, phase guards, and alarms are BPCS-layer protection functions. Under IEC 61511 Clause 9.2 a BPCS interlock or alarm can be credited as an independent protection layer. Whether a given hazard (emergency shutdown, overpressure trip, LEL/combustible-gas trip, relief-valve integrity, burner management, or any function whose failure causes injury or loss of containment) also requires an independent SIS is the outcome of your process hazard analysis / LOPA, decided per process and per function. An independent SIS here means a separate, independently certified safety PLC or hardwired trip system. Assign protection layers at design time accordingly. The product does not make that call on your behalf (ADR 0011).

Within the BPCS scope, protection is layered per ADR 0007: device interlocks in the FB scan on the edge (always-on, partition-tolerant), phase guards for procedural safe-state sequencing (made partition-tolerant at the edge by ADR 0008), alarms for annunciation and audit. Reaction-time bounds are documented per alarm path in Alarm Management → Reaction time and the underlying decision is recorded in ADR-0002.

The pieces fit together as the CRD model, the operators that reconcile it, the state machine that governs batch execution, and the per-unit runtime that bridges Kubernetes to physical I/O. Standards alignment (ISA-88, IEC 61131-3, 21 CFR Part 11, IEC 62443) is summarized at the end of this document and detailed in the compliance traceability matrices.

What the System Does

At a high level, Cloud-Native DCS lets you:

• Model a plant as a tree of Enterprise → Site → Area → ProcessCell → Unit → ControlModule resources (ISA-88 physical model).
• Author recipes as MasterRecipe templates that the batch-operator clones into per-run ControlRecipe instances.
• Execute batches against those recipes: the batch-operator allocates units, builds a procedural tree (Procedure → UnitProcedure → Operation → Phase), and drives each element through an ISA-88 state machine.
• Run control logic on edge devices via a per-unit runtime pod that executes IEC 61131-3 function block programs against real or simulated I/O (Modbus TCP, OPC UA, and experimental EtherNet/IP).
• Manage alarms with an ISA-18.2 four-state lifecycle.
• Record everything with audit trails, historian telemetry, and auto-generated batch production records.

All of this is driven from a single Web UI, a REST API, the dcs CLI, or directly as Kubernetes resources, whichever fits the workflow.

The clip below is that claim performed on a single object. One control module, fermenter-1's agitator, is reached through the same equipment hierarchy in all three endpoints. It is read from a terminal through the same REST API the pages render from, and written once from the operator's HMI faceplate. Every surface then agrees on the new value: the engineering FBD streams it, the historian trend carries the step, and the audit trail carries the write that caused it.

One control module through every surface: the same hierarchy in /system, /hmi and /data, a dcs CLI read, an HMI faceplate write, and the FBD, trend and audit record all agreeing.

ISA-88 Overview

ISA-88 (IEC 61512) is the international standard for batch process control. It defines a hierarchy of models that separate what to make (recipes) from how to make it (equipment and procedures). Cloud-Native DCS maps each ISA-88 concept to a Kubernetes Custom Resource Definition (CRD).

ISA-88 / standard concept	Kubernetes CRD (API group)
Physical Model	physical.dcs.io/*
Enterprise	Enterprise (cluster-scoped)
Site	Site (cluster-scoped)
Area	Area
Process Cell	ProcessCell
Unit	Unit
Control Module	ControlModule
Controller	Controller
Procedural Model	procedural.dcs.io/*
Procedure	Procedure
Unit Procedure	UnitProcedure
Operation	Operation
Phase	Phase
Synchronization Barrier (Part 1 Clause 7.2)	SyncBarrier
Recipe Model	recipe.dcs.io/*
Master Recipe	MasterRecipe
Control Recipe	ControlRecipe
Batch Model	batch.dcs.io/*
Batch	Batch
Control Logic (IEC 61131-3)	control.dcs.io/*
Control Program	ControlProgram
Function Block Type	FunctionBlockType
Alarm Management (ISA-18.2)	alarm.dcs.io/*
Alarm	Alarm
Audit Trail (21 CFR Part 11)	audit.dcs.io/*
Audit Record	AuditRecord
Change Control (21 CFR Part 11 §11.10(k))	changecontrol.dcs.io/*
Change Control Policy	ChangeControlPolicy (cluster-scoped)
Change Request	ChangeRequest

API Groups

The tables below cover the CRDs each API group offers as a documented feature. For each one they give the operator that reconciles it and what it represents. A few kinds are deliberately absent. Their CRDs and controllers still ship, but their user-facing surface was withdrawn under #480. They are not something to build against. The README's API-group table is the complete inventory of what config/crd/bases/ installs.

physical.dcs.io/v1alpha1

Models the plant's physical equipment hierarchy.

CRD	Scope	Description
Enterprise	Cluster	Top-level organization (e.g., a company)
Site	Cluster	Physical location (creates a site-{name} namespace)
Area	Namespaced	Logical grouping within a site
ProcessCell	Namespaced	Collection of units that work together
Unit	Namespaced	Equipment that executes phases (reactor, mixer, filter-dryer)
ControlModule	Namespaced	Lowest-level I/O point (sensor, valve, actuator, PID loop). A smart instrument's own asset identity rides on spec.fieldDevice — the IEC 61987 nameplate, serial number included (ADR 0043)
ControlModuleTemplate	Cluster	Reusable CM template — compiled by the control-operator into a ControlProgram when referenced by a ControlModule
Controller	Namespaced	Physical computing device, bound to the cluster node of the same name. Set spec.simulation: true to create a virtual node with no real one to wait for
IOModule	Namespaced	The record of one addressable field device: the endpoint the runtime exchanges process data with, plus spec.fieldDevice (declared type of io-module, control-device, instrument or gateway, with vendor/model/serialNumber/firmware) saying what the box behind it is. Covers a fieldbus coupler (e.g., Wago 750, Turck TBEN), a third-party PLC we do not run, and a simulation stand-in. Simulation IOModules carry inline spec.simulation config (behaviors, faults, RNG seed, optional preset reference + address map)
SimulationPreset	Namespaced	Reusable equipment simulation template with generic address names. Referenced by an IOModule's inline spec.simulation.preset; expanded at reconcile time
NodeIdentity	Cluster	Product-authored attributes (displayName) of a machine that has no Controller to author through — a control-plane node or an unclassified worker. One per node, named for the node (ADR 0040)
NodeMaintenance	Cluster	An operator's intent to take a node out of service (cordon, drain, reboot, shutdown) or bring one back (power-on), with a lifecycle the physical-operator advances (ADR 0034)
SiteOutage	Cluster	A planned shutdown of a whole site — the plant safed and quiesced, then either powered down through a channel that survives the cluster (unattended) or left running for a manual cut at the rack (attended). See Planned Outage

Hierarchy:

flowchart TD
    Enterprise --> Site[Site<br/>creates namespace site-name]
    Site --> Area
    Area --> PC[ProcessCell]
    PC --> Unit[Unit<br/>runs runtime pod]
    Unit --> CM[ControlModule]

procedural.dcs.io/v1alpha1

Models the sequence of actions to execute a batch, plus the reusable templates authors work with.

CRD	Scope	Description
Procedure	Namespaced	Top of the procedural tree. Sequences one or more UnitProcedures via an SFC chart (spec.chart)
UnitProcedure	Namespaced	Sequences Operations on a single Unit via an SFC chart
Operation	Namespaced	Sequences Phases within a UnitProcedure via an SFC chart
Phase	Namespaced	Atomic unit of work — an SFC chart whose ST actions read/write tags and call PROMPT() for operator interaction
ProcedureTemplate	Namespaced	Reusable template cloned by the batch-operator into a Procedure at batch instantiation
UnitProcedureTemplate	Namespaced	Reusable template that the batch-operator clones into a UnitProcedure at batch instantiation. spec.category (equipment / process) gates whether the template is eligible for ad-hoc execution outside a Batch — see Ad-Hoc Execution
OperationTemplate	Namespaced	Reusable template cloned into an Operation. Same spec.category gating as UnitProcedureTemplate
PhaseTemplate	Namespaced	Reusable template cloned into a Phase. Declares its CM-role requirements via spec.cmRoles — validated against the target Unit at batch instantiation
SyncBarrier	Namespaced	Coordinates multiple phases running across different units — they must all reach the barrier before any can proceed (ISA-88 Part 1 Clause 7.2)

Hierarchy:

flowchart TD
    Procedure --> UP[UnitProcedure]
    UP --> Op[Operation]
    Op --> Phase[Phase<br/>executes logic on unit runtime]

recipe.dcs.io/v1alpha1

Ready-to-execute recipe templates and the immutable per-batch copies cloned from them.

CRD	Scope	Description
MasterRecipe	Namespaced	Ready-to-execute recipe targeting specific equipment
ControlRecipe	Namespaced	Immutable runtime copy cloned from a MasterRecipe by the batch-operator when a Batch is created
RecipeRevision	Namespaced	Immutable snapshot of a MasterRecipe created on every approval. Provides version history and rollback per 21 CFR Part 11 § 11.10(k)

See Recipes for the authoring and approval lifecycle.

Recipe flow:

flowchart TD
    MR[MasterRecipe<br/>targets specific units] --> CR[ControlRecipe<br/>created per batch, immutable]

batch.dcs.io/v1alpha1

CRD	Scope	Description
Batch	Namespaced	A single execution of a MasterRecipe
BatchRecord	Namespaced	Auto-created consolidated production record when a Batch reaches a terminal phase (Complete, Failed, Aborted). Reviewed and finalized with an electronic signature per 21 CFR Part 11 § 11.50/11.70

Batch execution flow:

1. A user creates a Batch referencing a MasterRecipe (via the HMI, CLI, or API).
2. The batch-operator's unit allocator reserves the Units required by the recipe's equipment requirements.
3. The recipe instantiator clones the MasterRecipe's templates into a full procedural tree (Procedure → UnitProcedure → Operation → Phase), all within the Batch's namespace.
4. The Batch then waits in Pending until an operator issues the Start command (the Start button in the HMI, dcs command Batch <name> Start, or a dcs.io/command annotation). That command is the only thing that starts a batch. spec.scheduledStartTime is the planned start and does not run anything, so a batch whose planned time has passed still waits here (ADR 0052). Two fields describe this pre-Start moment: the Batch's status.phase is Pending while its procedural tree's ISA-88 status.state is Idle. The HMI shows the ISA-88 state, so "Idle" there and "Pending" here name the same moment. Start cascades down the tree through the ISA-88 state machine. Every state transition emits an AuditRecord.
5. When the Batch reaches a terminal phase, the BatchRecord reconciler creates a pending-review BatchRecord aggregating the batch's state history, phase history, and operator actions. The /data endpoint's Batch Records view lists these for quality-review sign-off.

control.dcs.io/v1alpha1

Models IEC 61131-3 control programs for cyclic-scan function block execution.

CRD	Scope	Description
ControlProgram	Namespaced	IEC 61131-3 control program — FBD (blocks + connections) or ST (structured text), with variable declarations for tag binding
FunctionBlockType	Cluster	Composite function block type (IEC 61131-3) — a reusable sub-network of blocks with declared input/output ports, referenced by name from ControlProgram spec.blocks. The unit runtime flattens one when it loads the program (ADR 0065): the inner blocks are spliced into the scan at the instance's position under a /-joined path name such as loop/ctrl, so a composite costs nothing at scan time and every name-addressed surface reaches an inner block by its path. A composite declares its parameters in spec.parameters, and an inner block reads one as {{.params.<name>}}; the runtime substitutes the instance's value while it flattens, so one type serves any number of devices (#1660)
CoordinationBlock	Namespaced	ISA-88 coordination control (Part 1 Clause 7.2) — signal publish/subscribe between units, where a producer raises a named signal and its consumers wait for it. Reconciled by the batch-operator

ControlModules reference ControlModuleTemplates, which the control-operator compiles into ControlPrograms and deploys via HTTP to the corresponding unit runtime. Updates use hot-swap (see below). (ProcessException was purged in #863. The live exception path is AlarmDefinition.spec.exceptionAction. CoordinationBlock's Lock and ExceptionPropagation modes were purged in #862. Signal is the one shipped mode.)

alarm.dcs.io/v1alpha1

ISA-18.2 compliant alarm management with a four-state lifecycle.

CRD	Scope	Description
AlarmDefinition	Namespaced	Declarative condition (state-equals or tag threshold high/low/deviation) attached to a ControlModule, Unit, or IOModule. May carry an exceptionAction (Hold/Stop/Abort) that fires automatically when the condition is met
Alarm	Namespaced	Alarm instance with ISA-18.2 state model (Active/Cleared × Unacknowledged/Acknowledged). Created by the alarm generator when an AlarmDefinition's condition fires, or from IOModule / Unit status changes

Four reconcilers cover this group, all hosted in the physical-operator binary: the Alarm reconciler (watches for dcs.io/command=Acknowledge annotations), the AlarmGenerator reconciler (auto-creates Alarms from equipment status), the AlarmDefinition reconciler (evaluates conditions against live tag values), and the ServerAlarm reconciler (annunciates node readiness, node resource pressure, and lost etcd quorum as ISA-18.2 alarms, fanned out one Alarm per site namespace, per #1170 and ADR 0032). All state transitions are published to MQTT. See Alarm Management for details.

audit.dcs.io/v1alpha1

21 CFR Part 11 compliant audit trail records.

CRD	Scope	Description
AuditRecord	Namespaced	Immutable record of a user action, with electronic signature support

Audit records capture who, what, when, and why for all significant actions in the control system. See Security Operations for details.

changecontrol.dcs.io/v1alpha1

21 CFR Part 11 §11.10(k) change control for mutations to change-controlled resources (ControlProgram, ControlModule) in production namespaces.

CRD	Scope	Description
ChangeControlPolicy	Cluster	Allowlist of ServiceAccounts that may mutate change-controlled resources in namespaces labelled dcs.io/production=true, plus the external-reference and e-signature annotation requirements each mutation must carry
ChangeRequest	Namespaced	Proposed mutation to a change-controlled resource, routed through one of three backends (directApply, inClusterQueue, gitPR) for review and apply. Carries the proposed JSON patch, e-signature, external reference, and reviewer attribution. Reconciled by the changecontrol controller hosted in the batch-operator binary

Two enforcement surfaces work together:

• Admission webhook (hosted in batch-operator) — gates direct writes to change-controlled resources against the active ChangeControlPolicy, rejecting mutations that lack required annotations or originate from a non-allowlisted ServiceAccount.
• ValidatingAdmissionPolicy primitives (issue #288) — the chart ships ValidatingAdmissionPolicy + ValidatingAdmissionPolicyBinding resources that enforce GitOps ownership labels (dcs.io/owned-by=flux, dcs.io/owned-by=engineering) on engineering-class CRs. CEL expressions reject writes from the wrong ownership class without webhook round-trips. See GitOps Enforcement.

The ChangeRequest reconciler dispatches to one of three backends based on the policy's spec.backend:

Backend	Behavior
directApply	Applies the proposed patch directly once approval annotations are present. Used for low-risk environments and break-glass changes.
inClusterQueue	Holds the request in a queue for asynchronous review; an approver acks via UI or dcs changerequest approve.
gitPR	Opens a pull request against a configured GitOps repo; the change lands when the PR is merged and Flux reconciles. Closes the loop with the audit-flux-bridge for SHA correlation.

See Change Control and ChangeRequest Backends for the full workflow.

Operators

Operator binaries reconcile the CRDs, plus two kinds of pods that run on controller devices: a per-Unit runtime pod and a per-Controller io-probe pod. Each binary hosts multiple reconcilers. The grouping below is by binary, which cuts across the API groups.

physical-operator

• Reconciles: Enterprise, Site, Area, ProcessCell, Unit, ControlModule, Controller, IOModule, NodeIdentity, NodeMaintenance, SiteOutage, plus the alarm group (Alarm, AlarmGenerator, AlarmDefinition, ServerAlarm)
• Smart vendor-packaged devices (PLCs, MTP skids) are modeled as Units with spec.serviceBinding. A service is either a raw OPC UA Method or a declared MTP module service carrying its ServiceControl addresses (ADR-0045, with the VDI 2658-4 state/command vocabulary in pkg/mtpservice). Phase ST/SFC bodies dispatch Method services through the CALL_SERVICE builtin (internal/controller/procedural/call_service.go) and drive MTP module services through MTP_COMMAND / MTP_STATE / MTP_COMMAND_ENABLED (internal/controller/procedural/mtp_command.go), which hold one OPC UA session across the phase run. The physical operator does not run a per-device OPC UA session. See ADR-0001 for the rationale and the Asset dissolution migration guide for how the previous Asset CRD mapped to this shape.
• Creates site-{name} namespaces when a Site resource is created
• Replicates Secrets labelled dcs.io/replicate-to-sites=true from the release namespace into each site-{name} namespace and patches the namespace's default ServiceAccount with replicated dockerconfigjson Secrets as imagePullSecrets. Operator-created pods (io-probe, unit-runtime) can then pull private images and mount MQTT TLS / auth Secrets without per-site copying. Configurable via the siteSecretReplication values block (issue #306).
• Manages the unit runtime pod lifecycle — creates/deletes runtime pods for each Unit, copying nodeSelector to pin pods to device nodes. A runtime pod is recreated on termination, on spec or availability drift, on an mTLS Secret rotation, and on a superseded container image. The image case is the only one that is conditional: the operator recreates the pod when the unit is free, and where a batch is actively driving I/O it reports status.conditions[RuntimeImageDrift] and dcs_unit_runtime_image_drift and changes nothing, because taking the runtime away mid-step can leave actuators in indeterminate states. Without that report a chart upgrade would leave the edge binary on the previous release with every surface reading healthy (#1762). The gateway serves that verdict as runtimeBuild on the Unit. It reaches dcs get units, the Unit detail page and both tabs of the Diagnose panel. Nobody has to reach for kubectl to read it (#1771). The verdict has four values where the condition has three statuses. True covers both a roll under way and a roll deliberately withheld under a running batch, and those are different answers for whoever reads them
• Manages the io-probe pod lifecycle — creates a per-Controller io-probe pod when IOModules reference it, providing infrastructure-level health monitoring independent of the equipment hierarchy. A probe pod is recreated when it terminates. It is never recreated for a superseded image, and the operator reports that state instead: every IOModule the probe serves carries status.conditions[IOProbeImageDrift] naming the pod, and dcs_ioprobe_image_drift counts the probes. The gateway serves that verdict as probe on the IOModule. It reaches dcs io list, dcs get iomodules and the IOModule detail page. Nobody has to reach for kubectl to read it (#1768). Nothing rolls a probe on its own because its read cadence is what feeds every device comm-loss watchdog a module declares through spec.failSafe, and a recreate that includes an image pull on a device node can outlast a declared timeout. The remedy is the audited POST /api/v1/sites/{site}/iomodules/{name}/ioprobe/restart, which refuses until the caller acknowledges that exposure (#1765)
• Projects every OPC UA credentials Secret an IOModule names through spec.security into the unit-runtime pod and the io-probe pod, one directory per Secret under /etc/dcs/opcua-credentials/, and the driver reads it there (ADR 0087, #1912). The iomodules.json both pods read carries the Secret's name and never its contents. The runtime pod is recreated when the set of Secrets its modules name changes. The probe is reported instead, under the same #1762 rule as image drift, through status.conditions[CredentialsSecured]
• Reconciles Controller resources (Pending → Joining → Joined): binds each Controller to the cluster node of the same name, converges spec.nodeLabels (dcs.io/ prefix) onto the bound node, and auto-creates Controllers for nodes labeled dcs.io/site=<site> (node discovery). Node join is the deployment layer's job (ADR 0004). A Controller declared ahead of the machine it names waits in Pending, for as long as that takes (#1838) Simulation controllers (spec.simulation: true) create and maintain a virtual Kubernetes Node and heartbeat Lease, with no physical node to wait for
• Processes RequestedState in Unit status (set by the procedural layer) and dcs.io/command annotations to trigger state transitions on Units
• Uses the ActiveWork status field on Units for orphan detection — a Running unit with no ActiveWork is auto-stopped back to Idle

procedural-operator

• Reconciles: Procedure, UnitProcedure, Operation, Phase, SyncBarrier, plus the recipe group (MasterRecipe, ControlRecipe)
• Cascades Start / Stop / Hold / Restart commands down the procedural tree via RequestedState generations and dcs.io/command annotations
• Phase controller runs the SFC engine, evaluates ST transition conditions, and publishes live step/variable snapshots to Phase status
• Phase controller is resilient to runtime outages: the first time the unit runtime is unreachable in a run, Status.RuntimeUnreachableSince is stamped and the reconciler requeues with backoff (transient blips like pod restarts or cert refreshes resolve without disturbing the phase). Outages exceeding a grace window cause a clean self-hold with audit and alarm. A self-hold records exactly one AuditRecord on the transition into the held condition: while a phase keeps failing identically and bounces Running ↔ Held every reconcile, the persisted Status.SelfHoldSignature suppresses duplicate records (a changed error is audited afresh), so an idle-ish failing batch no longer grows etcd without bound. As a safety net, a stuck-phase watchdog reads Status.LastProgressAt, a heartbeat with four writers: the runtime readiness check before a chart starts, every tick of the SFC status publisher while a chart runs, the control-gap ruling, and a transition the pass itself made out of the state it read (an operator command, or a resume off a satisfied sync barrier or a raised coordination signal, #2122). The watchdog's own force-abort is not progress and does not stamp it, because the wedged-abort escalation is reached through the same threshold check. A phase that stays in an active state with no progress for the watchdog threshold is force-aborted, converting silent stalls into loud, audited failures so the parent batch is never blocked indefinitely. A running ActionChart blocks its reconcile for the whole chart (a timed charge step legitimately dwells for tens of minutes), so during execution the SFC status publisher's one-second tick is what keeps the stamp alive. LastProgressAt is therefore a high-water mark: the reconciler's own copy is a snapshot from before the chart started, and writing it back would report a healthy dwell as a stall exactly as long as the dwell itself, so no writer may move the stamp backwards. The watchdog stands down where the stall is a control gap (#2121). Its heartbeat and the last-scan stamp the gap detector reads are written by the same publisher tick, so a rack that lost power for twenty minutes reads here exactly like an engine that wedged for twenty. The detector can tell the two apart, and the ruling it takes on the resume is the one an operator can act on. A Running phase resuming a persisted position across a gap is therefore ruled on first. The ruling raises the heartbeat, so the pass after it does not abort on an interval already answered. With no position to rule from, the watchdog fires as before.
• A publisher tick proves the engine is alive. The chart advancing is a separate claim, and the two fail differently. Because reconciles for one Phase are serialised, a chart that wedges blocks its own reconcile, so the stuck-phase watchdog is never even evaluated while the wedge lasts. The publisher is the only thing still running. It therefore also watches the engine's ProgressCount, which counts scan cycles that evaluated a transition condition or changed a step. A timed dwell advances it every scan (the guard is re-evaluated continuously) while a chart whose step is blocked in an action that never returns stops it dead. The other wedge shape, a guard erroring every scan, is ended by the engine's own escalation after two minutes and so never reaches this five-minute grace. That is the point of the ordering, because that failure carries a cause worth reporting and this one does not. Past the grace the phase records Status.EngineStalledSince, raises a dedicated engine-stall alarm, and audits the diagnosis once, non-destructively, because "the engine stopped advancing" is a diagnosis and not a verdict on the batch. A chart that then runs out its timeoutSeconds names the stall in its report, and the report is more than a bare deadline. Sanctioned indefinite waits (an outstanding operator prompt, a held engine) count as advance. Sync-barrier and coordination waits park the phase in Held, where no chart is running at all.
• A phase whose runtime dies mid-Holding cannot run its HoldingChart at all, so it settles Held with its declared posture uncommanded (#1387) and records that in Status.HoldingActionDeferredSince with its own hold-deferred alarm. The reconnect poll then commands the deferred holding action and re-arms the phase chart at the edge, driving the run against a state machine positioned in Holding so the interrupted transition is what completes. ISA-88 makes Held a waiting state that directs no actions (ADR 0048). Recovery still takes an explicit Restart.
• A Hold the physical layer raises on the Unit reaches the phase running on it. The Unit carries status.conditions[HeldByRuntimeFault] for every Hold the physical layer takes on its own initiative, whether the runtime was lost, the control lease expired, or the driver watchdog found the field I/O gone under a running batch. The condition's reason says which, and a lost runtime carries one of two: PodNotReady for a pod that did not recover on a node that is still Ready, and NodeNotReady for a node that stopped reporting, read at the moment the Hold is taken. The pod on a partitioned node never crashed, and an operator sent to look at it finds nothing wrong (#2153). The phase reflects Held from that condition without re-running its HoldingChart, since the equipment is already where the chart would put it. The Operation, UnitProcedure, Procedure and Batch then follow through the ordinary child-resting-state propagation. The whole tree holds, and recovery takes an explicit Restart. The same condition bars the drift repair in convergeUnitState, which reads a Held unit under a Running procedure as a lost command and would otherwise restart it (#1723). The phase records which marker it answered in status.reflectedUnitHoldAt, so the Restart out of that Held is not re-read as a fresh fault by a pass whose view of the Unit predates the Restart. The UnitProcedure records when it left Held in status.leftHeldAt. A marker stamped after that is a new fault and stays where it is. One stamped before it is a Restart the Unit never received, and the repair sends it again (#2155).
• MasterRecipe controller writes an immutable RecipeRevision on every generation bump

batch-operator

• Reconciles: Batch, BatchRecord, plus the changecontrol group (ChangeRequest) and the control group's CoordinationBlock
• On Batch creation: allocates required Units, clones the referenced MasterRecipe's templates into a full procedural tree, and parks the Batch in Pending until an operator issues Start (via a dcs.io/command annotation)
• On Batch terminal phase: creates a pending-review BatchRecord aggregating the batch's state history, operator actions, and audit trail
• Change-control admission webhook: gates writes to change-controlled resources in production namespaces against the active ChangeControlPolicy
• ChangeRequest reconciler: dispatches proposed mutations to one of three pluggable backends: directApply, inClusterQueue, or gitPR (issue #289). The gitPR backend opens a PR against a GitOps repo. Combined with the audit-flux-bridge (below), the resulting Flux reconciliation creates an AuditRecord with the merge commit SHA attached

control-operator

• Reconciles: ControlProgram, ControlModule
• ControlProgram deployment: watches ControlProgram CRs, resolves spec.unitRef → Unit → runtime pod IP, and deploys the program definition to the runtime via HTTP. Supports initial deploy and hot-swap replace (generation-based detection).
• ControlModule template compilation: compiles ControlModuleTemplate + parameter overrides into ControlProgram CRs for cyclic scan control logic. Template-defined tags (e.g., PV, CV) are merged with instance tags into status.effectiveTags. Instances therefore inherit standard tags automatically. Tags may carry per-tag engMin/engMax engineering ranges (literal values or block parameter references like "read_pv.engMin") and per-tag trueLabel/falseLabel for Boolean tags (overriding CM-level parameters["true_label"]/parameters["false_label"]). The compiler resolves a block parameter reference to a literal, which is what lets the unit controller carry the range into the runtime's tag map and the runtime refuse a write outside it (ADR 0050). The compiler also checks I/O directions: an input block (DI, AI) must not land on an address the template declares as an output, and an output block (DO, AO) must not land on one it declares as an input. The one exemption is a read-back. An output port carrying readBack: true says its device point can also be read. That licenses an input block whose address is written as {{.outputs.<name>}} and nothing else, so one point keeps one binding. An instance that aliases an independent input port onto the driven address is still refused (ADR 0066).
• Sets Deployed condition and tracks execution state in ControlProgram status.

Hot-swap policy

Hot-swap is the spec-replace path taken when the runtime is running a program whose content no longer matches what the control-operator computes for it. Two changes reach it. The ControlProgram's own generation can advance, which is the direct edit. Or the content resolved into the program can drift: a composite FunctionBlockType it uses is edited or deleted, which moves the computed spec hash while the ControlProgram itself is untouched (#1574). Both rewire function-block state on a running runtime, so both are disruptive to any batch executing against that unit. The control-operator gates hot-swap according to batch state, and the cause makes no difference to the gate:

• Blocked: if any Batch in the ControlProgram's namespace is in phase Running, Holding, Held, or Aborting and has the program's unit in its status.allocatedUnits, the replace is deferred. The controller sets a HotSwapDeferred status condition (reason BatchActive) naming the blocking batch and which of the two causes made the logic stale, emits a Rejected AuditRecord correlated to the batch, and requeues every 30 seconds. A watch on Batches re-enqueues the ControlProgram the moment the blocking batch leaves its active phase.
• Allowed via annotation bypass: an operator can override the gate by annotating the ControlProgram with both dcs.io/allow-hot-swap=true and a non-empty dcs.io/hot-swap-reason. The trimmed reason is written into the resulting Update AuditRecord so the override is permanently attributable for 21 CFR Part 11 purposes. Neither annotation alone satisfies the bypass. The bypass releases a drift-driven swap exactly as it releases a generation-driven one.
• Pod-recovery is never gated: the deploy path (deployToRuntime) fires when the runtime does not hold the program at all, which is what the verify probe reports as NotFound, as an empty state, or as an unreachable runtime. It runs unconditionally, because it is the safety net that restores control logic to a freshly restarted runtime pod and the alternative is a unit left with no logic. Audit records from this path carry reason=pod-recovery when the program was previously deployed, reason=initial-deploy otherwise.

A successful replace records reason=generation change or reason=content drift, alongside the before and after spec hashes. It is correlated to the batch whose logic it changed. That is the blocking batch's spec.batchID on the bypass path, and it is empty on the ordinary path, where no batch held the unit. The correlation is what puts a bypassed mid-batch change into the lot's BatchRecord change history, alongside the operator's justification. A replace correlated to nothing is found by its target ref, which every record carries. The Runtime.OverrideSnapshot / ApplyOverrides machinery in pkg/fbruntime/runtime.go carries block state across a replace, so a swap does not reset the live values a batch has already written.

Minimal annotated override example:

UIYAML

/system → ControlProgram detail → Hot-Swap Override opens a modal that writes both annotations and an AuditRecord under the operator's identity. The reason is required and written verbatim to the audit trail under 21 CFR Part 11 §11.10(k).

apiVersion: control.dcs.io/v1alpha1
kind: ControlProgram
metadata:
  name: reactor-pid
  namespace: site-demo
  annotations:
    # Both annotations are required. allow-hot-swap on its own is ignored.
    dcs.io/allow-hot-swap: "true"
    dcs.io/hot-swap-reason: "CAPA-2026-04-12: PID Ki reduced to damp oscillation observed during batch-7421"
spec:
  unitRef: reactor-a
  scanInterval: 100ms
  # ...

Clearing the HotSwapDeferred condition manually is not required. The controller removes it on the next successful reconcile once the blocking batch clears or a valid bypass is in place.

Auxiliary Binaries

The operators above reconcile CRDs. The system also ships auxiliary binaries that are not reconcilers but are part of the running deployment.

• historian (cmd/historian/) — MQTT-to-PostgreSQL collector. Subscribes to state transitions, alarm events, and control-logic tag snapshots published by unit runtimes and operators, and writes them to a PostgreSQL database for long-term trending and compliance retention. See Historian and MQTT Telemetry.
• audit-archiver (cmd/audit-archiver/) — Kubernetes CronJob. Moves AuditRecord CRs older than a configurable retention window from etcd into the historian PostgreSQL database, satisfying 21 CFR Part 11 §11.10(c) long-term retention while keeping etcd bounded. Archival is off by default (audit.archival.enabled). When it is switched on, the CronJob runs hourly, at :17 to dodge the top-of-hour stampede (#313).
• audit-flux-bridge (cmd/audit-flux-bridge/) — Flux notification receiver. Subscribes to flux.toolkit.fluxcd.io/v1beta3 Provider + Alert events and emits an AuditRecord for each Flux reconciliation of an engineering CR, attaching the source Git commit SHA for full 21 CFR Part 11 §11.10(k) traceability of GitOps-driven changes. HMAC-signed webhook authentication via a shared secret in both dcs-system and flux-system namespaces. See Audit Linkage.
• io-probe (cmd/io-probe/) — Probes each referenced IOModule's fieldbus link (Modbus TCP, EtherNet/IP, OPC UA) and publishes infrastructure-level connectivity health independent of the equipment hierarchy. Two flavors: a per-Controller pod (node-pinned, owned by the Controller) for IOModules that name a controllerRef, and a namespace-shared network-io-probe pod (node-unpinned by default, no owner) for controller-less network IOModules (ADR 0021). On zoned networks a module's spec.probePlacement.nodeSelector pins its shared probe to the nodes with field reach, one pod per distinct selector (ADR 0042). Whichever flavor serves a module, the pod's name is published on that module's status.probePodName. A reader routing a request to the probe takes the published name and never re-derives it (#1480). The physical-operator manages both lifecycles: the ControllerReconciler the former, the IOModuleReconciler the latter. Gateway proxies to io-probe over TLS when configured (#307). Each cadence is an exchange with the device rather than a read of the driver's connected flag: a module that answered anything has proved its link, and one that answered nothing is asked directly so the answer decides the verdict. Whether it answered comes from a count the driver keeps of the exchanges its endpoint has replied to, read before the pass touches the module. Deciding it from the fail-safe readback instead was wrong twice. That readback reports whether a fail-safe was read, which is a narrower question than whether anything was: no readback runs for an EtherNet/IP module at all (#1759), and a Modbus device no profile claims answers the profile lookup's register read and then fails the readback anyway (#1763). Each of those was asked on every cadence, forever. A flag only a failed I/O operation clears cannot report a cable that was pulled, so a probe that performed no exchange left a dead module reading Online indefinitely (#1755). What the ask is belongs to the protocol. An EtherNet/IP module is asked over the session already open, because CIP obliges every device to answer a read of its Identity object, and the re-dial is then the remedy for a session that did not answer (#1759). Modbus reserves no such register, so a fresh dial is the only exchange it has.
• dcs CLI (cmd/dcs/) — Operator-facing CLI. Covers batch, recipe, equipment, audit verification, historian queries, tag simulation, diagnostics, change-control approval workflows, and a laptop-mode embedded gateway (dcs ui, #286). Preferred over kubectl for day-to-day operator tasks (see CLI Reference): it is gateway-mediated, so it carries a persona, a Site scope, and an audit trail that raw kubectl does not. Which humans hold Kubernetes credentials at all, and why only the platform engineer needs them, is mapped in Security Hardening § Personas, Surfaces, and Privilege Tiers.

State Machine

All stateful resources (Unit, Phase, Operation, UnitProcedure, Procedure, Batch) use the ISA-88 state machine.

States (13)

stateDiagram-v2
    [*] --> Idle

    Idle --> Running: Start

    Running --> Complete: SC
    Running --> Holding: Hold
    Running --> Stopping: Stop
    Running --> Aborting: Abort
    Running --> Pausing: Pause

    Pausing --> Paused: SC
    Pausing --> Stopping: Stop
    Pausing --> Holding: Hold
    Pausing --> Aborting: Abort

    Paused --> Running: Resume
    Paused --> Stopping: Stop
    Paused --> Holding: Hold
    Paused --> Aborting: Abort

    Holding --> Held: SC
    Holding --> Stopping: Stop
    Holding --> Aborting: Abort

    Held --> Restarting: Restart
    Held --> Stopping: Stop
    Held --> Aborting: Abort

    Restarting --> Running: SC
    Restarting --> Stopping: Stop
    Restarting --> Holding: Hold
    Restarting --> Aborting: Abort

    Complete --> Resetting: Reset

    Stopping --> Stopped: SC
    Stopping --> Aborting: Abort

    Stopped --> Resetting: Reset
    Stopped --> Aborting: Abort

    Aborting --> Aborted: SC

    Aborted --> Resetting: Reset

    Resetting --> Idle: SC

The 13 states comprise seven resting states (Idle, Running, Complete, Paused, Held, Stopped, Aborted, of which Running is a stable executing state and the rest are terminal or quiescent) and six transitional states (Pausing, Holding, Restarting, Stopping, Aborting, Resetting). Transitional states execute platform-defined actions and then auto-advance via the internal SC (State Complete) transition. Resting states wait for an operator or controller command. Note that Start moves Idle directly to Running. There is no transitional Starting state.

Commands (8 + StateComplete)

Command	Valid From	Transitions To
Start	Idle	Running
Stop	Running, Pausing, Paused, Holding, Held, Restarting	Stopping
Hold	Running, Pausing, Paused, Restarting	Holding
Restart	Held	Restarting
Abort	Running, Pausing, Paused, Holding, Held, Restarting, Stopping, Stopped	Aborting
Pause	Running	Pausing
Resume	Paused	Running
Reset	Complete, Stopped, Aborted	Resetting
SC (StateComplete)	Running, Pausing, Holding, Restarting, Stopping, Aborting, Resetting	Complete, Paused, Held, Running, Stopped, Aborted, Idle (respectively)

See Phases → The ISA-88 state machine wrapping for a per-state walkthrough of what each state means for a running SFC chart and which commands an operator can issue from it.

The state machine is thread-safe, supports callbacks on transitions, and maintains a history of all state changes. Commands reach equipment in two ways:

1. RequestedState (structural): The procedural layer writes a RequestedState field in the Unit's status subresource with a monotonic generation counter. The Unit controller processes new generations idempotently. This path also sets ActiveWork on the Unit, providing a persistent binding between the Unit and the procedural element executing on it.
2. dcs.io/command annotation (legacy/manual): Controllers and the gateway set the annotation on the target resource. This path remains for backward compatibility and manual operator commands.

A command travels on exactly one of these roads. The procedural layer used to mirror each command into the annotation as well as into RequestedState. The Unit then took the same command twice. Whichever copy arrived second was validated against the state the first had just produced, so a Start accepted into Running came back as "Start not allowed in state Running". Since #1774 the UnitProcedure and Phase reconcilers drive a Unit through RequestedState alone. The annotation road carries the commands RequestedState is not carrying: an operator or gateway command, the batch controller's Stop at a terminal batch, the IOModule controller's Hold on a device fault, and convergeUnitState, the repair path that exists precisely to reach a Unit whose RequestedState went nowhere.

Whose command it was is written down beside it. The annotation road carries two different things under one name. A person can address a Phase, an Operation or a UnitProcedure directly, and a parent forwards a command down the tree through the same annotation. The two deserve different answers when the receiving element refuses. A refusal of a person's command is an event in the plant and belongs in the Part 11 trail. A refusal of a forwarded one is not. forwardCommandToChild decides on the child's own live copy and writes at the resourceVersion it decided at, so the command was legal at the moment it landed. The child can still finish in the window between that write and the reconcile that processes it, and nothing in the forwarder is able to bind a child's own completion. The forwarder therefore writes dcs.io/forwarded-command in the same update, carrying the name of the command it forwarded. A refusal under a stamp naming that same command records nothing and is counted as dcs_reconcile_total{result="forwarded_command_overtaken"}. A command with no stamp is a person's, and so is one under a stamp that names a different command. Both keys are cleared together, so a stamp can only ever excuse the command it was written beside (#2105).

A repair is made for the element that occupies the unit, and for no other. convergeUnitState derives its command from the UnitProcedure's state against the Unit's. A terminal UnitProcedure over a Running unit therefore always has one to send: SC when it is Complete, Abort when it is Aborted. Terminal UnitProcedures are never requeued, which kept the converger quiet in steady state and hid that it never asked whose unit it was repairing. A fresh procedural-operator process reconciles every UnitProcedure in the namespace off its informer's initial list, and controller-runtime resyncs the same set every ten hours. On the bench that was 336 stale UnitProcedures, and the last to land, an Aborted one from a batch two weeks finished, aborted the live batch one second after the evicted operator came back (#1917). The converger now reads the Unit's own claim first. status.activeWork names the element running on the unit, and when it is set it decides. When it is empty, status.allocatedTo is the remaining claim, which is how the batch's own UnitProcedure still reaches a unit whose SC went nowhere after its completion cleared the snapshot. A UnitProcedure with neither has nothing to converge, and a Running unit with neither is the physical layer's orphan check to settle.

A command the state machine refuses is recorded on the Unit as the CommandRejected condition, naming the state the refusal was computed in. A refused command is consumed and nothing retries it, so the only thing that can answer the condition is something the state machine takes next. Both roads write the condition, and both write it below the metadata update that strips the annotation. A status change made above that update is replaced by the stored status and never reaches etcd.

The condition is removed by the next command the state machine accepts, including the ones the Unit issues to itself. The orphan Stop, the terminal-batch Stop, the auto-reset out of a resting state and the completion of a transactional state are all accepted commands, and each of them leaves the unit somewhere other than the state the refusal names. Before this was so, a Hold refused at Held survived the unit being stopped, reset to Idle and released from its batch, and it went on standing on an idle unit nobody was commanding.

The Hold the physical layer issues for itself counts the same way, and it arrives on a road further from the command paths than any of those. There are three of them: the runtime crash Hold once the recovery grace has run out, the Hold that precedes an automatic failover, and the watchdog Hold on lost field I/O. Each is taken well after the pass has written its verdict for the command roads. A refusal recorded while the unit was Running therefore outlived the crash that Held it, and the Progression column named Running over equipment that was Held.

A parent never forwards a command its child would refuse. The procedural controllers fan Hold, Stop, Abort, Restart, Resume and Pause down the tree, and in a sequential chart the child at the current step is usually the step that has just finished, because finishing is what makes the parent advance past it. Complete plus Abort is not in the transition table above. An operator aborting a batch whose earlier steps had completed therefore left the trail carrying refusals of a command nobody could have avoided, one per child that had got there first.

forwardCommandToChild takes that verdict from the transition table itself, and it takes the verdict and writes the annotation on one live copy of the child. The write carries the resourceVersion the verdict was computed at. Whichever of the two writes lands first, the other conflicts and re-reads. A child that finishes in the gap makes the forward fail, and its retry then declines to forward at all. A forward that lands first makes the child's own completing status write fail, so the child re-reads and processes the command it now carries. The same rule governs the level-triggered convergence a parent runs while it sits in a transitional state, which reads its children from the informer cache and so is the road most exposed to a state that has moved.

The batch controller forwards to its Procedure under the same rule, and it answers the consequence as well. A batch aborting a Procedure that has already reached Complete settles into Aborted by itself. Nothing is left to abort, and waiting for a state the Procedure can no longer reach would park the batch in Aborting for the life of the cluster.

Beside the condition, status.commandRejection carries the same refusal in structured form: the command, the state it was refused in, the requester where there is one, and the instant. The two are written and removed together. A surface renders the refusal from those fields and never by parsing the condition's sentence, which is why the Progression column on a batch's Allocated Units table can name a command without a copy of the reconciler's wording in the gateway.

Mode Management (ISA-88 Clause 7.3.2)

ISA-88 defines modes for procedural elements and equipment entities:

Entity Type	Modes	Behavior
Procedural (Procedure, UnitProcedure, Operation, Phase)	Automatic, Semi-Automatic, Manual	Controls how step transitions occur
Equipment (Unit, ControlModule)	Automatic, Manual	Controls whether control algorithms may manipulate the equipment

Automatic (default): Transitions occur without interruption when conditions are met.

Semi-Automatic: Transitions require operator confirmation. When a child step completes, the parent auto-pauses (Running → Pausing) and waits for a Resume command before starting the next step.

Manual: Steps are not auto-started. The operator must issue Start commands to each child explicitly and may execute steps in any order.

Mode is set via spec.mode (desired) and reflected in status.mode (observed). Runtime mode changes use the dcs.io/mode annotation, processed by controllers alongside commands.

Execution Model and Timing Guarantees

Cloud-Native DCS runs control logic on a cyclic-scan model that matches traditional PLC and DCS expectations, but it is hosted on Linux pods under Kubernetes, with no real-time microcontroller underneath. Engineers coming from hard real-time environments should calibrate their expectations here before deploying against fast processes.

Scan rates

Layer	Default rate	Configurable
Control Module — discrete-valve, solenoid-valve, analog-control	200 ms	via template scanInterval
Control Module — analog-sensor	500 ms	via template scanInterval
Control Module — pid-loop, pid-cascade, vfd	100 ms	via template scanInterval
SFC transition evaluation (phase logic)	Tied to the unit runtime's FB scan tick — each transition guard is evaluated once per scan cycle while the phase is in the Running state
Audit record write path	Synchronous with the action that generated it — not buffered
MQTT telemetry publish	Batched per scan cycle, store-and-forward queue on broker loss

Jitter and worst-case expectations

The unit runtime is a Go process scheduled as a Kubernetes pod with hostNetwork: true. Scan-cycle jitter depends on:

• Linux kernel scheduler latency (tens of microseconds to low milliseconds under normal load, spiking under memory pressure or noisy-neighbour CPU contention).
• Go garbage collector pauses (sub-millisecond to a few milliseconds on the current runtime heap sizes).
• Driver round-trip time to field I/O — Modbus TCP on a quiet LAN is typically 1–10 ms per read, EtherNet/IP assembly-object reads are similar, and OPC UA subscriptions amortise across the publish interval.
• The node-level PodDisruptionBudget and resource limits configured in Helm values.

Customers with hard real-time requirements (sub-millisecond jitter, guaranteed worst-case) should not rely on the Kubernetes-hosted runtime for those paths and should push that logic into a dedicated real-time controller or safety PLC wired as a downstream I/O consumer.

What this means for phase logic

• A transition guard that turns true between scan ticks will fire on the next tick. For a 200 ms-scanned discrete valve, expect up to 200 ms between "valve reports OPEN" and "SFC advances to the next step".
• TON / TOF / TP timers in the function-block runtime are scan-discretized. A T#100ms timer on a 200 ms scan layer effectively rounds up to the next tick.
• Operator prompts (PROMPT() calls in ST) block the SFC until the operator acknowledges and so inherit human-scale latency regardless of scan rate. AWAIT_RESULT() blocks on a laboratory. A laboratory's latency runs to hours routinely, which is why the step is required to declare its own timeout (ADR 0055).

Not a real-time operating system

The Kubernetes node kernel is a standard Linux kernel without PREEMPT_RT. Worst-case execution time is bounded by the Go runtime and kernel scheduler, with no formal WCET analysis behind it. This is appropriate for batch process control, where seconds-scale transitions are the norm. It is not appropriate for motion control, safety shutdown, or any function with a sub-10-ms response requirement.

See also High Availability and Failure Modes for failure-mode analysis and Capacity Planning for sizing guidance.

Unit Runtime

The unit runtime is a per-unit pod deployed by the physical operator. It bridges the Kubernetes control plane and physical hardware.

flowchart TD
    subgraph Pod["Unit Runtime Pod"]
        HTTP["HTTP API<br/>:61152"] --> FB["FB Runtime<br/>IEC 61131-3 scan"]
        FB --> Router["Address Router<br/>addr → default<br/>mod:addr → IO module"]
        Router --> Default["Default Driver<br/>modbus / opcua / sim / ethernetip"]
        Router --> IOD["IO Module Drivers<br/>from ConfigMap"]
    end
    Default --> Local["Modbus TCP<br/>or OPC UA"]
    IOD --> Remote["Remote I/O modules<br/>EtherNet/IP, Modbus"]

HTTP API

The unit-runtime exposes a JSON HTTP API consumed by the control-operator, procedural-operator, and gateway. All routes except /healthz require a Bearer token when --api-token is set, and use mTLS when the server is configured with a TLS cert/key/CA bundle (IEC 62443 SR 4.1).

Endpoint	Method	Description
/api/v1/controlprograms	GET	List all deployed ControlPrograms (map keyed by program name)
/api/v1/controlprograms/{name}	POST	Deploy a ControlProgram (replaces if already present)
/api/v1/controlprograms/{name}	GET	Get one ControlProgram's status (state, message, outputs, inputs, scanCount, specHash)
/api/v1/controlprograms/{name}	DELETE	Stop and remove a deployed ControlProgram
/api/v1/controlprograms/{name}/replace	POST	Hot-swap a running ControlProgram with a new definition
/api/v1/controlprograms/{name}/blocks/{block}/bypass	POST	Install a time-boxed interlock bypass on an output block, returning the computed deadline so the gateway can audit it (ADR 0010)
/api/v1/controlprograms/{name}/blocks/{block}/bypass	DELETE	Clear an interlock bypass early
/api/v1/read	POST	Read a process value ({address} → {address,value,quality,timestamp})
/api/v1/read-batch	POST	Read many addresses in one call; a per-address failure is reported inline (quality Bad) and the batch call still succeeds
/api/v1/write	POST	Write a process value ({address,value} → {success})
/api/v1/release	POST	Clear a simulation write override so the module's configured behaviors resume driving the channel (#990)
/api/v1/status	GET	Runtime status snapshot (protocol, connected, drivers, healthStatus)
/api/v1/tags	GET	Enumerate known tag addresses with cached values
/api/v1/tagmap	POST	Replace the in-memory CM tag map (pushed by the physical-operator when the tag map ConfigMap changes)
/api/v1/ioconfig	POST	Reconcile the live IO drivers against a fresh IOModule config set (pushed by the physical-operator when the iomodules.json ConfigMap changes)
/api/v1/diagnostics	GET	Driver + runtime diagnostics summary
/api/v1/lease/renew	POST	Record a control-lease grant from the physical-operator (ADR 0006, Failover mode). An equal-or-newer epoch renews the lease and unfences the runtime; a stale epoch gets 409, and Autonomy mode gets 400 because there is no lease to renew
/api/v1/hold/arm	POST	Arm an edge-local hold chart — the running phase's holdingChart, or the unit's safeStateChart baseline (ADR 0008)
/api/v1/hold/disarm	POST	Disarm an armed hold chart
/api/v1/hold/release	POST	Release a self-hold and the edge watchdog latch. The body names the road: an ISA-88 Restart out of Held, a later phase starting on the unit, or the engaged phase ending (#1919); no body is the Restart, and an unknown road is refused
/api/v1/heartbeat	POST	Control-plane liveness heartbeat; silence past availability.holdGraceSeconds is what fires the armed chart in Autonomy mode
/api/v1/stop/arm	POST	Declare that the next stop of this runtime ends it, so the field is safed on the way down and never frozen (#1283). Armed explicitly because the kubelet sends SIGTERM and nothing in the chain can be made to send anything else
/api/v1/stop/disarm	POST	Withdraw a terminal-stop arming
/api/v1/stop	GET	Read the current terminal-stop arming state
/healthz	GET	Health check (always unauthenticated)

ControlProgram deploy request body:

{
  "blocks":            [{"name": "...", "type": "...", "params": {...}}],
  "connections":       [{"sourceBlock": "...", "sourcePort": "...", "destBlock": "...", "destPort": "..."}],
  "varBindings":       [{"variable": "...", "blockRef": "...", "portRef": "..."}],
  "compositeTypes":    [{"typeName": "...", "dataInputs": [...], "dataOutputs": [...], "parameters": [...], "blocks": [...], "connections": [...]}],
  "interval":          "100ms",
  "specHash":          "<sha256 of canonical request>",
  "controlModuleRef":  "tank-1",
  "mode":              "continuous"
}

controlModuleRef is identity metadata (not hashed). Changes to it do not trigger redeploys. specHash is computed by the controller over the deploy-request JSON and is compared against the runtime's stored hash to detect stale content.

compositeTypes carries the composite FunctionBlockTypes the program's blocks instantiate, resolved cluster-scoped by the control operator and recursed to a nesting cap of 8. Each definition carries its declared parameters, because the runtime substitutes an instance's values into the inner blocks while it flattens (#1660). The runtime flattens them away in Load, before any block is instantiated (ADR 0065). Nothing downstream of that call ever sees a composite. The definitions enter specHash, which is what makes an edit to a block type redeploy every program resolving it.

Between the IEC 61499 to 61131-3 cutover and #1658 this field was sent by the control operator and read by nothing: the runtime's request struct did not declare it, so the bundle was dropped on decode and the composite type name left in blocks[].type resolved in no registry. A ControlProgram naming a composite was answered 400 unknown block type and reported as Deployed=False. The gateway now runs the same flattening the runtime does when a ControlProgram is created or updated, so a program the runtime could not load is refused where it is written.

IEC 61131-3 Phase 3 cutover (post-#361)

The HTTP API and persisted state file were renamed in the Phase 3 cutover:

Aspect	Pre-Phase-3 (legacy)	Post-Phase-3 (current)
Collection route	/api/v1/fbnetworks	/api/v1/controlprograms
Item route	/api/v1/fbnetwork/{name}	/api/v1/controlprograms/{name}
Replace route	/api/v1/fbnetwork/{name}/replace	/api/v1/controlprograms/{name}/replace
Legacy oneshot	POST /api/v1/fbnetwork, /status, /stop	removed
Wire field	dataConns	connections
HostPath state file	last-network.json	removed — see § Local Autonomy

Migration: the cutover is a hard switch across the runtime and all in-cluster callers (control-operator, procedural-operator, gateway). To upgrade a fleet:

1. Roll the new images for dcs-control-operator, dcs-procedural-operator, dcs-gateway, and dcs-unit-runtime together. The new operators only speak the new routes, and the new runtime only serves the new routes.
2. On the first reconcile after upgrade, every ControlProgram's specHash will differ (the canonical JSON now uses connections), so the control-operator will redeploy each program once. This is expected and leaves the runtime on the new wire shape.
3. The device node keeps local autonomy across the upgrade through networks/all.json, which the redeploy in step 2 rewrites. The cutover also removed the only writer of the singleton last-network.json state file, so the runtime no longer reads either name. A leftover copy on a node's hostPath is inert and can be deleted.

Function Block Runtime (IEC 61131-3)

The FB runtime implements an IEC 61131-3 cyclic scan execution engine. All blocks execute in declared order each scan cycle.

I/O blocks: - DI (alias IX): Digital input — calls driver.ReadValue(address) and coerces the result to bool via toBool(), guaranteeing OUT is always a boolean. It publishes a reading only if it stands behind it (ADR 0074), on the same terms as AI: when the driver reports Bad quality or the read fails, it holds OUT at the last trusted value, raises its PV_BAD output, and returns an error so the program reports Degraded with the block named. There is no fail band, because a bit has no out-of-range. A broken contact reads exactly like an open one. The driver's quality is the only evidence a digital failure leaves. Uncertain is treated as usable, matching AI and interlock.evaluate. A channel the instance declares unconnected still holds and still raises the flag, and does not fault (#1920). The held value is an operating point and survives a process restart (ADR 0080) - DO (alias QX): Digital output — coerces the bus input to bool via toBool(), calls driver.WriteValue(address, value), and exposes the coerced value on OUT (always a boolean) - AI: Analog input — reads a value via driver.ReadValue(address) and applies linear scaling (rawMin/rawMax → engMin/engMax). When engMin/engMax or rawMin/rawMax are omitted, the block queries the driver's TagMetadataProvider interface on first execution to auto-discover the correct ranges. Explicit params override auto-discovery. Each range is declared whole or left entirely to discovery: engMin without engMax is refused at Init, and rawMin without rawMax is refused the way the AO block already refuses it, since inventing the other bound would leave the runtime enforcing an interval the document does not declare. For simulation Expr behaviors (which store engineering values directly), the driver reports RawMin=EngMin, RawMax=EngMax, producing identity scaling. The block publishes a reading only if it stands behind it (ADR 0074): when the driver reports Bad quality, the read fails, or the raw value falls outside an optional rawFailLow/rawFailHigh band, it holds OUT at the last trusted engineering value, raises its PV_BAD output, and returns an error so the program reports Degraded with the block named. RAW deliberately does not hold, since it is the diagnostic. Uncertain is treated as usable, matching interlock.evaluate. The fail band is absent by default and exists for the drivers with no quality channel of their own, which are Modbus and EtherNet/IP. On a card whose device profile knows its in-band range diagnostic, the Modbus driver reports Bad from the card's own bits and no band is needed (#1740). - AO: Analog output — clamps the bus input to outMin/outMax (engineering units) and converts it to the device's raw range (rawMin/rawMax) at the write. Explicit params override driver TagMetadataProvider auto-discovery. With neither, the value passes through unconverted (identity), and that is what simulation drivers expect. Giving rawMin without rawMax is refused at Init, since a defaulted span would invert the range. Drivers whose target is an integer register round the converted value to the nearest count. The OUT port — and therefore PID TRK_VAL back-calculation — stays in engineering units

Logic blocks: - AND, OR, NOT: Boolean logic - BOOL_CONST, REAL_CONST: Constant value sources (boolean, floating-point) - GT, LT, EQ: Comparison operators - ADD, SUB, MUL, DIV, MOD: Arithmetic operators - ABS, MAX, MIN: Magnitude and selection arithmetic - SEL: Multiplexer (select between inputs) - PID: Proportional-integral-derivative controller — parallel form with output clamping and integral anti-windup. TRK/TRK_VAL inputs back-calculate the integrator against a forced value, giving bumpless transfer when it releases: wired from a device interlock's actually-written output, or from FBContext.SelfOverridden when a readwrite tag bound to the block's own OUT port (e.g. a pid-loop instance's CV) is written directly, which only reaches the block while the ControlModule is in Manual mode (#1255). The PV_BAD input holds the loop when the upstream input block cannot trust its reading: it folds into the same TRK path, so with TRK_VAL wired the loop holds at the value it actually wrote and resumes bumplessly, and with TRK_VAL unwired the output is held outright, since a memoryless P term would otherwise keep answering a stale error. It is a separate port because Bus resolves a destination to the first matching connection and a second wire into TRK would be silently dropped (ADR 0074). A promotion is the fourth source of a forced value (ADR 0082, #1815). While the block feeding TRK_VAL is holding what it read back from the plant, the loop tracks that value and does not act on its own error against a setpoint constant whose promoted copy is a compile-time default. The release is one last back-calculation. That is what leaves the loop with an integrator answering to the setpoint that arrived with the command. FBContext.SourceOutputPhase is how the block asks, since the condition is a runtime state and no value on the bus can carry it. A withheld source is deliberately not tracked onto, because its OUT port carries a compile-time default as well. PID is the first ManualOverridable block — Runtime.ReleaseManualOverrides clears such an override on return to Automatic mode (#1715), scoped to the interface so a plain readwrite constant (e.g. a REAL_CONST-backed SP) is never swept up - DEVSTATE: Device-state priority encoder (ADR 0018) — Boolean inputs IN1…IN8 evaluated in order, params state1…state8 name the word for each, first true wins, default when none, and OUT is a string. Backs a template's derived role: state tag. Its output is read-only at every layer: the tag is accessLevel: read and the runtime refuses output overrides on the block (ReadOnlyOutputBlock)

Stateful blocks: - TON, TOF, TP: On-delay, off-delay, and pulse timers - CTU, CTD: Up and down counters - SR, RS: Set-dominant and reset-dominant flip-flops - R_TRIG, F_TRIG: Rising and falling edge detectors

Execution model: All programs run as cyclic scans at a configurable interval (e.g., 100ms). Each cycle executes all blocks in the declared order, reads inputs from data connections, and writes outputs. There is no event-driven scheduling.

Connection validation: every endpoint of every data connection is resolved when the program loads. A program carrying a dangling one is refused, and never reaches the scan. Both block names have been checked since the runtime was written. The source port is checked against the source block's Outputs() (#1650) and the destination port against the destination block's InputPorts() (#1655). The runtime is the layer that answers this, because it holds the instantiated block and can ask it. A connection that resolves to nothing is silent at run time, so a refusal at load is the only place it surfaces. See ADR 0064 and its amendments for what each half costs.

Parameter validation: a parameter a document does not set takes the block's documented default, and so does one set to an empty string, because a template declares a placeholder that way for an instance to fill in and pkg/templatecompiler drops a reference that resolved to nothing. A parameter set to a value that does not parse is refused at load (#1663). The two used to be answered the same way for the engineering ranges and the PID gains. A mistyped bound or gain became the default with no log line, no condition and no faulted block, and the program reported Running on a number nobody wrote. The refusal names the block type and the parameter, since the load reports only the block name from the document.

Deployment flow: The control-operator watches ControlProgram CRs and automatically deploys them to the unit runtime via HTTP. Updates to existing programs use hot-swap (POST /replace) for zero-downtime changes.

Protocol Drivers

Drivers implement the Driver interface:

Driver	Status	Hardware
Modbus	Implemented	Modbus TCP devices
OPC UA	Implemented	Industrial PLCs
Simulation	Implemented	No hardware (in-memory); configured inline on each IOModule via spec.simulation with optional SimulationPreset references for reusable equipment templates; fault injection and deterministic RNG. It publishes only what it holds: an address its store has no value for reads Bad with the reason, unless it is a declared output, which models an unwritten register and reads zero (ADR 0074)
EtherNet/IP	Experimental — opt-in only	EtherNet/IP I/O modules (assembly objects); requires --enable-experimental-drivers

The Modbus driver supports configurable unitID, timeout, speed, byteOrder and wordOrder options, and refuses any other key outright. A channel address carries an optional width suffix (holding.100:float32) for values spanning more than one register. A bare address is one 16-bit register, as it has always been. The two order options say how a device lays such a value out, which is the one thing Modbus itself does not specify (see 32-bit and wider values). It auto-reconnects on read/write failure: if a TCP connection drops, the driver transparently re-establishes it and retries the operation once. A block that errors on a scan is skipped and named in ControlProgram.status.faultedBlocks, which the physical operator raises a High equipment alarm from and clears when the scan comes clean (ADR 0078). The batch is untouched: a degraded program is still regulating every healthy output. The fault is announced once per episode and not once per scan. The runtime logs one ERROR line when a block's run of faulted scans begins and one INFO line when it ends, each naming the network, the module and the block, and it counts the episodes in dcs_cm_block_faults_total with the blocks currently faulted in dcs_cm_blocks_faulted (#1920). Those two move on the block's own edge. The program-level Degraded state is an OR over the blocks, so a second block faulting under a standing one moves neither it nor dcs_cm_program_degraded_total. Losing field I/O is the other half of the same ruling. The unit controller's watchdog Holds a running batch when any of the unit's field drivers disconnects, annunciating Critical for a total loss and High for a partial one, naming the drivers that went.

If the device is unreachable at startup, the runtime starts in a degraded state and the driver lazily reconnects on the first successful I/O attempt. The Modbus driver also implements the Discoverer interface for WAGO 750-series I/O systems. It reads the coupler's module identification registers to auto-detect installed modules and generate channel layouts with correct process image addresses. The table's first word identifies the controller. Analog and specialty modules report their part number. Digital modules report a bit-coded word (bit 15 set, direction bits, and the module's size in bits), and both encodings are parsed. The table is fetched as one block read from its base, because a coupler may serve it only that way: on a 750-352 a single-register read of the base answers the coupler's own ID and every register after it answers zero, and so does a block that starts past the base. A profile walking such a table one register at a time sees a single entry and concludes there is no table (#1740). A device that refuses the block read is still walked, so the register-at-a-time path remains the fallback. On couplers that expose no module table at all, it falls back to reading the process image size registers and creates channels from the aggregate I/O counts. Those registers report bits, including for the analog areas.

That fall through is quiet and costs more than a count. The process-image path names no card. The channels it creates therefore carry neither a raw span nor a fault mask, and every consumer downstream reverts to its own defaults with nothing reporting a problem.

An address in that layout is the running sum of the footprints of the modules in front of it, so one card the profile cannot size moves every card behind it in the same image (#1744). A digital card's size is in its own identification word. An analog or complex card's is not, and two cases leave it open: a card configurable for more than one channel count, and a card the catalog does not carry. Both are settled against the coupler's analog process image totals, which name the one remaining footprint once the cards of known width are subtracted. What that cannot settle is not guessed. Discovery publishes the channels in front of the ambiguous card and returns a deviceprofile.PartialDiscovery naming what it left out, which the driver logs while keeping the channels. A layout that places nothing falls back to the process image totals, which give every address in the node and name no card.

Device-specific knowledge (register maps, module catalogs) lives in device profiles, analogous to EDS files (EtherNet/IP) or GSD files (PROFINET). Each profile implements the ModbusProfile interface (Probe + Discover) and self-registers at startup. The Modbus driver iterates registered profiles to identify the connected device and delegate discovery. The first shipped profile covers WAGO 750-series couplers.

A profile also carries what a card's own range diagnostic looks like. Some cards publish one inside the measurement word itself, with no second address to read (ADR 0074, #1740). A WAGO 750-454 or 750-459 holds a 12-bit value in B14 ... B3 and sets the low bits for measurement range underflow, a broken wire, or overrange. The Modbus driver resolves that mask off the same discovered layout it serves scaling metadata from, and reports Bad quality on a read that carries the bits. Nothing is declared per channel to get it. The mask is claimed only for a card whose manual documents those bits, and only a single-register read can carry it, since a wider read pulls a second register into the same value.

Drivers may also implement the optional TagMetadataProvider interface to expose scaling metadata (EngMin/EngMax, RawMin/RawMax) for their tag addresses. The simulation driver implements this, allowing AI and AO blocks to auto-discover the correct scaling without manual configuration on every control module. The Modbus driver implements it from device-profile discovery: channels whose card family defines a raw span (a WAGO 4-20 mA card's left-justified 0–32767) serve that span, and only the raw half. Engineering ranges are process knowledge the wire cannot carry, so blocks keep their own engineering defaults unless configured. A zero-width range is never served, and consumers ignore a zero-width half. Scaling into a constant is what that avoids. By default every simulation behavior stores engineering units and reports RawMin=EngMin, RawMax=EngMax (identity scaling), so the IOModule channel table shows believable values. A behavior that models real analog-input hardware can set raw: "true" to emit raw ADC counts instead, reporting a raw range of 0–65535 (16-bit ADC) so AI blocks perform a real raw-to-engineering conversion.

Local Autonomy

The runtime persists every deployed control program to the host filesystem at /var/lib/dcs/runtime/{unit-name}/networks/all.json, rewritten by NetworkManager.saveAll whenever a program is deployed or stopped. On startup NetworkManager.RestoreAll reads that file and redeploys each program as a cyclic scan, before the runtime has spoken to the control plane and whether or not it can. This is what provides local autonomy for edge devices: a restarted pod resumes its programs from disk alone.

A restored program's outputs are still fenced until the runtime holds its lease in Failover mode (ADR 0006), so the scan runs and the writes are dropped until the node is authoritative again. In Autonomy mode there is no fence and the outputs resume immediately, which is the mode's point.

There used to be a second, singleton copy of this state in last-program.json, read once at startup and scanned four times. Its writer was the legacy oneshot deploy route, removed in the IEC 61131-3 Phase 3 cutover below. The reader outlived it by three months. It loaded nothing and logged "no persisted network found", which is also what a healthy first boot on a fresh node logs. Had the file existed, the program it held would have run against the runtime's primary driver, an unconfigured simulation driver that at the time answered an unknown address with 0.0 at quality Good (#1748). That driver reports Bad for such an address now (#1955). The path was deleted, because RestoreAll already delivers the promise correctly and continuously. make lint-runtime-state-files now pairs every data-directory read against a writer. A reader cannot outlive its file again.

Device interlocks ride inside the control program, so they remain active through a control-plane partition: a DO/AO block with an ILCK input or interlockAddress parameter keeps evaluating its trip condition every scan and forces its safe value when tripped, with no cluster round-trip. They are the only always-on, mode-independent protection layer with this property. Phase guards and alarm exception actions normally execute control-plane-side (ADR 0007), though the procedural holding response can additionally be armed at the edge to survive a partition (see below). A first-class interlock bypass is enforced the same way. Its time-boxed deadline is evaluated in the FB scan, so it auto-expires and re-arms the interlock on the edge even during a partition. It is permission-gated (interlock:bypass) and audited at the gateway (ADR 0010).

Edge-local holding (ADR 0008) makes the procedural safe-state layer partition-tolerant. A device interlock forces a single output to a single value. It cannot sequence. The layer that runs a deliberate, ordered safe-state response (close the charge valve, then stop the agitator, then open the vent) is the phase SFC, and that normally executes in the procedural operator. On a partition it stalls mid-step, and the equipment freezes wherever the sequence happened to be. To close that gap the unit runtime embeds the pkg/sfc engine with a tag handler bound to the local control-module space, and arms a hold chart: the running phase's holdingChart (downloaded at phase start), or the UnitSpec.safeStateChart baseline when no phase is active. When the control-plane heartbeat watchdog fires (the lease renew in Failover, a lightweight liveness heartbeat past availability.holdGraceSeconds in Autonomy), the embedded engine runs the armed chart to drive a sequenced safe state by writing FB input variables. The FB scan remains the sole driver writer, and device interlocks still sit underneath. In Autonomy the runtime holds and keeps holding (it is the legitimate single writer). In Failover it holds for a bounded sequence then self-fences so a standby can re-bind cleanly (the re-bind margin is ≥ the hold duration, so there is never a dual writer). An armed chart may only READ/WRITE the unit's own control-module tag space, and may call only the builtins that work with no control plane to reach, a constraint shared by the runtime and make lint-edge-armable. A chart that prompts an operator, changes a mode, commands the phase state machine, or reaches another machine is refused at arming time and the phase falls back to the unit baseline.

Arming is process memory in the runtime, so it goes with the pod. Both brokers therefore keep it repaired, and neither sets it once and walks away. The physical operator arms the baseline for every unit in either availability mode, and the runtime answers with what it currently has armed on every lease renewal and every heartbeat. A runtime reporting its baseline slot empty is re-armed on the next one. That is how a restarted pod recovers the posture with no operator involved. The last report is on Unit.status.runtimeBinding.edgeHold. That gives the question "is anything armed at the edge" an answer outside the runtime's own metrics port (ADR 0008 amendment, #1806). The gateway resolves that report, plus the unit's own spec.safeStateChart, into one verdict served as edgeHold on the Unit. It reaches the Unit detail page's Edge Hold row and the EDGE HOLD column of dcs get units, and dcs get runtime prints the runtime's own live arming beside it. Nobody has to reach for kubectl to read it (#1833). The report belongs to the runtime that made it: a re-bind or an in-place move onto a different address drops the loop's memory of the previous runtime, so the record reads not reported until the promoted runtime answers. A moved report nudges the reconciler, so the record does not wait on its next pass (#2137).

Hold-lifecycle events buffer onto the MQTT store-and-forward queue and materialize as AuditRecords on reconnect. On reconnect the runtime's self-held status reconciles the phase to Held, where that status is about the phase reading it. A hold belonging to other work is not that phase's cover, and it is not reflected onto it (#1804). Recovery of a Held phase is an explicit ISA-88 Restart, and a silent resume does not exist. A hold that no Restart can reach is answered by the work that moves on instead (#1919): a phase starting on the unit while the runtime is self-held under other work, or the engaged phase reaching a terminal state. The control plane names the road on the release, and the runtime records it. There is one road the runtime takes on its own. A pod that restarts mid-partition inherits a hold snapshot off disk without running the hold, and the runtime retires that snapshot when the control plane proves it is running that runtime again. A truly dead node (not merely partitioned) runs no software, so that case still falls back to device fail-safe / last value (ADR 0006) until a standby re-binds.

Because interlocks (and any edge-local holding logic) ride inside the scan, the scan is kept alive through survivable faults: a single block's runtime error degrades the program, and the rest keep regulating and protecting every healthy output. Deliberate program removal drives each output to its configured fail state, with nothing left frozen: failState: safeValue (the default) writes the block's safeValue, and failState: holdLast keeps the last commanded value (IEC 62443 SR 3.6, #972). Ungraceful process death falls back to the remote-I/O module's comm-loss watchdog. ADR 0009 left that watchdog to the deployment layer, because its configuration is vendor-specific.

What the replacement process writes when it comes back is a separate question, and until #1776 the answer was a compile-time default. RestoreAll redeployed the programs from disk and rebuilt every block empty. A restart therefore discarded both the commanded values an operator or a recipe had written and the operating point inside each block that regulates. A settled loop came back computing zero against a setpoint it was already at, and drove its final element to the bottom of its range on the first scan. The commanded values and the operating points travel in networks/state.json now, snapshotted once a second and installed between Load and the scan goroutine. The first scan after a restart therefore writes a continuation. An orderly stop takes one more snapshot and waits for it before anything else in shutdown moves. A rolling update, an image bump and a graceful eviction therefore resume exactly. Up to a second of commanded state was lost on each of them until #1808. fbruntime.StatefulBlock is what a block implements to carry one, and its doc comment says which blocks do and which deliberately do not. See ADR 0080.

The same issue made the graceful degrade reachable. Every field driver held one mutex across a whole exchange and took the same mutex to answer IsConnected, and the Modbus transport's dial timeout is five seconds against a one-second gRPC liveness budget. So under total field loss the pod was killed on probe timeouts, and never got to report NOT_SERVING after its 60-second grace. IsConnected answers from memory on every driver now, and TestIsConnectedNeverWaitsOnIO reads the source of each one to keep it that way. The IOModule declares it since ADR 0068. spec.failSafe is where a node says what its outputs do with nothing of ours running. An undeclared one reports itself, and the product writes nothing to the device (#1683).

The declared timeout carries a floor of three io-probe read cadences, 45 s, refused at admission (ADR 0071). Any telegram feeds a device watchdog. The io-probe reads every device it serves once every 15 s, so a shorter deadline is reset by our own traffic before it can expire on purpose. The FailSafeTimeoutMargin condition then reports the gaps each device actually sees, because the probe loop is serial and an unreachable module costs every module behind it a dial timeout.

Telemetry Store-and-Forward

When the MQTT broker is unreachable (WAN outage, broker restart), the runtime buffers telemetry messages to a persistent on-disk queue at {DataDir}/queue/queue.jsonl. Messages are replayed in FIFO order when connectivity is restored. This prevents telemetry gaps that could trigger compliance questions in pharmaceutical environments (21 CFR Part 11).

The queue is also what a starting runtime talks to. Start opens the broker connection in the background and returns (#1933). The retained status it publishes is queued until the broker answers. Every connect re-announces the current status through the same queue, so a broker that comes back holding nothing is told what the runtime is now. The health server and the lease endpoint therefore never wait on the broker, and a Failover runtime started during a broker outage is granted its lease inside the outage.

The device drivers are dialled the same way (#1936). A driver on a network protocol dials on a goroutine of its own, and Start returns with the dial in flight. The scan's first I/O and the driver's health monitor retry a dial that failed, and the monitor now starts on a failed first dial as well as a successful one. The simulation driver is connected before Start returns, because its reads refuse until it is. On the bench one Modbus dial against a coupler the node had no route to was 3s of the 15s to grant (ADR 0006, 2026-09-03 amendment).

The queue uses an append-only JSON Lines file with a separate offset tracker. On crash recovery, partial writes are safely discarded. When the queue exceeds its size limit, the oldest entries are evicted via file rotation. The --mqtt-queue-size flag controls the maximum buffered messages (default 10000, 0 disables the queue).

A runtime with the queue disabled has nowhere to put a message the broker did not take, so it discards it. That is the intended behaviour of 0, and it is counted: dcs_runtime_mqtt_publishes_total{outcome="dropped"} rises for every message lost that way, and for any payload that fails to marshal.

Messages the broker refuses

The queue outwaits a broker that is unreachable. A broker that is reachable and refuses the message is a different fact, and it used to be read as the first one. A refusal answers the publish over a working link with a reason code saying the message will not be accepted. The replay loop treated that as a lost connection and left the offset where it was. The only thing that could call the replay loop again was a reconnect, and a healthy link never produces one. One refused message therefore held the entire queue for the life of the pod. A bench runtime sat at a depth of 8404 for half an hour with a single unauthorized topic at the head. The hold-lifecycle events queued behind it never reached the control plane at all (#1780).

The runtime now separates the two:

• A refusal the broker will repeat (an invalid topic name, an invalid payload format) is written to {DataDir}/queue/deadletter.jsonl with its payload, its reason and the time it was given up on, counted on dcs_runtime_mqtt_queue_dead_lettered_total, and advanced past. The queue is one FIFO with one offset, so this is what makes delivery of the messages behind it possible at all. Nothing in the product reads the dead-letter file. It is read by hand, from the pod, when the counter moves. It is capped at the configured queue size, and the topic is named in the runtime log the first time it is refused.
• A refusal that says the runtime is not authorized for the topic is held instead, in {DataDir}/queue/deferred.jsonl, and offered again for fifteen minutes from its first refusal. It comes off the head of the queue like a dead-lettered one, so nothing behind it waits either way. Every other code reads the packet. This one reads the broker's ACL. The remedy for a wrongly refused topic is a chart change, so the verdict is at its least reliable during exactly the operation that fixes it. A message the window runs out on is dead-lettered then. See ADR 0081 § Amendment.
• A refusal that may be answered differently later, such as a quota or a packet identifier already in use, is left where it is and retried.
• Everything else is treated as a possible connection loss, and the pass stops at the entry that failed without offering the ones behind it.

The replay loop runs on a 30-second timer as well as on every reconnect. A queue that stalls for any reason therefore resumes on its own. It does not wait for a connection edge that a healthy link will not produce. The deferred store is offered on the same wake-ups, paced to one sweep every two minutes and un-paced by a reconnect. It cannot wait for a connection edge either, and for a sharper reason: the chart delivers an ACL change to a running broker through the passwd-reloader sidecar, so widening a grant produces no disconnect at all.

Prometheus metrics: dcs_runtime_mqtt_queue_depth (current pending messages), dcs_runtime_mqtt_queue_replayed_total, dcs_runtime_mqtt_queue_evicted_total, dcs_runtime_mqtt_queue_dead_lettered_total, dcs_runtime_mqtt_queue_blocked (1 while the head of the queue is not moving), dcs_runtime_mqtt_queue_deferred with _deferred_total and _deferred_recovered_total beside it, and dcs_runtime_mqtt_publishes_total broken out by kind and outcome. Depth and the replay counter cannot tell a held queue from an idle one, which is why dcs_runtime_mqtt_queue_blocked exists.

Port Allocation Policy

Unit runtime pods use hostNetwork: true so they can reach field I/O devices on the host's physical network interfaces. Because host-network pods bind directly to the node's network stack, their server ports must not fall within the Linux ephemeral port range (default 32768–60999, net.ipv4.ip_local_port_range). Ports in this range can collide with kernel-assigned source ports for outbound connections, causing random bind: address already in use failures.

Policy: all DCS server ports must be < 32768 or > 60999.

Component	Port	Protocol	Notes
Unit runtime health	61052 (default)	gRPC	Per-unit unique; set via runtimeHealthPort. Two named health services: readiness (fenced or field-less is NOT_SERVING) and liveness (alive and scanning), so the two probes can disagree (#1918)
Unit runtime HTTP API	61152 (default)	HTTP	Per-unit unique; set via runtimeGRPCPort (name kept for API compatibility — the port serves HTTP despite the name)
Unit runtime metrics	61252 (default)	HTTP	Per-unit unique; set via runtimeMetricsPort
io-probe HTTP	8090	HTTP	Below ephemeral range. Runs per-Controller and per-namespace network-io-probe pods — no collision with the gateway (different pods, different IPs)
io-probe health	8091	HTTP	Below ephemeral range
Gateway HTTP	8090	HTTP	ClusterIP service, without hostNetwork
Gateway metrics	61051	HTTP	ClusterIP service, plaintext /metrics
Historian HTTP	8092	HTTP	ClusterIP service
Historian health	8093	HTTP	ClusterIP service
Historian metrics	8094	HTTP	ClusterIP service, plaintext /metrics
OMF egress health	8095	HTTP	ClusterIP service. Readiness follows the MQTT broker alone: an OMF endpoint that is down is the case the component rides out, and reporting unready would restart the one process holding the queue
OMF egress metrics	8096	HTTP	ClusterIP service, plaintext /metrics
Operators (metrics)	8080	HTTP	ClusterIP service
Operators (health)	8081	HTTP	ClusterIP service
MQTT broker	1883 / 8883	MQTT	ClusterIP service

When deploying multiple unit runtimes on the same node, each of the three runtime ports (runtimeHealthPort, runtimeGRPCPort, runtimeMetricsPort) must be unique across pods on that node, since hostNetwork: true makes them global on the node. The three ports are independent. Set all three explicitly in the Unit spec to co-tenant multiple runtimes on one node. Example: two units could use (61060, 61160, 61260) and (61061, 61161, 61261) respectively.

Multi-tenant demo deployments do this assignment for you. Both provisioning paths, the gateway's auto-provisioning and hack/demo-provision.sh, give each tenant its own consecutive slice of three reserved ranges, 61020–61499 for health, 61500–61979 for the HTTP API, and 61980–62459 for metrics, rewriting the fixture set's own declared ports as they template it. The starting point is a checksum of the username, and the search walks forward over what the cluster already holds. A tenant therefore keeps the same ports across re-provisioning without ever landing on another tenant's. Leave those three ranges alone when hand-assigning ports on a node that also hosts demo tenants.

For the full Prometheus metric catalogue, recommended alerts, and Grafana dashboards that consume these endpoints, see Monitoring and Metrics.

Defense-in-depth: on production nodes, consider reserving the DCS port block in ip_local_reserved_ports (e.g., via Talos machine config):

sysctl -w net.ipv4.ip_local_reserved_ports="61051-61252"

Gateway

The DCS gateway provides:

• REST API for managing the ISA-88 hierarchy (CRUD operations on all CRDs)
• Web UI for browsing the physical hierarchy, managing batches, and monitoring the procedural tree
• Device registration — Controller CRUD with node-label management. Cluster node join happens at the deployment layer (ADR 0004)

Enrollment Flow

Cluster join is the deployment layer's job, done with Talos machine config, a k3s token, or a cloud nodegroup (see ADR 0004). The product adopts the resulting node:

sequenceDiagram
    participant Deploy as Deployment layer
    participant K8s as Kubernetes
    participant Op as Physical operator
    Deploy->>K8s: 1. Join node (dcs.io/site=<site> label)
    K8s-->>Op: 2. Node appears
    Op->>Op: 3. Auto-create Controller CR (or bind pre-created CR by name)
    Op->>K8s: 4. Reconcile spec.nodeLabels onto node
    Op->>Op: 5. Controller: Pending -> Joining -> Joined

See the Device Enrollment Guide for the step-by-step workflow.

Simulation controllers (spec.simulation: true) skip node adoption entirely. The Controller reconciler creates a virtual Kubernetes Node and maintains its heartbeat Lease, allowing simulation IOModules and unit runtimes to operate without physical hardware.

Operator Prompts

Phase SFC charts pause execution for operator input through three typed prompt builtins (ADR 0017). PROMPT('message') requests an acknowledgement and returns the fixed sentinel 'Acknowledged'. PROMPT_CHOICE('message', 'opt-a', 'opt-b') requests an enumerated decision and returns the selected option. PROMPT_VALUE('message',min, max, 'unit') requests a validated numeric entry within required bounds and returns REAL. The flow:

1. ST calls a prompt builtin — the bridge blocks, pending prompt state (including the type and, per type, the option list or bounds/unit) is published to Phase.status.pendingPrompt
2. UI shows the typed card — the phase detail view polls for pendingPrompt and renders the affordance matching the type: an Acknowledge button, one button per option, or a bounded numeric input, plus the e-signature reason field
3. Operator answers — POST /api/v1/sites/{site}/phases/{name}/prompt-response validates the response against the prompt type fail-closed (sentinel forced for acknowledge, option membership for choice, bounds for value), validates the e-signature (HMAC-SHA256, fresh OIDC token per 21 CFR 11.200), sets a dcs.io/prompt-response annotation, and creates a PromptAcknowledge audit record

4. Controller delivers response — a watcher goroutine detects the annotation and unblocks the bridge, which re-validates the response against the spec before SFC execution continues

5. Run ends — a prompt only lives as long as the runtime episode that raised it. If that episode ends without an answer (the phase aborts, stops, completes, or is reset), the operator clears Phase.status.pendingPrompt and the gateway rejects any late response with 409 Conflict. Otherwise the field outlives its owner: the HMI counts a prompt nobody can action, and a response against it would record an electronic signature bound to nothing (#992). Held and Paused are excluded. A Restart or Resume returns to the chart, so the prompt is still live.

This implements ISA-88 Part 4 Table 7 Prompt/Prompt Response event types. Free-form text is excluded from flow control. Operator notes belong in batch-record annotations.

Validation rule: Transitions from steps that contain a prompt builtin must have an explicit conditionST. Capture the prompt response in a variable (ack := PROMPT('...')) and use it as the transition guard (conditionST: "ack <> ''", conditionST: "cut = 'Cut now'", or a numeric comparison for PROMPT_VALUE()). Unconditional transitions from prompt steps are rejected by both the API and gateway validators.

Node Scheduling

Units declare a nodeSelector that is copied to the runtime pod's spec.nodeSelector. By convention, device nodes are labeled with dcs.io/device=<unit-name> to pin runtime pods to specific hardware.

Control Module Role Bindings

Phase templates must reference control modules unit-agnostically so that the same phase logic can run on different units with differently-named CMs. Units provide this abstraction through spec.cmRoles, a map from abstract role names to concrete ControlModule instance names:

UIYAML

/system → Equipment Library → Units → open the unit → scroll to CM Role Bindings. Each row maps a role name (declared by the phase templates this unit runs) to one of the unit's ControlModule instances. Edit or delete rows with the per-row action buttons.

spec:
  cmRoles:
    outlet_valve: discharge-valve
    temp_sensor: product-temp

The unit controller emits role-aliased entries in the tag map ConfigMap alongside the concrete CM entries. The runtime's tag resolution is transparent. WRITE('outlet_valve.CMD', TRUE) resolves to the same physical address as WRITE('discharge-valve.CMD', TRUE).

The tag map reaches the runtime on two paths: the ConfigMap volume (durable: what a restarted runtime loads, polled for changes every 10 s) and a direct push to the runtime's POST /api/v1/tagmap whenever the unit controller rewrites the ConfigMap. The push closes the kubelet volume-sync latency window (up to ~a minute) so a newly created ControlModule's tags resolve as soon as its FB network deploys. If the push fails, the file poll converges on its own.

The IOModule driver config (iomodules.json) follows the identical two-path pattern: the ConfigMap volume the runtime polls every 10 s (durable/fallback) and a direct push to POST /api/v1/ioconfig whenever the unit controller rewrites the ConfigMap. The push closes the same kubelet volume-sync window so a newly bound SimulationPreset's behaviors re-seed on the live driver in seconds. The interactive virtual-commissioning workflow would otherwise show a 60–90 s dead window.

Both polls reconcile only when the file's own content hash changes. A file that merely differs from what the runtime is holding is not a change. The push installs a set the volume has not caught up with yet, so a poll that compared the two would read the stale volume as a change and undo the push. The file keeps its authority. Once the ConfigMap itself changes, the poll applies it over anything a push installed.

PhaseTemplates declare their CM role requirements via spec.cmRoles for validation and documentation. At batch instantiation, the batch controller verifies that the target unit provides all required role bindings.

The Controller reconciler converges Controller.spec.nodeLabels onto the bound node (restricted to the dcs.io/ prefix), and Controller.spec.displayName onto the node's dcs.io/display-name annotation.

Both are instances of one rule. The gateway holds no write privilege against a Kubernetes node (ADR 0034), so anything the product authors about a node is written to a product-owned object and applied by the operator that already holds the privilege. Every surface then reads the value back from the node at request time, which keeps one answer to what a machine is called whether the deployment layer named it at enrolment or an engineer typed it in the UI.

Each converged field tracks what the operator last applied, in a physical.dcs.io/applied-* annotation on the Controller. Clearing the spec removes only what the operator itself wrote, so a label or a name the deployment layer set directly is never removed by a product edit.

Naming a machine that is not a Controller

A control-plane node or an unclassified worker has no Controller to author through, so NodeIdentity is that object for it: cluster-scoped, one per node and named for the node, carrying every product-authored attribute of the machine (displayName today, and a second attribute becomes a second field on the same kind). The NodeIdentityReconciler converges it onto the same annotation under the same ownership rules, tracking what it applied in status.appliedDisplayName.

The two authors never write the same node. A node a Controller has adopted is named on that Controller, and a NodeIdentity for it is Refused with the owning Controller named, because two authors converging on one annotation would each undo the other on every reconcile. ADR 0040 records the decision, including why a Server CRD and a narrow Node patch grant are both closed.

Taking a node out of service

NodeMaintenance is the same division applied to the node admin verbs. It is a cluster-scoped record of an operator's intent (cordon, drain, reboot, shutdown or power-on) with a lifecycle the physical operator advances, and the gateway writes the record without touching the node (ADR 0034). One object covers the whole procedure, so a cordon-only maintenance reaches Cordoned and waits there for return to service. It is the same kind of request as a drain, stopped earlier.

power-on is the one intent that runs the other way, and it lives on this object for the reason the rest do. The gateway holds no write against a node and no channel to the management network, so an operator's click has to become an object the physical operator acts on. It shares nothing else with its siblings: nothing is cordoned, nothing is drained, and none of the refusals below apply, because they protect a running machine and this one is already off.

Three refusals are terminal and carry their reasons. A virtual node has no hardware to service. An action that would drop ready control-plane nodes below etcd quorum is refused outright and never upgraded into a ceremony, because the result is an API server that has gone read-only and cannot be recovered from the UI that offered the action. And a node hosting live unit runtimes refuses a drain, naming the units, because moving process control runs through the ADR 0006 failover path: only an operator can certify that a node is fenced, and a reconciler cannot self-certify.

Return to service uncordons only what the maintenance itself cordoned. A node an engineer had already cordoned by hand stays cordoned, which is the same ownership rule the converged fields above follow.

A drain holds that cordon. It is re-applied on every pass, and no eviction is issued until it holds. An eviction is a request to terminate a pod and says nothing about where the replacement goes. On a cordoned node there is nowhere here, so the node empties. On a schedulable one the replacement lands back where the original was, and the next pass evicts that. A cordon the drain re-applied is one the maintenance owns and lifts, whoever removed it. A node is never left unschedulable with no path back through the product.

Every eviction the drain issues is written down. The pod gets a Normal EvictedForDrain Event naming the node and the record that asked for the drain. That record is either the NodeMaintenance an operator abandons or the Controller whose removal they cancel, and the remedy is the reason the Event names it. The physical operator logs the same line. The Event goes on the pod because the pod is what somebody is looking at when a workload keeps disappearing. Only an eviction the API server accepted is recorded. A PodDisruptionBudget refusal is logged as the refusal it is, and a pod already inside its termination grace period is never re-evicted at all.

Consent to stop a machine expires. The gateway bounds the impact check an operator acknowledges at ten minutes, so a reboot or a shutdown cannot be requested on evidence that has stopped describing the plant. That bound is enforced once, at admission. The drain between admission and the power verb is unbounded on purpose. A PodDisruptionBudget refusing an eviction is the normal outcome on a correctly configured cluster. The drain keeps its cordon and its retry, and it does not fail out. A maintenance can therefore reach the power verb hours after a person last looked at it. One on the bench sat draining for sixteen and a half hours with a live power channel behind it.

The refusals above do re-run on every pass, so the consequential checks are current whenever a machine actually stops. The operator's consent is not one of them and cannot be re-derived. It happened at one instant, in front of a person. So a reboot or shutdown reaching the power verb on an authorisation older than an hour does not stop the machine. It parks with the node drained, the cordon held, the census standing and the machine untouched, and publishes PowerConsentFresh: False.

Parking is not failing. Nothing this maintenance achieved is undone, and both ways forward are named on the record. Abandoning returns the node to service. Posting the same verb to the same node authorises it again, through the whole ceremony a first request walks. That ceremony is the reason, a freshly acknowledged impact check, and the typed node name. The second impact snapshot is then recorded beside the first. spec.preflight is what the operator was shown when they asked. spec.reconfirmed.preflight is what they were shown when they authorised the machine to actually stop, and it is what a reviewer reads beside a machine that stopped.

The hour is chosen from both ends. It has to clear any drain the product still considers to be working. That is the five-minute drain timeout, past which the maintenance already tells the operator the drain is not finishing. It also has to sit under anything an operator walks away from, since a break, a shift change and a night all exceed it. A drain intent is not bounded this way, because no machine stops. The node stays cordoned and under the operator's control, and abandoning returns it.

Standards Compliance

Cloud-Native DCS is designed for pharmaceutical and regulated manufacturing environments. The following standards guide the architecture. Each has a dedicated traceability matrix that maps requirements to CRDs and code:

Standard	Scope	Status	Traceability
ISA-88 / IEC 61512	Batch process control — physical, procedural, recipe, and batch models	Core architecture	Traceability matrix
IEC 61131-3	Programmable controller languages — SFC, ST, FBD for control logic	SFC engine + ST interpreter + FB runtime	Traceability matrix
21 CFR Part 11	FDA electronic records and signatures — audit trails, access control, electronic signatures, change control	Substantially Implemented	Traceability matrix
IEC 62443	Industrial cybersecurity — network segmentation, security levels, secure development lifecycle	Substantially Implemented	Traceability matrix

21 CFR Part 11 Implications

FDA 21 CFR Part 11 establishes requirements for electronic records and electronic signatures in regulated industries (pharmaceutical, biotech, food). Key requirements that affect this system:

• Audit trails: All changes to records (recipes, batch parameters, control logic, equipment configuration) must be logged with who, what, when, and why. Kubernetes resource versioning provides a foundation, but a dedicated audit log with user attribution is required.
• Electronic signatures: Certain actions (recipe approval, batch release, control logic deployment) must be attributable to an identified individual. Signatures must be linked to the specific record version.
• Access control: Role-based access ensuring only authorized individuals can create, modify, or delete records. Different roles for operators, engineers, and quality assurance.
• Change control: Modifications to approved recipes and control logic require documented review and approval before taking effect. Implemented via the ChangeControlPolicy CRD (cluster-scoped, changecontrol.dcs.io) and a change-control admission webhook hosted in batch-operator: in namespaces labelled dcs.io/production=true, mutations to change-controlled resources must originate from an allowlisted ServiceAccount and carry the required external-reference and e-signature annotations. See Change Control for the full workflow and approval gates.
• Record retention: Electronic records must be retained for the required duration and be available for FDA inspection.

IEC 62443 Implications

IEC 62443 defines a framework for securing industrial automation and control systems. Key areas:

• Zones and conduits (IEC 62443-3-2): Network segmentation between control plane (Kubernetes), field devices, and enterprise networks.
• Security levels (IEC 62443-3-3): System security requirements ranging from SL 1 (casual) to SL 4 (state-sponsored threats).
• Secure development (IEC 62443-4-1): Secure coding practices, threat modeling, vulnerability management.
• Component security (IEC 62443-4-2): Authentication, authorization, integrity, and confidentiality at the component level.

IEC 61131-3 Alignment

IEC 61131-3 defines five programming languages for control systems. Cloud-Native DCS implements three:

• SFC (Sequential Function Chart): Procedural sequencing. The engine publishes live execution status (active steps, transition evaluation results, and ST variable snapshots) to the Phase CRD status, enabling real-time monitoring. Transition conditions with AND-joined sub-expressions are evaluated per-operand for diagnostic visibility.
• ST (Structured Text): Step actions, transition conditions, and control module templates. Templates support direct ST authoring as an alternative to visual FBD editing.
• FBD (Function Block Diagram): Control programs with standard function blocks (PID, timers, counters, I/O, logic, arithmetic) executing in a cyclic scan loop. There is no event-driven scheduling, because every block runs every scan.
• ControlProgram CRD: Declarative definition of control programs (FBD or ST) as Kubernetes resources, with variable declarations for tag binding. Deployed automatically to unit runtimes.
• Hot-swap: Running programs can be updated in-place without stopping other programs on the same runtime.