Architecture

Cloud-Native DCS is a Distributed Control System that runs on Kubernetes. Every domain object (equipment hierarchy, recipes, batches, control logic, alarms, audit records) is a Kubernetes Custom Resource, reconciled by operators, and exposed through a REST API and browser-based Web UI. The Web UI is split into three role-aligned endpoints, /system (engineering), /hmi (operations), and /data (production information), documented in Terminology.

BPCS-layer protection — an independent SIS is a PHA/LOPA decision

Cloud-Native DCS is a Basic Process Control System (BPCS) in IEC 61511 terms. It is not a Safety Instrumented System (SIS) and is not rated for SIL 1, SIL 2, SIL 3, or SIL 4 functions under IEC 61508 / IEC 61511. Its device interlocks, phase guards, and alarms are BPCS-layer protection functions. Under IEC 61511 Clause 9.2 a BPCS interlock or alarm can be credited as an independent protection layer. Whether a given hazard (emergency shutdown, overpressure trip, LEL/combustible-gas trip, relief-valve integrity, burner management, or any function whose failure causes injury or loss of containment) also requires an independent SIS is the outcome of your process hazard analysis / LOPA, decided per process and per function. An independent SIS here means a separate, independently certified safety PLC or hardwired trip system. Assign protection layers at design time accordingly. The product does not make that call on your behalf (ADR 0011).

Within the BPCS scope, protection is layered per ADR 0007: device interlocks in the FB scan on the edge (always-on, partition-tolerant), phase guards for procedural safe-state sequencing (made partition-tolerant at the edge by ADR 0008), alarms for annunciation and audit. Reaction-time bounds are documented per alarm path in Alarm Management → Reaction time and the underlying decision is recorded in ADR-0002.

The pieces fit together as the CRD model, the operators that reconcile it, the state machine that governs batch execution, and the per-unit runtime that bridges Kubernetes to physical I/O. Standards alignment (ISA-88, IEC 61131-3, 21 CFR Part 11, IEC 62443) is summarized at the end of this document and detailed in the compliance traceability matrices.

What the System Does

At a high level, Cloud-Native DCS lets you:

• Model a plant as a tree of Enterprise → Site → Area → ProcessCell → Unit → ControlModule resources (ISA-88 physical model).
• Author recipes as MasterRecipe templates that the batch-operator clones into per-run ControlRecipe instances.
• Execute batches against those recipes: the batch-operator allocates units, builds a procedural tree (Procedure → UnitProcedure → Operation → Phase), and drives each element through an ISA-88 state machine.
• Run control logic on edge devices via a per-unit runtime pod that executes IEC 61131-3 function block programs against real or simulated I/O (Modbus TCP, OPC UA, and experimental EtherNet/IP).
• Manage alarms with an ISA-18.2 four-state lifecycle.
• Record everything with audit trails, historian telemetry, and auto-generated batch production records.

All of this is driven from a single Web UI, a REST API, the dcs CLI, or directly as Kubernetes resources, whichever fits the workflow.

The clip below is that claim performed on a single object. One control module, fermenter-1's agitator, is reached through the same equipment hierarchy in all three endpoints. It is read from a terminal through the same REST API the pages render from, and written once from the operator's HMI faceplate. Every surface then agrees on the new value: the engineering FBD streams it, the historian trend carries the step, and the audit trail carries the write that caused it.

One control module through every surface: the same hierarchy in /system, /hmi and /data, a dcs CLI read, an HMI faceplate write, and the FBD, trend and audit record all agreeing.

ISA-88 Overview

ISA-88 (IEC 61512) is the international standard for batch process control. It defines a hierarchy of models that separate what to make (recipes) from how to make it (equipment and procedures). Cloud-Native DCS maps each ISA-88 concept to a Kubernetes Custom Resource Definition (CRD).

ISA-88 / standard concept	Kubernetes CRD (API group)
Physical Model	physical.dcs.io/*
Enterprise	Enterprise (cluster-scoped)
Site	Site (cluster-scoped)
Area	Area
Process Cell	ProcessCell
Unit	Unit
Control Module	ControlModule
Controller	Controller
Procedural Model	procedural.dcs.io/*
Procedure	Procedure
Unit Procedure	UnitProcedure
Operation	Operation
Phase	Phase
Synchronization Barrier (Part 1 Clause 7.2)	SyncBarrier
Recipe Model	recipe.dcs.io/*
Master Recipe	MasterRecipe
Control Recipe	ControlRecipe
Batch Model	batch.dcs.io/*
Batch	Batch
Control Logic (IEC 61131-3)	control.dcs.io/*
Control Program	ControlProgram
Function Block Type	FunctionBlockType
Alarm Management (ISA-18.2)	alarm.dcs.io/*
Alarm	Alarm
Audit Trail (21 CFR Part 11)	audit.dcs.io/*
Audit Record	AuditRecord
Change Control (21 CFR Part 11 §11.10(k))	changecontrol.dcs.io/*
Change Control Policy	ChangeControlPolicy (cluster-scoped)
Change Request	ChangeRequest

API Groups

The tables below cover the CRDs each API group offers as a documented feature. For each one they give the operator that reconciles it and what it represents. A few kinds are deliberately absent. Their CRDs and controllers still ship, but their user-facing surface was withdrawn under #480. They are not something to build against. The README's API-group table is the complete inventory of what config/crd/bases/ installs.

physical.dcs.io/v1alpha1

Models the plant's physical equipment hierarchy.

CRD	Scope	Description
Enterprise	Cluster	Top-level organization (e.g., a company)
Site	Cluster	Physical location (creates a site-{name} namespace)
Area	Namespaced	Logical grouping within a site
ProcessCell	Namespaced	Collection of units that work together
Unit	Namespaced	Equipment that executes phases (reactor, mixer, filter-dryer)
ControlModule	Namespaced	Lowest-level I/O point (sensor, valve, actuator, PID loop). A smart instrument's own asset identity rides on spec.fieldDevice — the IEC 61987 nameplate, serial number included (ADR 0043)
ControlModuleTemplate	Cluster	Reusable CM template — compiled by the control-operator into a ControlProgram when referenced by a ControlModule
Controller	Namespaced	Physical computing device, bound to the cluster node of the same name. Set spec.simulation: true to create a virtual node with no real one to wait for
IOModule	Namespaced	The record of one addressable field device: the endpoint the runtime exchanges process data with, plus spec.fieldDevice (declared type of io-module, control-device, instrument or gateway, with vendor/model/serialNumber/firmware) saying what the box behind it is. Covers a fieldbus coupler (e.g., Wago 750, Turck TBEN), a third-party PLC we do not run, and a simulation stand-in. Simulation IOModules carry inline spec.simulation config (behaviors, faults, RNG seed, optional preset reference + address map)
SimulationPreset	Namespaced	Reusable equipment simulation template with generic address names. Referenced by an IOModule's inline spec.simulation.preset; expanded at reconcile time
NodeIdentity	Cluster	Product-authored attributes (displayName) of a machine that has no Controller to author through — a control-plane node or an unclassified worker. One per node, named for the node (ADR 0040)
NodeMaintenance	Cluster	An operator's intent to take a node out of service (cordon, drain, reboot, shutdown) or bring one back (power-on), with a lifecycle the physical-operator advances (ADR 0034)
SiteOutage	Cluster	A planned shutdown of a whole site — the plant safed and quiesced, then either powered down through a channel that survives the cluster (unattended) or left running for a manual cut at the rack (attended). See Planned Outage

Hierarchy:

flowchart TD
    Enterprise --> Site[Site<br/>creates namespace site-name]
    Site --> Area
    Area --> PC[ProcessCell]
    PC --> Unit[Unit<br/>runs runtime pod]
    Unit --> CM[ControlModule]

procedural.dcs.io/v1alpha1

Models the sequence of actions to execute a batch, plus the reusable templates authors work with.

CRD	Scope	Description
Procedure	Namespaced	Top of the procedural tree. Sequences one or more UnitProcedures via an SFC chart (spec.chart)
UnitProcedure	Namespaced	Sequences Operations on a single Unit via an SFC chart
Operation	Namespaced	Sequences Phases within a UnitProcedure via an SFC chart
Phase	Namespaced	Atomic unit of work — an SFC chart whose ST actions read/write tags and call PROMPT() for operator interaction
ProcedureTemplate	Namespaced	Reusable template cloned by the batch-operator into a Procedure at batch instantiation
UnitProcedureTemplate	Namespaced	Reusable template that the batch-operator clones into a UnitProcedure at batch instantiation. spec.category (equipment / process) gates whether the template is eligible for ad-hoc execution outside a Batch — see Ad-Hoc Execution
OperationTemplate	Namespaced	Reusable template cloned into an Operation. Same spec.category gating as UnitProcedureTemplate
PhaseTemplate	Namespaced	Reusable template cloned into a Phase. Declares its CM-role requirements via spec.cmRoles — validated against the target Unit at batch instantiation
SyncBarrier	Namespaced	Coordinates multiple phases running across different units — they must all reach the barrier before any can proceed (ISA-88 Part 1 Clause 7.2)

Hierarchy:

flowchart TD
    Procedure --> UP[UnitProcedure]
    UP --> Op[Operation]
    Op --> Phase[Phase<br/>executes logic on unit runtime]

recipe.dcs.io/v1alpha1

Ready-to-execute recipe templates and the immutable per-batch copies cloned from them.

CRD	Scope	Description
MasterRecipe	Namespaced	Ready-to-execute recipe targeting specific equipment
ControlRecipe	Namespaced	Immutable runtime copy cloned from a MasterRecipe by the batch-operator when a Batch is created
RecipeRevision	Namespaced	Immutable snapshot of a MasterRecipe created on every approval. Provides version history and rollback per 21 CFR Part 11 § 11.10(k)

See Recipes for the authoring and approval lifecycle.

Recipe flow:

flowchart TD
    MR[MasterRecipe<br/>targets specific units] --> CR[ControlRecipe<br/>created per batch, immutable]

batch.dcs.io/v1alpha1

CRD	Scope	Description
Batch	Namespaced	A single execution of a MasterRecipe
BatchRecord	Namespaced	Auto-created consolidated production record when a Batch reaches a terminal phase (Complete, Failed, Aborted). Reviewed and finalized with an electronic signature per 21 CFR Part 11 § 11.50/11.70

Batch execution flow:

1. A user creates a Batch referencing a MasterRecipe (via the HMI, CLI, or API).
2. The batch-operator's unit allocator reserves the Units required by the recipe's equipment requirements.
3. The recipe instantiator clones the MasterRecipe's templates into a full procedural tree (Procedure → UnitProcedure → Operation → Phase), all within the Batch's namespace.
4. The Batch then waits in Pending until an operator issues the Start command (the Start button in the HMI, dcs command Batch <name> Start, or a dcs.io/command annotation). That command is the only thing that starts a batch. spec.scheduledStartTime is the planned start and does not run anything, so a batch whose planned time has passed still waits here (ADR 0052). Two fields describe this pre-Start moment: the Batch's status.phase is Pending while its procedural tree's ISA-88 status.state is Idle. The HMI shows the ISA-88 state, so "Idle" there and "Pending" here name the same moment. Start cascades down the tree through the ISA-88 state machine. Every state transition emits an AuditRecord.
5. When the Batch reaches a terminal phase, the BatchRecord reconciler creates a pending-review BatchRecord aggregating the batch's state history, phase history, and operator actions. The /data endpoint's Batch Records view lists these for quality-review sign-off.

control.dcs.io/v1alpha1

Models IEC 61131-3 control programs for cyclic-scan function block execution.

CRD	Scope	Description
ControlProgram	Namespaced	IEC 61131-3 control program — FBD (blocks + connections) or ST (structured text), with variable declarations for tag binding
FunctionBlockType	Cluster	Composite function block type (IEC 61131-3) — a reusable sub-network of blocks with declared input/output ports, referenced by name from ControlProgram spec.blocks. The unit runtime flattens one when it loads the program (ADR 0065): the inner blocks are spliced into the scan at the instance's position under a /-joined path name such as loop/ctrl, so a composite costs nothing at scan time and every name-addressed surface reaches an inner block by its path. A composite declares its parameters in spec.parameters, and an inner block reads one as {{.params.<name>}}; the runtime substitutes the instance's value while it flattens, so one type serves any number of devices (#1660)
CoordinationBlock	Namespaced	ISA-88 coordination control (Part 1 Clause 7.2) — signal publish/subscribe between units, where a producer raises a named signal and its consumers wait for it. Reconciled by the batch-operator

ControlModules reference ControlModuleTemplates, which the control-operator compiles into ControlPrograms and deploys via HTTP to the corresponding unit runtime. Updates use hot-swap (see below). (ProcessException was purged in #863. The live exception path is AlarmDefinition.spec.exceptionAction. CoordinationBlock's Lock and ExceptionPropagation modes were purged in #862. Signal is the one shipped mode.)

alarm.dcs.io/v1alpha1

ISA-18.2 compliant alarm management with a four-state lifecycle.

CRD	Scope	Description
AlarmDefinition	Namespaced	Declarative condition (state-equals or tag threshold high/low/deviation) attached to a ControlModule, Unit, or IOModule. May carry an exceptionAction (Hold/Stop/Abort) that fires automatically when the condition is met
Alarm	Namespaced	Alarm instance with ISA-18.2 state model (Active/Cleared × Unacknowledged/Acknowledged). Created by the alarm generator when an AlarmDefinition's condition fires, or from IOModule / Unit status changes

Four reconcilers cover this group, all hosted in the physical-operator binary: the Alarm reconciler (watches for dcs.io/command=Acknowledge annotations), the AlarmGenerator reconciler (auto-creates Alarms from equipment status), the AlarmDefinition reconciler (evaluates conditions against live tag values), and the ServerAlarm reconciler (annunciates node readiness, node resource pressure, and lost etcd quorum as ISA-18.2 alarms, fanned out one Alarm per site namespace, per #1170 and ADR 0032). All state transitions are published to MQTT. See Alarm Management for details.

audit.dcs.io/v1alpha1

21 CFR Part 11 compliant audit trail records.

CRD	Scope	Description
AuditRecord	Namespaced	Immutable record of a user action, with electronic signature support

Audit records capture who, what, when, and why for all significant actions in the control system. See Security Operations for details.

changecontrol.dcs.io/v1alpha1

21 CFR Part 11 §11.10(k) change control for mutations to change-controlled resources (ControlProgram, ControlModule) in production namespaces.

CRD	Scope	Description
ChangeControlPolicy	Cluster	Allowlist of ServiceAccounts that may mutate change-controlled resources in namespaces labelled dcs.io/production=true, plus the external-reference and e-signature annotation requirements each mutation must carry
ChangeRequest	Namespaced	Proposed mutation to a change-controlled resource, routed through one of three backends (directApply, inClusterQueue, gitPR) for review and apply. Carries the proposed JSON patch, e-signature, external reference, and reviewer attribution. Reconciled by the changecontrol controller hosted in the batch-operator binary

Two enforcement surfaces work together:

• Admission webhook (hosted in batch-operator) — gates direct writes to change-controlled resources against the active ChangeControlPolicy, rejecting mutations that lack required annotations or originate from a non-allowlisted ServiceAccount.
• ValidatingAdmissionPolicy primitives (issue #288) — the chart ships ValidatingAdmissionPolicy + ValidatingAdmissionPolicyBinding resources that enforce GitOps ownership labels (dcs.io/owned-by=flux, dcs.io/owned-by=engineering) on engineering-class CRs. CEL expressions reject writes from the wrong ownership class without webhook round-trips. See GitOps Enforcement.

The ChangeRequest reconciler dispatches to one of three backends based on the policy's spec.backend:

Backend	Behavior
directApply	Applies the proposed patch directly once approval annotations are present. Used for low-risk environments and break-glass changes.
inClusterQueue	Holds the request in a queue for asynchronous review; an approver acks via UI or dcs changerequest approve.
gitPR	Opens a pull request against a configured GitOps repo; the change lands when the PR is merged and Flux reconciles. Closes the loop with the audit-flux-bridge for SHA correlation.

See Change Control and ChangeRequest Backends for the full workflow.

Operators

Operator binaries reconcile the CRDs, plus two kinds of pods that run on controller devices: a per-Unit runtime pod and a per-Controller io-probe pod. Each binary hosts multiple reconcilers. The grouping below is by binary, which cuts across the API groups.

physical-operator

• Reconciles: Enterprise, Site, Area, ProcessCell, Unit, ControlModule, Controller, IOModule, NodeIdentity, NodeMaintenance, SiteOutage, plus the alarm group (Alarm, AlarmGenerator, AlarmDefinition, ServerAlarm)
• Smart vendor-packaged devices (PLCs, MTP skids) are modeled as Units with spec.serviceBinding. A service is either a raw OPC UA Method or a declared MTP module service carrying its ServiceControl addresses (ADR-0045, with the VDI 2658-4 state/command vocabulary in pkg/mtpservice). Phase ST/SFC bodies dispatch Method services through the CALL_SERVICE builtin (internal/controller/procedural/call_service.go) and drive MTP module services through MTP_COMMAND / MTP_STATE / MTP_COMMAND_ENABLED (internal/controller/procedural/mtp_command.go), which hold one OPC UA session across the phase run. The physical operator does not run a per-device OPC UA session. See ADR-0001 for the rationale and the Asset dissolution migration guide for how the previous Asset CRD mapped to this shape.
• Creates site-{name} namespaces when a Site resource is created
• Replicates Secrets labelled dcs.io/replicate-to-sites=true from the release namespace into each site-{name} namespace and patches the namespace's default ServiceAccount with replicated dockerconfigjson Secrets as imagePullSecrets. Operator-created pods (io-probe, unit-runtime) can then pull private images and mount MQTT TLS / auth Secrets without per-site copying. Configurable via the siteSecretReplication values block (issue #306).
• Manages the unit runtime pod lifecycle — creates/deletes runtime pods for each Unit, copying nodeSelector to pin pods to device nodes
• Manages the io-probe pod lifecycle — creates a per-Controller io-probe pod when IOModules reference it, providing infrastructure-level health monitoring independent of the equipment hierarchy
• Reconciles Controller resources (Pending → Joining → Joined): binds each Controller to the cluster node of the same name, converges spec.nodeLabels (dcs.io/ prefix) onto the bound node, and auto-creates Controllers for nodes labeled dcs.io/site=<site> (node discovery). Node join is the deployment layer's job (ADR 0004). Simulation controllers (spec.simulation: true) create and maintain a virtual Kubernetes Node and heartbeat Lease, with no physical node to wait for
• Processes RequestedState in Unit status (set by the procedural layer) and dcs.io/command annotations to trigger state transitions on Units
• Uses the ActiveWork status field on Units for orphan detection — a Running unit with no ActiveWork is auto-stopped back to Idle

procedural-operator

• Reconciles: Procedure, UnitProcedure, Operation, Phase, SyncBarrier, plus the recipe group (MasterRecipe, ControlRecipe)
• Cascades Start / Stop / Hold / Restart commands down the procedural tree via RequestedState generations and dcs.io/command annotations
• Phase controller runs the SFC engine, evaluates ST transition conditions, and publishes live step/variable snapshots to Phase status
• Phase controller is resilient to runtime outages: the first time the unit runtime is unreachable in a run, Status.RuntimeUnreachableSince is stamped and the reconciler requeues with backoff (transient blips like pod restarts or cert refreshes resolve without disturbing the phase). Outages exceeding a grace window cause a clean self-hold with audit and alarm. A self-hold records exactly one AuditRecord on the transition into the held condition: while a phase keeps failing identically and bounces Running ↔ Held every reconcile, the persisted Status.SelfHoldSignature suppresses duplicate records (a changed error is audited afresh), so an idle-ish failing batch no longer grows etcd without bound. As a safety net, a stuck-phase watchdog stamps Status.LastProgressAt on every observable progress event (state transition, SFC step change, successful runtime poll). A phase that stays in an active state with no progress for the watchdog threshold is force-aborted, converting silent stalls into loud, audited failures so the parent batch is never blocked indefinitely. A running ActionChart blocks its reconcile for the whole chart (a timed charge step legitimately dwells for tens of minutes), so during execution the SFC status publisher's one-second tick is what keeps the stamp alive. LastProgressAt is therefore a high-water mark: the reconciler's own copy is a snapshot from before the chart started, and writing it back would report a healthy dwell as a stall exactly as long as the dwell itself, so no writer may move the stamp backwards.
• A publisher tick proves the engine is alive. The chart advancing is a separate claim, and the two fail differently. Because reconciles for one Phase are serialised, a chart that wedges blocks its own reconcile, so the stuck-phase watchdog is never even evaluated while the wedge lasts. The publisher is the only thing still running. It therefore also watches the engine's ProgressCount, which counts scan cycles that evaluated a transition condition or changed a step. A timed dwell advances it every scan (the guard is re-evaluated continuously) while a chart whose step is blocked in an action that never returns stops it dead. The other wedge shape, a guard erroring every scan, is ended by the engine's own escalation after two minutes and so never reaches this five-minute grace. That is the point of the ordering, because that failure carries a cause worth reporting and this one does not. Past the grace the phase records Status.EngineStalledSince, raises a dedicated engine-stall alarm, and audits the diagnosis once, non-destructively, because "the engine stopped advancing" is a diagnosis and not a verdict on the batch. A chart that then runs out its timeoutSeconds names the stall in its report, and the report is more than a bare deadline. Sanctioned indefinite waits (an outstanding operator prompt, a held engine) count as advance. Sync-barrier and coordination waits park the phase in Held, where no chart is running at all.
• A phase whose runtime dies mid-Holding cannot run its HoldingChart at all, so it settles Held with its declared posture uncommanded (#1387) and records that in Status.HoldingActionDeferredSince with its own hold-deferred alarm. The reconnect poll then commands the deferred holding action and re-arms the phase chart at the edge, driving the run against a state machine positioned in Holding so the interrupted transition is what completes. ISA-88 makes Held a waiting state that directs no actions (ADR 0048). Recovery still takes an explicit Restart.
• MasterRecipe controller writes an immutable RecipeRevision on every generation bump

batch-operator

• Reconciles: Batch, BatchRecord, plus the changecontrol group (ChangeRequest) and the control group's CoordinationBlock
• On Batch creation: allocates required Units, clones the referenced MasterRecipe's templates into a full procedural tree, and parks the Batch in Pending until an operator issues Start (via a dcs.io/command annotation)
• On Batch terminal phase: creates a pending-review BatchRecord aggregating the batch's state history, operator actions, and audit trail
• Change-control admission webhook: gates writes to change-controlled resources in production namespaces against the active ChangeControlPolicy
• ChangeRequest reconciler: dispatches proposed mutations to one of three pluggable backends: directApply, inClusterQueue, or gitPR (issue #289). The gitPR backend opens a PR against a GitOps repo. Combined with the audit-flux-bridge (below), the resulting Flux reconciliation creates an AuditRecord with the merge commit SHA attached

control-operator

• Reconciles: ControlProgram, ControlModule
• ControlProgram deployment: watches ControlProgram CRs, resolves spec.unitRef → Unit → runtime pod IP, and deploys the program definition to the runtime via HTTP. Supports initial deploy and hot-swap replace (generation-based detection).
• ControlModule template compilation: compiles ControlModuleTemplate + parameter overrides into ControlProgram CRs for cyclic scan control logic. Template-defined tags (e.g., PV, CV) are merged with instance tags into status.effectiveTags. Instances therefore inherit standard tags automatically. Tags may carry per-tag engMin/engMax engineering ranges (literal values or block parameter references like "read_pv.engMin") and per-tag trueLabel/falseLabel for Boolean tags (overriding CM-level parameters["true_label"]/parameters["false_label"]). The compiler resolves a block parameter reference to a literal, which is what lets the unit controller carry the range into the runtime's tag map and the runtime refuse a write outside it (ADR 0050). The compiler also checks I/O directions: an input block (DI, AI) must not land on an address the template declares as an output, and an output block (DO, AO) must not land on one it declares as an input. The one exemption is a read-back. An output port carrying readBack: true says its device point can also be read. That licenses an input block whose address is written as {{.outputs.<name>}} and nothing else, so one point keeps one binding. An instance that aliases an independent input port onto the driven address is still refused (ADR 0066).
• Sets Deployed condition and tracks execution state in ControlProgram status.

Hot-swap policy

Hot-swap is the spec-replace path taken when the runtime is running a program whose content no longer matches what the control-operator computes for it. Two changes reach it. The ControlProgram's own generation can advance, which is the direct edit. Or the content resolved into the program can drift: a composite FunctionBlockType it uses is edited or deleted, which moves the computed spec hash while the ControlProgram itself is untouched (#1574). Both rewire function-block state on a running runtime, so both are disruptive to any batch executing against that unit. The control-operator gates hot-swap according to batch state, and the cause makes no difference to the gate:

• Blocked: if any Batch in the ControlProgram's namespace is in phase Running, Holding, Held, or Aborting and has the program's unit in its status.allocatedUnits, the replace is deferred. The controller sets a HotSwapDeferred status condition (reason BatchActive) naming the blocking batch and which of the two causes made the logic stale, emits a Rejected AuditRecord correlated to the batch, and requeues every 30 seconds. A watch on Batches re-enqueues the ControlProgram the moment the blocking batch leaves its active phase.
• Allowed via annotation bypass: an operator can override the gate by annotating the ControlProgram with both dcs.io/allow-hot-swap=true and a non-empty dcs.io/hot-swap-reason. The trimmed reason is written into the resulting Update AuditRecord so the override is permanently attributable for 21 CFR Part 11 purposes. Neither annotation alone satisfies the bypass. The bypass releases a drift-driven swap exactly as it releases a generation-driven one.
• Pod-recovery is never gated: the deploy path (deployToRuntime) fires when the runtime does not hold the program at all, which is what the verify probe reports as NotFound, as an empty state, or as an unreachable runtime. It runs unconditionally, because it is the safety net that restores control logic to a freshly restarted runtime pod and the alternative is a unit left with no logic. Audit records from this path carry reason=pod-recovery when the program was previously deployed, reason=initial-deploy otherwise.

A successful replace records reason=generation change or reason=content drift, alongside the before and after spec hashes. The Runtime.OverrideSnapshot / ApplyOverrides machinery in pkg/fbruntime/runtime.go carries block state across a replace, so a swap does not reset the live values a batch has already written.

Minimal annotated override example:

UIYAML

/system → ControlProgram detail → Hot-Swap Override opens a modal that writes both annotations and an AuditRecord under the operator's identity. The reason is required and written verbatim to the audit trail under 21 CFR Part 11 §11.10(k).

apiVersion: control.dcs.io/v1alpha1
kind: ControlProgram
metadata:
  name: reactor-pid
  namespace: site-demo
  annotations:
    # Both annotations are required. allow-hot-swap on its own is ignored.
    dcs.io/allow-hot-swap: "true"
    dcs.io/hot-swap-reason: "CAPA-2026-04-12: PID Ki reduced to damp oscillation observed during batch-7421"
spec:
  unitRef: reactor-a
  scanInterval: 100ms
  # ...

Clearing the HotSwapDeferred condition manually is not required. The controller removes it on the next successful reconcile once the blocking batch clears or a valid bypass is in place.

Auxiliary Binaries

The operators above reconcile CRDs. The system also ships auxiliary binaries that are not reconcilers but are part of the running deployment.

• historian (cmd/historian/) — MQTT-to-PostgreSQL collector. Subscribes to state transitions, alarm events, and control-logic tag snapshots published by unit runtimes and operators, and writes them to a PostgreSQL database for long-term trending and compliance retention. See Historian and MQTT Telemetry.
• audit-archiver (cmd/audit-archiver/) — Kubernetes CronJob. Moves AuditRecord CRs older than a configurable retention window from etcd into the historian PostgreSQL database, satisfying 21 CFR Part 11 §11.10(c) long-term retention while keeping etcd bounded. Archival is off by default (audit.archival.enabled). When it is switched on, the CronJob runs hourly, at :17 to dodge the top-of-hour stampede (#313).
• audit-flux-bridge (cmd/audit-flux-bridge/) — Flux notification receiver. Subscribes to flux.toolkit.fluxcd.io/v1beta3 Provider + Alert events and emits an AuditRecord for each Flux reconciliation of an engineering CR, attaching the source Git commit SHA for full 21 CFR Part 11 §11.10(k) traceability of GitOps-driven changes. HMAC-signed webhook authentication via a shared secret in both dcs-system and flux-system namespaces. See Audit Linkage.
• io-probe (cmd/io-probe/) — Probes each referenced IOModule's fieldbus link (Modbus TCP, EtherNet/IP, OPC UA) and publishes infrastructure-level connectivity health independent of the equipment hierarchy. Two flavors: a per-Controller pod (node-pinned, owned by the Controller) for IOModules that name a controllerRef, and a namespace-shared network-io-probe pod (node-unpinned by default, no owner) for controller-less network IOModules (ADR 0021). On zoned networks a module's spec.probePlacement.nodeSelector pins its shared probe to the nodes with field reach, one pod per distinct selector (ADR 0042). Whichever flavor serves a module, the pod's name is published on that module's status.probePodName. A reader routing a request to the probe takes the published name and never re-derives it (#1480). The physical-operator manages both lifecycles: the ControllerReconciler the former, the IOModuleReconciler the latter. Gateway proxies to io-probe over TLS when configured (#307).
• dcs CLI (cmd/dcs/) — Operator-facing CLI. Covers batch, recipe, equipment, audit verification, historian queries, tag simulation, diagnostics, change-control approval workflows, and a laptop-mode embedded gateway (dcs ui, #286). Preferred over kubectl for day-to-day operator tasks (see CLI Reference): it is gateway-mediated, so it carries a persona, a Site scope, and an audit trail that raw kubectl does not. Which humans hold Kubernetes credentials at all, and why only the platform engineer needs them, is mapped in Security Hardening § Personas, Surfaces, and Privilege Tiers.

State Machine

All stateful resources (Unit, Phase, Operation, UnitProcedure, Procedure, Batch) use the ISA-88 state machine.

States (13)

stateDiagram-v2
    [*] --> Idle

    Idle --> Running: Start

    Running --> Complete: SC
    Running --> Holding: Hold
    Running --> Stopping: Stop
    Running --> Aborting: Abort
    Running --> Pausing: Pause

    Pausing --> Paused: SC
    Pausing --> Stopping: Stop
    Pausing --> Holding: Hold
    Pausing --> Aborting: Abort

    Paused --> Running: Resume
    Paused --> Stopping: Stop
    Paused --> Holding: Hold
    Paused --> Aborting: Abort

    Holding --> Held: SC
    Holding --> Stopping: Stop
    Holding --> Aborting: Abort

    Held --> Restarting: Restart
    Held --> Stopping: Stop
    Held --> Aborting: Abort

    Restarting --> Running: SC
    Restarting --> Stopping: Stop
    Restarting --> Holding: Hold
    Restarting --> Aborting: Abort

    Complete --> Resetting: Reset

    Stopping --> Stopped: SC
    Stopping --> Aborting: Abort

    Stopped --> Resetting: Reset
    Stopped --> Aborting: Abort

    Aborting --> Aborted: SC

    Aborted --> Resetting: Reset

    Resetting --> Idle: SC

The 13 states comprise seven resting states (Idle, Running, Complete, Paused, Held, Stopped, Aborted, of which Running is a stable executing state and the rest are terminal or quiescent) and six transitional states (Pausing, Holding, Restarting, Stopping, Aborting, Resetting). Transitional states execute platform-defined actions and then auto-advance via the internal SC (State Complete) transition. Resting states wait for an operator or controller command. Note that Start moves Idle directly to Running. There is no transitional Starting state.

Commands (8 + StateComplete)

Command	Valid From	Transitions To
Start	Idle	Running
Stop	Running, Pausing, Paused, Holding, Held, Restarting	Stopping
Hold	Running, Pausing, Paused, Restarting	Holding
Restart	Held	Restarting
Abort	Running, Pausing, Paused, Holding, Held, Restarting, Stopping, Stopped	Aborting
Pause	Running	Pausing
Resume	Paused	Running
Reset	Complete, Stopped, Aborted	Resetting
SC (StateComplete)	Running, Pausing, Holding, Restarting, Stopping, Aborting, Resetting	Complete, Paused, Held, Running, Stopped, Aborted, Idle (respectively)

See Phases → The ISA-88 state machine wrapping for a per-state walkthrough of what each state means for a running SFC chart and which commands an operator can issue from it.

The state machine is thread-safe, supports callbacks on transitions, and maintains a history of all state changes. Commands reach equipment in two ways:

1. RequestedState (structural): The procedural layer writes a RequestedState field in the Unit's status subresource with a monotonic generation counter. The Unit controller processes new generations idempotently. This path also sets ActiveWork on the Unit, providing a persistent binding between the Unit and the procedural element executing on it.
2. dcs.io/command annotation (legacy/manual): Controllers and the gateway set the annotation on the target resource. This path remains for backward compatibility and manual operator commands.

Mode Management (ISA-88 Clause 7.3.2)

ISA-88 defines modes for procedural elements and equipment entities:

Entity Type	Modes	Behavior
Procedural (Procedure, UnitProcedure, Operation, Phase)	Automatic, Semi-Automatic, Manual	Controls how step transitions occur
Equipment (Unit, ControlModule)	Automatic, Manual	Controls whether control algorithms may manipulate the equipment

Automatic (default): Transitions occur without interruption when conditions are met.

Semi-Automatic: Transitions require operator confirmation. When a child step completes, the parent auto-pauses (Running → Pausing) and waits for a Resume command before starting the next step.

Manual: Steps are not auto-started. The operator must issue Start commands to each child explicitly and may execute steps in any order.

Mode is set via spec.mode (desired) and reflected in status.mode (observed). Runtime mode changes use the dcs.io/mode annotation, processed by controllers alongside commands.

Execution Model and Timing Guarantees

Cloud-Native DCS runs control logic on a cyclic-scan model that matches traditional PLC and DCS expectations, but it is hosted on Linux pods under Kubernetes, with no real-time microcontroller underneath. Engineers coming from hard real-time environments should calibrate their expectations here before deploying against fast processes.

Scan rates

Layer	Default rate	Configurable
Control Module — discrete-valve, solenoid-valve, analog-control	200 ms	via template scanInterval
Control Module — analog-sensor	500 ms	via template scanInterval
Control Module — pid-loop, pid-cascade, vfd	100 ms	via template scanInterval
SFC transition evaluation (phase logic)	Tied to the unit runtime's FB scan tick — each transition guard is evaluated once per scan cycle while the phase is in the Running state
Audit record write path	Synchronous with the action that generated it — not buffered
MQTT telemetry publish	Batched per scan cycle, store-and-forward queue on broker loss

Jitter and worst-case expectations

The unit runtime is a Go process scheduled as a Kubernetes pod with hostNetwork: true. Scan-cycle jitter depends on:

• Linux kernel scheduler latency (tens of microseconds to low milliseconds under normal load, spiking under memory pressure or noisy-neighbour CPU contention).
• Go garbage collector pauses (sub-millisecond to a few milliseconds on the current runtime heap sizes).
• Driver round-trip time to field I/O — Modbus TCP on a quiet LAN is typically 1–10 ms per read, EtherNet/IP assembly-object reads are similar, and OPC UA subscriptions amortise across the publish interval.
• The node-level PodDisruptionBudget and resource limits configured in Helm values.

Customers with hard real-time requirements (sub-millisecond jitter, guaranteed worst-case) should not rely on the Kubernetes-hosted runtime for those paths and should push that logic into a dedicated real-time controller or safety PLC wired as a downstream I/O consumer.

What this means for phase logic

• A transition guard that turns true between scan ticks will fire on the next tick. For a 200 ms-scanned discrete valve, expect up to 200 ms between "valve reports OPEN" and "SFC advances to the next step".
• TON / TOF / TP timers in the function-block runtime are scan-discretized. A T#100ms timer on a 200 ms scan layer effectively rounds up to the next tick.
• Operator prompts (PROMPT() calls in ST) block the SFC until the operator acknowledges and so inherit human-scale latency regardless of scan rate. AWAIT_RESULT() blocks on a laboratory. A laboratory's latency runs to hours routinely, which is why the step is required to declare its own timeout (ADR 0055).

Not a real-time operating system

The Kubernetes node kernel is a standard Linux kernel without PREEMPT_RT. Worst-case execution time is bounded by the Go runtime and kernel scheduler, with no formal WCET analysis behind it. This is appropriate for batch process control, where seconds-scale transitions are the norm. It is not appropriate for motion control, safety shutdown, or any function with a sub-10-ms response requirement.

See also High Availability and Failure Modes for failure-mode analysis and Capacity Planning for sizing guidance.

Unit Runtime

The unit runtime is a per-unit pod deployed by the physical operator. It bridges the Kubernetes control plane and physical hardware.

flowchart TD
    subgraph Pod["Unit Runtime Pod"]
        HTTP["HTTP API<br/>:61152"] --> FB["FB Runtime<br/>IEC 61131-3 scan"]
        FB --> Router["Address Router<br/>addr → default<br/>mod:addr → IO module"]
        Router --> Default["Default Driver<br/>modbus / opcua / sim / ethernetip"]
        Router --> IOD["IO Module Drivers<br/>from ConfigMap"]
    end
    Default --> Local["Modbus TCP<br/>or OPC UA"]
    IOD --> Remote["Remote I/O modules<br/>EtherNet/IP, Modbus"]

HTTP API

The unit-runtime exposes a JSON HTTP API consumed by the control-operator, procedural-operator, and gateway. All routes except /healthz require a Bearer token when --api-token is set, and use mTLS when the server is configured with a TLS cert/key/CA bundle (IEC 62443 SR 4.1).

Endpoint	Method	Description
/api/v1/controlprograms	GET	List all deployed ControlPrograms (map keyed by program name)
/api/v1/controlprograms/{name}	POST	Deploy a ControlProgram (replaces if already present)
/api/v1/controlprograms/{name}	GET	Get one ControlProgram's status (state, message, outputs, inputs, scanCount, specHash)
/api/v1/controlprograms/{name}	DELETE	Stop and remove a deployed ControlProgram
/api/v1/controlprograms/{name}/replace	POST	Hot-swap a running ControlProgram with a new definition
/api/v1/controlprograms/{name}/blocks/{block}/bypass	POST	Install a time-boxed interlock bypass on an output block, returning the computed deadline so the gateway can audit it (ADR 0010)
/api/v1/controlprograms/{name}/blocks/{block}/bypass	DELETE	Clear an interlock bypass early
/api/v1/read	POST	Read a process value ({address} → {address,value,quality,timestamp})
/api/v1/read-batch	POST	Read many addresses in one call; a per-address failure is reported inline (quality Bad) and the batch call still succeeds
/api/v1/write	POST	Write a process value ({address,value} → {success})
/api/v1/release	POST	Clear a simulation write override so the module's configured behaviors resume driving the channel (#990)
/api/v1/status	GET	Runtime status snapshot (protocol, connected, drivers, healthStatus)
/api/v1/tags	GET	Enumerate known tag addresses with cached values
/api/v1/tagmap	POST	Replace the in-memory CM tag map (pushed by the physical-operator when the tag map ConfigMap changes)
/api/v1/ioconfig	POST	Reconcile the live IO drivers against a fresh IOModule config set (pushed by the physical-operator when the iomodules.json ConfigMap changes)
/api/v1/diagnostics	GET	Driver + runtime diagnostics summary
/api/v1/lease/renew	POST	Record a control-lease grant from the physical-operator (ADR 0006, Failover mode). An equal-or-newer epoch renews the lease and unfences the runtime; a stale epoch gets 409, and Autonomy mode gets 400 because there is no lease to renew
/api/v1/hold/arm	POST	Arm an edge-local hold chart — the running phase's holdingChart, or the unit's safeStateChart baseline (ADR 0008)
/api/v1/hold/disarm	POST	Disarm an armed hold chart
/api/v1/hold/release	POST	Release a self-hold on ISA-88 Restart
/api/v1/heartbeat	POST	Control-plane liveness heartbeat; silence past availability.holdGraceSeconds is what fires the armed chart in Autonomy mode
/api/v1/stop/arm	POST	Declare that the next stop of this runtime ends it, so the field is safed on the way down and never frozen (#1283). Armed explicitly because the kubelet sends SIGTERM and nothing in the chain can be made to send anything else
/api/v1/stop/disarm	POST	Withdraw a terminal-stop arming
/api/v1/stop	GET	Read the current terminal-stop arming state
/healthz	GET	Health check (always unauthenticated)

ControlProgram deploy request body:

{
  "blocks":            [{"name": "...", "type": "...", "params": {...}}],
  "connections":       [{"sourceBlock": "...", "sourcePort": "...", "destBlock": "...", "destPort": "..."}],
  "varBindings":       [{"variable": "...", "blockRef": "...", "portRef": "..."}],
  "compositeTypes":    [{"typeName": "...", "dataInputs": [...], "dataOutputs": [...], "parameters": [...], "blocks": [...], "connections": [...]}],
  "interval":          "100ms",
  "specHash":          "<sha256 of canonical request>",
  "controlModuleRef":  "tank-1",
  "mode":              "continuous"
}

controlModuleRef is identity metadata (not hashed). Changes to it do not trigger redeploys. specHash is computed by the controller over the deploy-request JSON and is compared against the runtime's stored hash to detect stale content.

compositeTypes carries the composite FunctionBlockTypes the program's blocks instantiate, resolved cluster-scoped by the control operator and recursed to a nesting cap of 8. Each definition carries its declared parameters, because the runtime substitutes an instance's values into the inner blocks while it flattens (#1660). The runtime flattens them away in Load, before any block is instantiated (ADR 0065). Nothing downstream of that call ever sees a composite. The definitions enter specHash, which is what makes an edit to a block type redeploy every program resolving it.

Between the IEC 61499 to 61131-3 cutover and #1658 this field was sent by the control operator and read by nothing: the runtime's request struct did not declare it, so the bundle was dropped on decode and the composite type name left in blocks[].type resolved in no registry. A ControlProgram naming a composite was answered 400 unknown block type and reported as Deployed=False. The gateway now runs the same flattening the runtime does when a ControlProgram is created or updated, so a program the runtime could not load is refused where it is written.

IEC 61131-3 Phase 3 cutover (post-#361)

The HTTP API and persisted state file were renamed in the Phase 3 cutover:

Aspect	Pre-Phase-3 (legacy)	Post-Phase-3 (current)
Collection route	/api/v1/fbnetworks	/api/v1/controlprograms
Item route	/api/v1/fbnetwork/{name}	/api/v1/controlprograms/{name}
Replace route	/api/v1/fbnetwork/{name}/replace	/api/v1/controlprograms/{name}/replace
Legacy oneshot	POST /api/v1/fbnetwork, /status, /stop	removed
Wire field	dataConns	connections
HostPath state file	last-network.json	last-program.json

Migration: the cutover is a hard switch across the runtime and all in-cluster callers (control-operator, procedural-operator, gateway). To upgrade a fleet:

1. Roll the new images for dcs-control-operator, dcs-procedural-operator, dcs-gateway, and dcs-unit-runtime together. The new operators only speak the new routes, and the new runtime only serves the new routes.
2. On the first reconcile after upgrade, every ControlProgram's specHash will differ (the canonical JSON now uses connections), so the control-operator will redeploy each program once. This is expected and leaves the runtime on the new wire shape.
3. The unit-runtime reads last-program.json on startup. If absent, it falls back to last-network.json once so the device node retains local autonomy across the upgrade. The next SaveNetwork writes the new filename. After the first redeploy, the legacy file can be deleted (the runtime never reads it again).

Function Block Runtime (IEC 61131-3)

The FB runtime implements an IEC 61131-3 cyclic scan execution engine. All blocks execute in declared order each scan cycle.

I/O blocks: - DI (alias IX): Digital input — calls driver.ReadValue(address) and coerces the result to bool via toBool(), guaranteeing OUT is always a boolean - DO (alias QX): Digital output — coerces the bus input to bool via toBool(), calls driver.WriteValue(address, value), and exposes the coerced value on OUT (always a boolean) - AI: Analog input — reads a value via driver.ReadValue(address) and applies linear scaling (rawMin/rawMax → engMin/engMax). When engMin/engMax or rawMin/rawMax are omitted, the block queries the driver's TagMetadataProvider interface on first execution to auto-discover the correct ranges. Explicit params override auto-discovery. Each range is declared whole or left entirely to discovery: engMin without engMax is refused at Init, and rawMin without rawMax is refused the way the AO block already refuses it, since inventing the other bound would leave the runtime enforcing an interval the document does not declare. For simulation Expr behaviors (which store engineering values directly), the driver reports RawMin=EngMin, RawMax=EngMax, producing identity scaling. - AO: Analog output — clamps the bus input to outMin/outMax (engineering units) and converts it to the device's raw range (rawMin/rawMax) at the write. Explicit params override driver TagMetadataProvider auto-discovery. With neither, the value passes through unconverted (identity), and that is what simulation drivers expect. Giving rawMin without rawMax is refused at Init, since a defaulted span would invert the range. Drivers whose target is an integer register round the converted value to the nearest count. The OUT port — and therefore PID TRK_VAL back-calculation — stays in engineering units

Logic blocks: - AND, OR, NOT: Boolean logic - BOOL_CONST, REAL_CONST: Constant value sources (boolean, floating-point) - GT, LT, EQ: Comparison operators - ADD, SUB, MUL, DIV, MOD: Arithmetic operators - ABS, MAX, MIN: Magnitude and selection arithmetic - SEL: Multiplexer (select between inputs) - PID: Proportional-integral-derivative controller - DEVSTATE: Device-state priority encoder (ADR 0018) — Boolean inputs IN1…IN8 evaluated in order, params state1…state8 name the word for each, first true wins, default when none, and OUT is a string. Backs a template's derived role: state tag. Its output is read-only at every layer: the tag is accessLevel: read and the runtime refuses output overrides on the block (ReadOnlyOutputBlock)

Stateful blocks: - TON, TOF, TP: On-delay, off-delay, and pulse timers - CTU, CTD: Up and down counters - SR, RS: Set-dominant and reset-dominant flip-flops - R_TRIG, F_TRIG: Rising and falling edge detectors

Execution model: All programs run as cyclic scans at a configurable interval (e.g., 100ms). Each cycle executes all blocks in the declared order, reads inputs from data connections, and writes outputs. There is no event-driven scheduling.

Connection validation: every endpoint of every data connection is resolved when the program loads. A program carrying a dangling one is refused, and never reaches the scan. Both block names have been checked since the runtime was written. The source port is checked against the source block's Outputs() (#1650) and the destination port against the destination block's InputPorts() (#1655). The runtime is the layer that answers this, because it holds the instantiated block and can ask it. A connection that resolves to nothing is silent at run time, so a refusal at load is the only place it surfaces. See ADR 0064 and its amendments for what each half costs.

Parameter validation: a parameter a document does not set takes the block's documented default, and so does one set to an empty string, because a template declares a placeholder that way for an instance to fill in and pkg/templatecompiler drops a reference that resolved to nothing. A parameter set to a value that does not parse is refused at load (#1663). The two used to be answered the same way for the engineering ranges and the PID gains. A mistyped bound or gain became the default with no log line, no condition and no faulted block, and the program reported Running on a number nobody wrote. The refusal names the block type and the parameter, since the load reports only the block name from the document.

Deployment flow: The control-operator watches ControlProgram CRs and automatically deploys them to the unit runtime via HTTP. Updates to existing programs use hot-swap (POST /replace) for zero-downtime changes.

Protocol Drivers

Drivers implement the Driver interface:

Driver	Status	Hardware
Modbus	Implemented	Modbus TCP devices
OPC UA	Implemented	Industrial PLCs
Simulation	Implemented	No hardware (in-memory); configured inline on each IOModule via spec.simulation with optional SimulationPreset references for reusable equipment templates; fault injection and deterministic RNG
EtherNet/IP	Experimental — opt-in only	EtherNet/IP I/O modules (assembly objects); requires --enable-experimental-drivers

The Modbus driver supports configurable unitID, timeout, speed, byteOrder and wordOrder options, and refuses any other key outright. A channel address carries an optional width suffix (holding.100:float32) for values spanning more than one register. A bare address is one 16-bit register, as it has always been. The two order options say how a device lays such a value out, which is the one thing Modbus itself does not specify (see 32-bit and wider values). It auto-reconnects on read/write failure: if a TCP connection drops, the driver transparently re-establishes it and retries the operation once. If the device is unreachable at startup, the runtime starts in a degraded state and the driver lazily reconnects on the first successful I/O attempt. The Modbus driver also implements the Discoverer interface for WAGO 750-series I/O systems. It reads the coupler's module identification registers to auto-detect installed modules and generate channel layouts with correct process image addresses. Analog and specialty modules report their part number. Digital modules report a bit-coded word (bit 15 set, channel count, direction bits), and both encodings are parsed. On couplers that expose no module table at all, it falls back to reading the process image size registers and creates channels from the aggregate I/O counts. Those registers report bits, including for the analog areas.

Device-specific knowledge (register maps, module catalogs) lives in device profiles, analogous to EDS files (EtherNet/IP) or GSD files (PROFINET). Each profile implements the ModbusProfile interface (Probe + Discover) and self-registers at startup. The Modbus driver iterates registered profiles to identify the connected device and delegate discovery. The first shipped profile covers WAGO 750-series couplers.

Drivers may also implement the optional TagMetadataProvider interface to expose scaling metadata (EngMin/EngMax, RawMin/RawMax) for their tag addresses. The simulation driver implements this, allowing AI and AO blocks to auto-discover the correct scaling without manual configuration on every control module. The Modbus driver implements it from device-profile discovery: channels whose card family defines a raw span (a WAGO 4-20 mA card's left-justified 0–32767) serve that span, and only the raw half. Engineering ranges are process knowledge the wire cannot carry, so blocks keep their own engineering defaults unless configured. A zero-width range is never served, and consumers ignore a zero-width half. Scaling into a constant is what that avoids. By default every simulation behavior stores engineering units and reports RawMin=EngMin, RawMax=EngMax (identity scaling), so the IOModule channel table shows believable values. A behavior that models real analog-input hardware can set raw: "true" to emit raw ADC counts instead, reporting a raw range of 0–65535 (16-bit ADC) so AI blocks perform a real raw-to-engineering conversion.

Local Autonomy

The runtime persists the last-executed control program to the host filesystem at /var/lib/dcs/runtime/{unit-name}/last-program.json. On startup, if the control plane is unreachable, the runtime replays the persisted program to maintain the last-known process state. This provides local autonomy for edge devices.

Device interlocks ride inside the control program, so they remain active through a control-plane partition: a DO/AO block with an ILCK input or interlockAddress parameter keeps evaluating its trip condition every scan and forces its safe value when tripped, with no cluster round-trip. They are the only always-on, mode-independent protection layer with this property. Phase guards and alarm exception actions normally execute control-plane-side (ADR 0007), though the procedural holding response can additionally be armed at the edge to survive a partition (see below). A first-class interlock bypass is enforced the same way. Its time-boxed deadline is evaluated in the FB scan, so it auto-expires and re-arms the interlock on the edge even during a partition. It is permission-gated (interlock:bypass) and audited at the gateway (ADR 0010).

Edge-local holding (ADR 0008) makes the procedural safe-state layer partition-tolerant. A device interlock forces a single output to a single value. It cannot sequence. The layer that runs a deliberate, ordered safe-state response (close the charge valve, then stop the agitator, then open the vent) is the phase SFC, and that normally executes in the procedural operator. On a partition it stalls mid-step, and the equipment freezes wherever the sequence happened to be. To close that gap the unit runtime embeds the pkg/sfc engine with a tag handler bound to the local control-module space, and arms a hold chart: the running phase's holdingChart (downloaded at phase start), or the UnitSpec.safeStateChart baseline when no phase is active. When the control-plane heartbeat watchdog fires (the lease renew in Failover, a lightweight liveness heartbeat past availability.holdGraceSeconds in Autonomy), the embedded engine runs the armed chart to drive a sequenced safe state by writing FB input variables. The FB scan remains the sole driver writer, and device interlocks still sit underneath. In Autonomy the runtime holds and keeps holding (it is the legitimate single writer). In Failover it holds for a bounded sequence then self-fences so a standby can re-bind cleanly (the re-bind margin is ≥ the hold duration, so there is never a dual writer). An armed chart may only READ/WRITE the unit's own control-module tag space, and may call only the builtins that work with no control plane to reach, a constraint shared by the runtime and make lint-edge-armable. A chart that prompts an operator, changes a mode, commands the phase state machine, or reaches another machine is refused at arming time and the phase falls back to the unit baseline. Hold-lifecycle events buffer onto the MQTT store-and-forward queue and materialize as AuditRecords on reconnect. On reconnect the runtime's self-held status reconciles the phase to Held. Recovery is an explicit ISA-88 Restart, and a silent resume does not exist. A truly dead node (not merely partitioned) runs no software, so that case still falls back to device fail-safe / last value (ADR 0006) until a standby re-binds.

Because interlocks (and any edge-local holding logic) ride inside the scan, the scan is kept alive through survivable faults: a single block's runtime error degrades the program, and the rest keep regulating and protecting every healthy output. Deliberate program removal drives each output to its configured fail state, with nothing left frozen: failState: safeValue (the default) writes the block's safeValue, and failState: holdLast keeps the last commanded value (IEC 62443 SR 3.6, #972). Ungraceful process death falls back to the remote-I/O module's comm-loss watchdog. ADR 0009 left that watchdog to the deployment layer, because its configuration is vendor-specific. The IOModule declares it since ADR 0068. spec.failSafe is where a node says what its outputs do with nothing of ours running. An undeclared one reports itself, and the product writes nothing to the device (#1683).

The declared timeout carries a floor of three io-probe read cadences, 45 s, refused at admission (ADR 0071). Any telegram feeds a device watchdog. The io-probe reads every device it serves once every 15 s, so a shorter deadline is reset by our own traffic before it can expire on purpose. The FailSafeTimeoutMargin condition then reports the gaps each device actually sees, because the probe loop is serial and an unreachable module costs every module behind it a dial timeout.

Telemetry Store-and-Forward

When the MQTT broker is unreachable (WAN outage, broker restart), the runtime buffers telemetry messages to a persistent on-disk queue at {DataDir}/queue/queue.jsonl. Messages are replayed in FIFO order when connectivity is restored. This prevents telemetry gaps that could trigger compliance questions in pharmaceutical environments (21 CFR Part 11).

The queue uses an append-only JSON Lines file with a separate offset tracker. On crash recovery, partial writes are safely discarded. When the queue exceeds its size limit, the oldest entries are evicted via file rotation. The --mqtt-queue-size flag controls the maximum buffered messages (default 10000, 0 disables the queue).

A runtime with the queue disabled has nowhere to put a message the broker did not take, so it discards it. That is the intended behaviour of 0, and it is counted: dcs_runtime_mqtt_publishes_total{outcome="dropped"} rises for every message lost that way, and for any payload that fails to marshal.

Prometheus metrics: dcs_runtime_mqtt_queue_depth (current pending messages), dcs_runtime_mqtt_queue_replayed_total, dcs_runtime_mqtt_queue_evicted_total, and dcs_runtime_mqtt_publishes_total broken out by kind and outcome.

Port Allocation Policy

Unit runtime pods use hostNetwork: true so they can reach field I/O devices on the host's physical network interfaces. Because host-network pods bind directly to the node's network stack, their server ports must not fall within the Linux ephemeral port range (default 32768–60999, net.ipv4.ip_local_port_range). Ports in this range can collide with kernel-assigned source ports for outbound connections, causing random bind: address already in use failures.

Policy: all DCS server ports must be < 32768 or > 60999.

Component	Port	Protocol	Notes
Unit runtime health	61052 (default)	gRPC	Per-unit unique; set via runtimeHealthPort
Unit runtime HTTP API	61152 (default)	HTTP	Per-unit unique; set via runtimeGRPCPort (name kept for API compatibility — the port serves HTTP despite the name)
Unit runtime metrics	61252 (default)	HTTP	Per-unit unique; set via runtimeMetricsPort
io-probe HTTP	8090	HTTP	Below ephemeral range. Runs per-Controller and per-namespace network-io-probe pods — no collision with the gateway (different pods, different IPs)
io-probe health	8091	HTTP	Below ephemeral range
Gateway HTTP	8090	HTTP	ClusterIP service, without hostNetwork
Gateway metrics	61051	HTTP	ClusterIP service, plaintext /metrics
Historian HTTP	8092	HTTP	ClusterIP service
Historian health	8093	HTTP	ClusterIP service
Historian metrics	8094	HTTP	ClusterIP service, plaintext /metrics
OMF egress health	8095	HTTP	ClusterIP service. Readiness follows the MQTT broker alone: an OMF endpoint that is down is the case the component rides out, and reporting unready would restart the one process holding the queue
OMF egress metrics	8096	HTTP	ClusterIP service, plaintext /metrics
Operators (metrics)	8080	HTTP	ClusterIP service
Operators (health)	8081	HTTP	ClusterIP service
MQTT broker	1883 / 8883	MQTT	ClusterIP service

When deploying multiple unit runtimes on the same node, each of the three runtime ports (runtimeHealthPort, runtimeGRPCPort, runtimeMetricsPort) must be unique across pods on that node, since hostNetwork: true makes them global on the node. The three ports are independent. Set all three explicitly in the Unit spec to co-tenant multiple runtimes on one node. Example: two units could use (61060, 61160, 61260) and (61061, 61161, 61261) respectively.

Multi-tenant demo deployments do this assignment for you. Both provisioning paths, the gateway's auto-provisioning and hack/demo-provision.sh, give each tenant its own consecutive slice of three reserved ranges, 61020–61499 for health, 61500–61979 for the HTTP API, and 61980–62459 for metrics, rewriting the fixture set's own declared ports as they template it. The starting point is a checksum of the username, and the search walks forward over what the cluster already holds. A tenant therefore keeps the same ports across re-provisioning without ever landing on another tenant's. Leave those three ranges alone when hand-assigning ports on a node that also hosts demo tenants.

For the full Prometheus metric catalogue, recommended alerts, and Grafana dashboards that consume these endpoints, see Monitoring and Metrics.

Defense-in-depth: on production nodes, consider reserving the DCS port block in ip_local_reserved_ports (e.g., via Talos machine config):

sysctl -w net.ipv4.ip_local_reserved_ports="61051-61252"

Gateway

The DCS gateway provides:

• REST API for managing the ISA-88 hierarchy (CRUD operations on all CRDs)
• Web UI for browsing the physical hierarchy, managing batches, and monitoring the procedural tree
• Device registration — Controller CRUD with node-label management. Cluster node join happens at the deployment layer (ADR 0004)

Enrollment Flow

Cluster join is the deployment layer's job, done with Talos machine config, a k3s token, or a cloud nodegroup (see ADR 0004). The product adopts the resulting node:

sequenceDiagram
    participant Deploy as Deployment layer
    participant K8s as Kubernetes
    participant Op as Physical operator
    Deploy->>K8s: 1. Join node (dcs.io/site=<site> label)
    K8s-->>Op: 2. Node appears
    Op->>Op: 3. Auto-create Controller CR (or bind pre-created CR by name)
    Op->>K8s: 4. Reconcile spec.nodeLabels onto node
    Op->>Op: 5. Controller: Pending -> Joining -> Joined

See the Device Enrollment Guide for the step-by-step workflow.

Simulation controllers (spec.simulation: true) skip node adoption entirely. The Controller reconciler creates a virtual Kubernetes Node and maintains its heartbeat Lease, allowing simulation IOModules and unit runtimes to operate without physical hardware.

Operator Prompts

Phase SFC charts pause execution for operator input through three typed prompt builtins (ADR 0017). PROMPT('message') requests an acknowledgement and returns the fixed sentinel 'Acknowledged'. PROMPT_CHOICE('message', 'opt-a', 'opt-b') requests an enumerated decision and returns the selected option. PROMPT_VALUE('message',min, max, 'unit') requests a validated numeric entry within required bounds and returns REAL. The flow:

1. ST calls a prompt builtin — the bridge blocks, pending prompt state (including the type and, per type, the option list or bounds/unit) is published to Phase.status.pendingPrompt
2. UI shows the typed card — the phase detail view polls for pendingPrompt and renders the affordance matching the type: an Acknowledge button, one button per option, or a bounded numeric input, plus the e-signature reason field
3. Operator answers — POST /api/v1/sites/{site}/phases/{name}/prompt-response validates the response against the prompt type fail-closed (sentinel forced for acknowledge, option membership for choice, bounds for value), validates the e-signature (HMAC-SHA256, fresh OIDC token per 21 CFR 11.200), sets a dcs.io/prompt-response annotation, and creates a PromptAcknowledge audit record

4. Controller delivers response — a watcher goroutine detects the annotation and unblocks the bridge, which re-validates the response against the spec before SFC execution continues

5. Run ends — a prompt only lives as long as the runtime episode that raised it. If that episode ends without an answer (the phase aborts, stops, completes, or is reset), the operator clears Phase.status.pendingPrompt and the gateway rejects any late response with 409 Conflict. Otherwise the field outlives its owner: the HMI counts a prompt nobody can action, and a response against it would record an electronic signature bound to nothing (#992). Held and Paused are excluded. A Restart or Resume returns to the chart, so the prompt is still live.

This implements ISA-88 Part 4 Table 7 Prompt/Prompt Response event types. Free-form text is excluded from flow control. Operator notes belong in batch-record annotations.

Validation rule: Transitions from steps that contain a prompt builtin must have an explicit conditionST. Capture the prompt response in a variable (ack := PROMPT('...')) and use it as the transition guard (conditionST: "ack <> ''", conditionST: "cut = 'Cut now'", or a numeric comparison for PROMPT_VALUE()). Unconditional transitions from prompt steps are rejected by both the API and gateway validators.

Node Scheduling

Units declare a nodeSelector that is copied to the runtime pod's spec.nodeSelector. By convention, device nodes are labeled with dcs.io/device=<unit-name> to pin runtime pods to specific hardware.

Control Module Role Bindings

Phase templates must reference control modules unit-agnostically so that the same phase logic can run on different units with differently-named CMs. Units provide this abstraction through spec.cmRoles, a map from abstract role names to concrete ControlModule instance names:

UIYAML

/system → Equipment Library → Units → open the unit → scroll to CM Role Bindings. Each row maps a role name (declared by the phase templates this unit runs) to one of the unit's ControlModule instances. Edit or delete rows with the per-row action buttons.

spec:
  cmRoles:
    outlet_valve: discharge-valve
    temp_sensor: product-temp

The unit controller emits role-aliased entries in the tag map ConfigMap alongside the concrete CM entries. The runtime's tag resolution is transparent. WRITE('outlet_valve.CMD', TRUE) resolves to the same physical address as WRITE('discharge-valve.CMD', TRUE).

The tag map reaches the runtime on two paths: the ConfigMap volume (durable: what a restarted runtime loads, polled for changes every 10 s) and a direct push to the runtime's POST /api/v1/tagmap whenever the unit controller rewrites the ConfigMap. The push closes the kubelet volume-sync latency window (up to ~a minute) so a newly created ControlModule's tags resolve as soon as its FB network deploys. If the push fails, the file poll converges on its own.

The IOModule driver config (iomodules.json) follows the identical two-path pattern: the ConfigMap volume the runtime polls every 10 s (durable/fallback) and a direct push to POST /api/v1/ioconfig whenever the unit controller rewrites the ConfigMap. The push closes the same kubelet volume-sync window so a newly bound SimulationPreset's behaviors re-seed on the live driver in seconds. The interactive virtual-commissioning workflow would otherwise show a 60–90 s dead window.

PhaseTemplates declare their CM role requirements via spec.cmRoles for validation and documentation. At batch instantiation, the batch controller verifies that the target unit provides all required role bindings.

The Controller reconciler converges Controller.spec.nodeLabels onto the bound node (restricted to the dcs.io/ prefix), and Controller.spec.displayName onto the node's dcs.io/display-name annotation.

Both are instances of one rule. The gateway holds no write privilege against a Kubernetes node (ADR 0034), so anything the product authors about a node is written to a product-owned object and applied by the operator that already holds the privilege. Every surface then reads the value back from the node at request time, which keeps one answer to what a machine is called whether the deployment layer named it at enrolment or an engineer typed it in the UI.

Each converged field tracks what the operator last applied, in a physical.dcs.io/applied-* annotation on the Controller. Clearing the spec removes only what the operator itself wrote, so a label or a name the deployment layer set directly is never removed by a product edit.

Naming a machine that is not a Controller

A control-plane node or an unclassified worker has no Controller to author through, so NodeIdentity is that object for it: cluster-scoped, one per node and named for the node, carrying every product-authored attribute of the machine (displayName today, and a second attribute becomes a second field on the same kind). The NodeIdentityReconciler converges it onto the same annotation under the same ownership rules, tracking what it applied in status.appliedDisplayName.

The two authors never write the same node. A node a Controller has adopted is named on that Controller, and a NodeIdentity for it is Refused with the owning Controller named, because two authors converging on one annotation would each undo the other on every reconcile. ADR 0040 records the decision, including why a Server CRD and a narrow Node patch grant are both closed.

Taking a node out of service

NodeMaintenance is the same division applied to the node admin verbs. It is a cluster-scoped record of an operator's intent (cordon, drain, reboot, shutdown or power-on) with a lifecycle the physical operator advances, and the gateway writes the record without touching the node (ADR 0034). One object covers the whole procedure, so a cordon-only maintenance reaches Cordoned and waits there for return to service. It is the same kind of request as a drain, stopped earlier.

power-on is the one intent that runs the other way, and it lives on this object for the reason the rest do. The gateway holds no write against a node and no channel to the management network, so an operator's click has to become an object the physical operator acts on. It shares nothing else with its siblings: nothing is cordoned, nothing is drained, and none of the refusals below apply, because they protect a running machine and this one is already off.

Three refusals are terminal and carry their reasons. A virtual node has no hardware to service. An action that would drop ready control-plane nodes below etcd quorum is refused outright and never upgraded into a ceremony, because the result is an API server that has gone read-only and cannot be recovered from the UI that offered the action. And a node hosting live unit runtimes refuses a drain, naming the units, because moving process control runs through the ADR 0006 failover path: only an operator can certify that a node is fenced, and a reconciler cannot self-certify.

Return to service uncordons only what the maintenance itself cordoned. A node an engineer had already cordoned by hand stays cordoned, which is the same ownership rule the converged fields above follow.

Standards Compliance

Cloud-Native DCS is designed for pharmaceutical and regulated manufacturing environments. The following standards guide the architecture. Each has a dedicated traceability matrix that maps requirements to CRDs and code:

Standard	Scope	Status	Traceability
ISA-88 / IEC 61512	Batch process control — physical, procedural, recipe, and batch models	Core architecture	Traceability matrix
IEC 61131-3	Programmable controller languages — SFC, ST, FBD for control logic	SFC engine + ST interpreter + FB runtime	Traceability matrix
21 CFR Part 11	FDA electronic records and signatures — audit trails, access control, electronic signatures, change control	Substantially Implemented	Traceability matrix
IEC 62443	Industrial cybersecurity — network segmentation, security levels, secure development lifecycle	Substantially Implemented	Traceability matrix

21 CFR Part 11 Implications

FDA 21 CFR Part 11 establishes requirements for electronic records and electronic signatures in regulated industries (pharmaceutical, biotech, food). Key requirements that affect this system:

• Audit trails: All changes to records (recipes, batch parameters, control logic, equipment configuration) must be logged with who, what, when, and why. Kubernetes resource versioning provides a foundation, but a dedicated audit log with user attribution is required.
• Electronic signatures: Certain actions (recipe approval, batch release, control logic deployment) must be attributable to an identified individual. Signatures must be linked to the specific record version.
• Access control: Role-based access ensuring only authorized individuals can create, modify, or delete records. Different roles for operators, engineers, and quality assurance.
• Change control: Modifications to approved recipes and control logic require documented review and approval before taking effect. Implemented via the ChangeControlPolicy CRD (cluster-scoped, changecontrol.dcs.io) and a change-control admission webhook hosted in batch-operator: in namespaces labelled dcs.io/production=true, mutations to change-controlled resources must originate from an allowlisted ServiceAccount and carry the required external-reference and e-signature annotations. See Change Control for the full workflow and approval gates.
• Record retention: Electronic records must be retained for the required duration and be available for FDA inspection.

IEC 62443 Implications

IEC 62443 defines a framework for securing industrial automation and control systems. Key areas:

• Zones and conduits (IEC 62443-3-2): Network segmentation between control plane (Kubernetes), field devices, and enterprise networks.
• Security levels (IEC 62443-3-3): System security requirements ranging from SL 1 (casual) to SL 4 (state-sponsored threats).
• Secure development (IEC 62443-4-1): Secure coding practices, threat modeling, vulnerability management.
• Component security (IEC 62443-4-2): Authentication, authorization, integrity, and confidentiality at the component level.

IEC 61131-3 Alignment

IEC 61131-3 defines five programming languages for control systems. Cloud-Native DCS implements three:

• SFC (Sequential Function Chart): Procedural sequencing. The engine publishes live execution status (active steps, transition evaluation results, and ST variable snapshots) to the Phase CRD status, enabling real-time monitoring. Transition conditions with AND-joined sub-expressions are evaluated per-operand for diagnostic visibility.
• ST (Structured Text): Step actions, transition conditions, and control module templates. Templates support direct ST authoring as an alternative to visual FBD editing.
• FBD (Function Block Diagram): Control programs with standard function blocks (PID, timers, counters, I/O, logic, arithmetic) executing in a cyclic scan loop. There is no event-driven scheduling, because every block runs every scan.
• ControlProgram CRD: Declarative definition of control programs (FBD or ST) as Kubernetes resources, with variable declarations for tag binding. Deployed automatically to unit runtimes.
• Hot-swap: Running programs can be updated in-place without stopping other programs on the same runtime.