ISA-88 / IEC 61512 Compliance Traceability¶
Standard: ISA-88.01 (IEC 61512-1) -- Batch Control Part 1: Models and Terminology Applicability: Complete batch process control architecture Overall Status: Substantially Implemented -- the physical, procedural, recipe, batch, state, mode, and exception models are implemented with tests and evidence. Named Partial rows remain in the tables below, and the Part 3 library / transformation features and Part 2 scheduling features are implemented at the data-model and controller layer with no UI/API/CLI surface (removed in #480 with CRDs and controllers retained, and the kubectl/GitOps authoring path remains).
Status Legend¶
| Indicator | Meaning |
|---|---|
| Implemented | Requirement is fully implemented with tests and evidence |
| Implemented (no UI surface) | Data model (CRD) and controller are implemented and shipped; the UI/API/CLI authoring surface was removed in #480 — resources are authored via kubectl/GitOps |
| Substantially Implemented | Majority of requirements met; named gaps remain, documented per-clause in the matrix |
| Partial | Core functionality exists but gaps remain |
| Planned | Architecture is designed but implementation has not started |
| N/A | Requirement does not apply to this system |
Physical Model¶
ISA-88 defines a 7-level physical hierarchy. Cloud-Native DCS maps each level to a Kubernetes CRD, plus adds Controller, ControlModuleTemplate, and IOModule for device management and reusable control logic.
| ISA-88 Level | CRD | Scope | Status |
|---|---|---|---|
| Enterprise | Enterprise | Cluster | Implemented |
| Site | Site | Cluster | Implemented |
| Area | Area | Namespaced | Implemented |
| Process Cell | ProcessCell | Namespaced | Implemented |
| Unit | Unit | Namespaced | Implemented |
| Control Module | ControlModule | Namespaced | Implemented |
| (Device) | Controller | Namespaced | Implemented |
| (Template) | ControlModuleTemplate | Cluster | Implemented |
| (I/O) | IOModule | Namespaced | Implemented |
| (Simulation) | SimulationPreset | Namespaced | Implemented |
| (Simulation) | IOModule.spec.simulation | Inline | Implemented (replaces former SimulationProfile CRD) |
Physical Hierarchy Validation¶
| Requirement | Status | Implementation |
|---|---|---|
| Parent-child relationships enforced | Implemented | ValidateHierarchy() ensures correct nesting (Enterprise > Site > Area > ProcessCell > Unit > ControlModule) |
| Equipment capabilities declared | Implemented | EquipmentCapability type on Units; used for batch equipment matching |
| Node scheduling for Units | Implemented | UnitSpec.NodeSelector copied to runtime pod; convention dcs.io/device=<unit-name> |
| CM role bindings | Implemented | UnitSpec.CMRoles maps abstract role names to concrete ControlModule instances; tag map includes role-aliased entries so phase logic can reference CMs unit-agnostically |
| Smart-skid orchestrate-mode service dispatch | Partial | UnitSpec.ServiceBinding (api/physical/v1alpha1/unit_types.go) models a vendor-packaged smart skid as a state-bearing Unit whose phase logic invokes remote OPC UA services, with no FB-mapped I/O to drive. Reuses the OPC UA driver layer via OPCUASecurityConfig (security policy/mode/auth/cert-pin shared with OPCUABindingOptions). Per ADR-0001 this is orchestrate-mode only — no remote-state mirror; CEL XValidation on UnitSpec rejects mixing serviceBinding with equipmentModules (the mirror-mode shape). Service argument types restricted to IEC 61131-3 ST primitives (BOOL/INT/DINT/REAL/LREAL/STRING) so the runtime can coerce values at CALL_SERVICE evaluation time without ambiguity. Issue #345; runtime CALL_SERVICE ST builtin is the followup #346. |
Device Discovery → Physical Model Classification (ADR 0020)¶
The Device Discovery wizard browses a live OPC UA address space and emits the ISA-88-correct instance shape for each discovered node, classified per node onto one of two planes. The mapping is grounded in ANSI/ISA-88.00.01-2010 Clause 3 (verified 2026-07-13 against the normative text) and recorded in ADR 0020.
| ISA-88 basis (Part 1, Clause 3) | Discovered node | Emitted shape | Status |
|---|---|---|---|
| Control Module — "the lowest level grouping of equipment in the physical model that can carry out basic control" (§3.1.13); NOTE 3: "The control module level contains the interfaces to the physical equipment." | Field device / instrument — exposes Variables, no autonomous services (data-plane) | Template-free ControlModule instance (templateRef: "") under an operator-chosen parent Unit, its tags[] carrying the discovered OPC UA variable node IDs with dataType/accessLevel; plus one shared opcua IOModule the tags route through. No Controller is fabricated. |
Implemented |
| Basic control — "the control dedicated to establishing and maintaining a specific state or behavior of equipment and process" incl. regulatory control, interlocking, monitoring (§3.1.5) | (as above) — the field device has no autonomous control | The supervising DCS runtime FB network supplies the CM's basic control over the discovered tags; discovery emits the tags roleless (semantics are declared later and never inferred, ADR 0016). | Implemented |
| Unit — "a collection of associated equipment modules and/or control modules that can carry out one or more major processing activities" (§3.1.71) | Module / skid — exposes Methods / a service state machine (service-plane) | Unit + serviceBinding (unchanged from the smart-skid path above). The discovered field-device CM's parentKind: Unit realises the §3.1.71 containment relation. |
Implemented |
Backend surface: POST /api/v1/discovery/opcua/variables reads per-variable
DataType/AccessLevel (both required on Tag). Client emission in
internal/gateway/static/js/views/discover.js (buildCRFromCandidate) emits
[IOModule, ControlModule] in dependency order for data-plane devices. Shipped
in 65207abb, no CRD change. Conformance scope: the #928 line has since
landed native type-definition classification (ADR 0022), PA-DIM ingestion
with NE 107 health (ADR 0043), and the VDI 2658-4 service model with its
declared ISA-88 correspondence
(ADR 0045, pkg/mtpservice), and still
claims no MTP or PA-DIM information-model conformance: the ISA-88/MTP
state-model alignment is not unified, and the MTP runtime binding
(VDI 2658 Blatt 5.1) and AML manifest import are not implemented.
Procedural Model¶
ISA-88 defines a 4-level procedural hierarchy for sequencing batch operations.
| ISA-88 Level | CRD | Status | Implementation |
|---|---|---|---|
| Procedure | Procedure | Implemented | Sequences UnitProcedures via SFC chart (spec.chart); supports sequential transitions and parallel divergences (see Parallel execution below for what a divergence reaches); cascades commands |
| Unit Procedure | UnitProcedure | Implemented | Scoped to a single Unit; sequences Operations via SFC chart (spec.chart) |
| Operation | Operation | Implemented | Sequences Phases via SFC chart (spec.chart) |
| Phase | Phase | Implemented | Leaf element; executes SFC charts with ST actions via runtime HTTP bridge |
| Operator prompts (Part 4 Table 7) | -- | Implemented | Typed prompt builtins (ADR 0017) pause SFC execution: PROMPT() acknowledge, PROMPT_CHOICE() enumerated decision, PROMPT_VALUE() bounded numeric entry; typed PendingPrompt status field; gateway endpoint with fail-closed response validation and e-signature; PromptAcknowledge audit records |
| External results (data, no Part 4 operator event) | -- | Implemented | AWAIT_RESULT() pauses SFC execution on a measurement produced outside the control system — a laboratory assay being the canonical case. Deliberately NOT modelled as a Part 4 Table 7 Prompt/Prompt Response pair: that pair records an operator action, and this records data with provenance and no electronic signature (ADR 0055). Typed PendingExternalResult status field; API-key delivery endpoint with the same fail-closed bounds validation the prompt path applies; ExternalResult audit records, aggregated into BatchRecord.spec.externalResults and kept out of operatorActions, so the Part 4 production record never attributes a machine's measurement to a person |
PROMPT() and parks "Waiting for Operator", announced by a top-bar Respond badge and a chip on the unit card, until the operator answers from the phase's Operator Prompt card.PROMPT_CHOICE renders one button per authored option and PROMPT_VALUE bounds the numeric entry to its declared range, each answered with a signed reason.Procedural Hierarchy Validation¶
| Requirement | Status | Implementation |
|---|---|---|
| Parent-child relationships enforced | Implemented | ValidateProceduralHierarchy() ensures Procedure > UnitProcedure > Operation > Phase |
| Command cascading | Implemented | Start/Stop/Hold/Restart/Abort propagate down the tree |
| Step sequencing | Implemented | Each level sequences its children in order; next step starts on child completion |
| Parallel execution | Partial | A Procedure, UnitProcedure or Operation whose own spec.chart carries a SimultaneousDiverge starts every one of its children together, with failure propagation (chartHasParallelDivergence, reconcileRunningParallel). The flag is read once per chart. Every child of the element starts, whichever branches the divergence names. It is reachable only on a hand-authored element. Batch instantiation builds each child chart with BuildSequentialChart, which carries no divergence, so a divergence declared on a recipe procedure or on an OperationTemplate, UnitProcedureTemplate or ProcedureTemplate never reaches execution. On a recipe procedure chart the field is refused (#1698). Above the phase it is still accepted and dropped (#1703). |
| Parameter binding | Implemented | ParameterSpec and ParameterValue support recipe-to-equipment parameter mapping; RecipeSFCStep.ParameterBindings and ${paramName} template variables resolve recipe formula values into step parameters at instantiation time, creating a traceable link for audit (21 CFR Part 11) |
| Hierarchical parameter propagation (Clause 8.7.2) | Implemented | Each procedural level (UnitProcedure, Operation, Phase) receives its own ParameterValues resolved from its template's formal interface. Parameters cascade: recipe > UP > OP > Phase. OutputValues on status carries the chart's terminal-time output snapshot for audit recording. |
| Process output capture (Part 4 §5.3 + Part 1 §6.3.3) | Implemented | spec.outputs[] declares process outputs by name and type; the chart's ST writes to declared output names like any ST variable. Phase reconciler snapshots the declared output values from the ST environment at terminal time including Abort/Stop paths; non-terminal charts (Holding/Restarting/Resetting) do not write to OutputValues. Outputs the chart never assigned to are omitted from the BPR — no sentinel rows. Capture timing is controlled by which chart steps assign to the output (issue #250). |
| Process inputs / parameters / outputs (Part 1 §6.3.3) | Implemented | Two categories distinguished in the procedural API: spec.parameters[] carries Process Parameters (recipe-commanded inputs); spec.outputs[] carries Process Outputs (chart-written values — paired actuals, totalisers, integrated quantities, environmental measurements, derived quality metrics). The BPR splits the rows into two flat lists (spec.parameterValues[] for inputs with recipe-layer provenance, spec.outputs[] for chart-written outputs with no provenance enum) so audit consumers render the two categories from list membership, with no Kind enum to consult. Names are unique across both lists per template (instantiator validates). Process Inputs are tracked separately via MaterialTrace. (Issues #248, #250.) |
| Acting-state logic coverage (Clause 7.5) | Implemented | A PhaseTemplate authored through the gateway must declare logic for every acting state — holdingChart, restartingChart, stoppingChart, abortingChart, resettingChart (bare …ST fields are normalized into single-step charts first). validatePhaseTemplateST (internal/gateway/validate.go) rejects a PUT /sites/{site}/phasetemplates/{name} that omits any of them. Convention: holdingChart/stoppingChart/abortingChart drive equipment to a safe state, restartingChart re-asserts the running outputs so a Held→Restart resumes what Holding safed, resettingChart returns to idle. The restarting chart is the only thing that re-asserts them: a resumed step whose action had completed does not re-run it, deliberately, so a resume repeats no prompt, record line, or service call. A phase whose steps command conflicting postures therefore branches its restarting chart on STEP_ACTIVE('<action step>'), which names the step the phase was held on (ADR 0047, issue #1389); before that ADR a transitional chart also overwrote the held position, so no Held→Restart resumed at all and no holding chart ran its safe-state action. The runtime tolerates a nil chart as an immediate auto-complete (null action) for CRs applied directly via SSA, but the editor requires the author to state each state's behavior explicitly. Reference examples model all five (see examples/*/08a-phase-templates.yaml, riverbend, docs-shots). Issues #246, #821, #1389. |
| CM role requirements | Implemented | PhaseTemplateSpec.CMRoles declares required ControlModule roles; validated at batch instantiation against the target unit's CMRoles bindings |
| Procedural model (ISA-88 Clause 6.5) | Implemented | RecipeSFCStep.TemplateRef + TemplateKind reference PhaseTemplate / OperationTemplate / UnitProcedureTemplate; ValidateRecipeSFCChartForLevel() enforces correct reference types per recipe level; ISA88Compliance status condition reports compliance on each recipe |
| Deterministic template expansion | Implemented | RecipeSFCStep.TemplateKind enables deterministic expansion in batch instantiator (PhaseTemplate vs OperationTemplate vs UnitProcedureTemplate); StepLevel tracks ISA-88 hierarchy level |
| Recipe procedure branching (Clause 5.3) | Not implemented | A RecipeSFCChart declares divergences[], and nothing reads them. createNestedProceduralTree expands a recipe procedure by creating one child procedural element per step in DECLARATION ORDER and giving that child a chart built from scratch, so neither the recipe's transitions nor its divergences reach execution. ValidateRecipeSFCChart() refuses any entry, so a parallel branch cannot run sequentially in silence. The SFC editor no longer offers the gesture that authored one. Recipe procedures are strictly sequential. Issue #1698. |
| Chart-level homogeneity (Clause 5.3.4) | Implemented | Every direct step in a RecipeSFCChart must occupy the same procedural level (e.g. all UnitProcedure, all Phase). validateChartHomogeneity() rejects mixed-level charts; authors descend levels via a step's subChart. Resolves issue #257. |
| Recipe-driven OPC UA service invocation | Implemented | Unit.spec.serviceBinding.services[] (api/physical/v1alpha1/unit_types.go) declares OPC UA Method services that phase ST/SFC bodies invoke via the CALL_SERVICE(unit, service, ...) builtin. The procedural reconciler dispatches the call (internal/controller/procedural/call_service.go) — connect, invoke, decode result, emit one AuditRecord per invocation correlated with the parent Batch ID. Failure path: a non-Good StatusCode surfaces as an ST runtime error so the chart can HOLD or ABORT. ADR-0001 + issues #345/#346/#348. |
Recipe Model¶
ISA-88 defines a 4-level recipe derivation chain from general to execution-specific. The two Part 3 levels above Master Recipe (General Recipe, Site Recipe) were removed in #858 (originally surface-reduced in #480): the product implements the Part 1 Master/Control levels, and the Part 3 kinds can be reintroduced on customer demand (git history has the CRDs, controllers, and transform engine).
| ISA-88 Level | CRD | Scope | Status | Implementation |
|---|---|---|---|---|
| General Recipe | — | — | Not implemented (removed in #858) | Part 3 kind removed with its controller and GR→MR transform engine; re-add on customer demand |
| Site Recipe | — | — | Not implemented (removed in #858) | Part 3 kind removed with its controller; re-add on customer demand |
| Master Recipe | MasterRecipe | Namespaced | Implemented | Targets a specific ProcessCell and equipment |
| Control Recipe | ControlRecipe | Namespaced | Implemented | Immutable runtime copy created per Batch execution |
Recipe Derivation Validation¶
| Requirement | Status | Implementation |
|---|---|---|
| Derivation chain enforced | Partial | Master and Control levels exist as CRDs (General/Site removed in #858). Master > Control creation is enforced at batch instantiation. ValidateRecipeDerivation() (pkg/s88/validation.go) covers the Master > Control kind chain. Content-level derivation (verifying a child's procedure/formula actually derives from the parent) is not checked |
| Recipe approval gate | Implemented | Annotation-driven (dcs.io/recipe-action: approve); validates parameters and equipment requirements before approval |
| Recipe immutability | Implemented | ControlRecipe is write-once; created by batch controller with all parameter bindings and template variables resolved at instantiation time |
| Equipment requirements | Implemented | EquipmentRequirement specifies required capabilities; matched during batch allocation |
| Formula selection | Implemented | BatchSpec.FormulaRef selects explicit formula; auto-selects closest by batch size when omitted; formula parameters layer between ParameterValues and ParameterOverrides |
| Formula parameter enrichment (Part 2 Table 10/41) | Implemented | FormulaParameter type carries engineering units, value limits, tolerance bands, and scaling metadata per ISA-88 Part 2 Table 10 (Parameter) and Table 41 (standard sub-parameters); FormulaParametersToValues() converts to lightweight RecipeParameterValue for downstream consumers |
| Recipe lifecycle states (ISA-88 Part 3 Table 7: Draft/Approved/Released/Effective/Withdrawn) | Implemented | RecipeStatus.Phase enum with enforced transitions; per-state audit fields (ApprovedBy/At, ReleasedBy/At, EffectiveAt, WithdrawnBy/At); gateway and CLI lifecycle commands. Table 7 is a state list, and Part 3's annex (Question 10) says there is no standard transition model, that valid transitions are company policy, and that reversals have to be considered. The one reversal this product defines is revision revert, which returns a recipe to Draft and is refused while the recipe is Released or Effective, the two states Table 7 describes as distributed and as available for use (internal/gateway/recipe_revert_guard.go, #1581, ADR 0060) |
Other Information (ISA-88 Clause 6)¶
| Requirement | Status | Implementation |
|---|---|---|
| Other Information category (Clause 6) | Implemented | RecipeOtherInformation struct with safety notes, regulatory notes, operator instructions, and document references |
| Document references | Implemented | DocumentReference type with enum validation (SDS, SOP, Specification, Diagram) |
| Available on all recipe levels | Implemented | OtherInformation field on MasterRecipe and ControlRecipe specs |
| Propagation to ControlRecipe | Implemented | InstantiateRecipe() deep-copies OtherInformation from MasterRecipe to ControlRecipe for batch traceability |
Process Action Library (ISA-88.00.03-2003 Clause 5.5.10)¶
Not implemented (removed in #859, originally surface-reduced in #480). The
Part 3 process-action library kind was purged with the General/Site recipe
levels (#858). Recipe steps reference equipment procedural elements directly
via templateRef. Re-add on customer demand, since git history has the CRD
and controller.
Equipment Requirement Library (ISA-88.00.03-2003 Clause 5.6)¶
Not implemented (removed in #859). Recipes declare equipment requirements
inline (EquipmentRequirement with a capability string). The standalone
requirement-definition library kind was purged. Re-add on customer demand.
Material Definition Library (ISA-88.00.03-2003 Clause 5.4)¶
Not implemented (removed in #859). Recipe formulas and material lots identify
materials by plain string identifiers (materialDefinitionRef /
materialClassRef field names retained as identifiers). The MES/ERP owns the
material master. The standalone library kinds were purged. Re-add on customer
demand.
Batch Model¶
ISA-88 Batch is the execution entity that ties recipes to equipment.
| Requirement | Status | Implementation |
|---|---|---|
| Batch lifecycle (Pending > Allocating > Running > Complete/Stopped/Failed/Aborted) | Implemented | BatchPhase enum in status; BatchReconciler manages transitions |
| Batch size validation against recipe header | Implemented | Pending phase validates spec.batchSize against MasterRecipe header.minBatchSize/header.maxBatchSize; fails batch with audit record on violation |
| Recipe instantiation | Implemented | Creates ControlRecipe from MasterRecipe at batch start |
| Equipment allocation | Implemented | Allocates Units based on recipe equipment requirements, capabilities, and property constraints (numeric lower-bound + exact string match); runtime re-validation includes property checks. A capability the recipe maps to a unit in targetUnits is a binding: an unavailable bound unit blocks allocation with no substitution, so a batch never executes on equipment its recipe did not name (#1129). The allocation is stored on both the Batch and the Unit and re-checked on every reconcile, keyed on the Unit's metadata.uid: a batch that lost the unit-side claim re-asserts it, and one whose Unit was deleted and recreated underneath it holds with the lost equipment named, refusing to run on equipment that disowns it (#1314) |
| Procedural tree creation | Implemented | Creates Procedure > UnitProcedure > Operation > Phase from ControlRecipe. A tree that cannot be built on the allocated equipment (phase-template cmRoles unsatisfied) holds the batch with the violation as status.holdReason, where reporting Running with no procedure behind it was the #1129 failure |
| Batch event tracking | Implemented | BatchEvent captures lifecycle events with Type, Reason, Message, Timestamp |
| Audit trail integration | Implemented | Batch creation, failures, and state changes recorded as AuditRecords |
| Batch production records (Part 4) | Implemented | BatchRecord CRD auto-created at batch completion; reconciler materialises phase history (with procedural paths), operator prompt acknowledgements, parameter values with recipe-layer source attribution, process outputs from chart snapshots, control-recipe snapshot, and change history into the spec so the record is covered by the finalization e-signature (issues #194, #159, #250). BatchRecordParameter.Source enum (formula-scaled / override / default) tags input rows with their recipe-layer provenance; output rows live on a separate Outputs list with no provenance enum because the chart's ST is the only producer. Consolidated record assembly API + BatchML XML export still available for live assembly when the record is missing. Controller failovers during the batch are materialised into spec.failoverEvents[] as Part 4 Table 7 Message/System events (controller failed) with §5.6 event association carried as AuditRecord references, the control and data gaps bounded separately (#1308, ADR 0041); the BatchML export renders them as FailoverEvent elements with the Part 4 EquipmentID attribute naming. The batch's own significant-event log is frozen into spec.executionEvents[] as Part 4 Table 7 Message/System events, so a procedural hold, an aborted unit or a rejected restart reaches the reviewer inside the record; before #1686 the consolidated view read them from the live Batch resource alone, so they expired with it. A record snapshotted while a phase was still transitioning is repaired until every entry rests at an ISA-88 resting state, and the verdict is written on every record as a PhaseHistorySettled condition, so a phase entry reading Running inside a Complete record is disclosed to the reviewer before it can be signed (#1686) |
| BatchRecord immutability | Implemented | Validating webhook rejects mutations after finalization (21 CFR Part 11) |
Lot Tracing (ISA-88.00.01-2010 Clause 8.5 / ISA-88.00.04-2006)¶
Implemented (no UI surface). Namespaced material-lot and material-trace CRDs
(materiallots.batch.dcs.io, materialtraces.batch.dcs.io) with lifecycle
controllers (internal/controller/batch/materiallot_controller.go,
internal/controller/batch/materialtrace_controller.go, registered in
cmd/batch-operator/main.go) that track lot state with audit records and
validate each trace's Batch and lot references, updating lot
consumption/production status. The UI/API/CLI authoring (CRUD) surface was
removed in #480. Lots and traces are authored via kubectl/GitOps against the
retained CRDs, while the read-only trace queries the BPR Materials section
depends on (/api/v1/sites/{site}/trace/... forward, backward, genealogy,
and batch-materials) remain live in the gateway.
State Machine¶
ISA-88 defines a state model for procedural elements with 13 states and 8 external commands.
| Requirement | Status | Implementation |
|---|---|---|
| 13 ISA-88 states | Implemented | Idle, Running, Complete, Pausing, Paused, Holding, Held, Restarting, Stopping, Stopped, Aborting, Aborted, Resetting |
| 8 external commands | Implemented | Start, Stop, Hold, Restart, Abort, Pause, Resume, Reset |
| Batch command surface (documented deviation) | Implemented | Terminal-phase batches reject all commands with an audited rejection (Action=Command, Result=Rejected) — batches are one-time execution instances; re-executing under the same batch ID would corrupt the BPR. The Part 1 example model's Reset → Idle applies to procedural elements and equipment (units auto-reset and re-allocate). It does not apply to Batch CRs. dcs qualify oq reset-path evidences the rejection contract (#1016). |
| Batch identifier uniqueness | Implemented | A batch is a one-time execution instance, so its identifier names one of them. Batch.spec.batchID is unique within a site, enforced by the Batch validating webhook on create and on any rename (internal/webhook/batch/batch_webhook.go); the same reasoning as the row above, applied here to the identifier. Two batches under one ID would split a single BPR's audit correlation between them (#1460, ADR 0051). |
| Internal SC (StateComplete) | Implemented | Automatic transition from transitional to resting states |
| Transition validation | Implemented | CanTransition() and ValidCommands() enforce valid transitions only |
| Thread safety | Implemented | RWMutex protects all state access |
| Transition callbacks | Implemented | Register callbacks for specific state transitions |
| History tracking | Implemented | Stores last 100 state transitions with timestamps |
| Command delivery | Implemented | Procedural controllers write RequestedState + ActiveWork to Unit status (structural, idempotent via generation counter); dcs.io/command annotation retained for backward compat and manual commands |
| Operational safeguards: control-logic change while active | Implemented | Control-operator defers ControlProgram hot-swaps while any Batch on the target unit is in Running/Holding/Held/Aborting; sets HotSwapDeferred status condition and emits a Rejected AuditRecord. Both causes are gated: a spec replace on the program itself, and content drift from a composite FunctionBlockType edited or deleted under it (#1574). Operator override requires dcs.io/allow-hot-swap=true together with a non-empty dcs.io/hot-swap-reason; reason is recorded in the success audit. Redeploys to a runtime that has lost the program are unconditional and always allowed |
State Transition Table¶
| From State | Start | Stop | Hold | Restart | Abort | Pause | Resume | Reset | SC |
|---|---|---|---|---|---|---|---|---|---|
| Idle | Running | -- | -- | -- | Aborting | -- | -- | -- | -- |
| Running | -- | Stopping | Holding | -- | Aborting | Pausing | -- | -- | Complete |
| Complete | -- | -- | -- | -- | -- | -- | -- | Resetting | -- |
| Pausing | -- | Stopping | -- | -- | Aborting | -- | -- | -- | Paused |
| Paused | -- | Stopping | -- | -- | Aborting | -- | Running | -- | -- |
| Holding | -- | Stopping | -- | -- | Aborting | -- | -- | -- | Held |
| Held | -- | Stopping | -- | Restarting | Aborting | -- | -- | -- | -- |
| Restarting | -- | Stopping | -- | -- | Aborting | -- | -- | -- | Running |
| Stopping | -- | -- | -- | -- | Aborting | -- | -- | -- | Stopped |
| Stopped | -- | -- | -- | -- | -- | -- | -- | Resetting | -- |
| Aborting | -- | -- | -- | -- | -- | -- | -- | -- | Aborted |
| Aborted | -- | -- | -- | -- | -- | -- | -- | Resetting | -- |
| Resetting | -- | Stopping | -- | -- | Aborting | -- | -- | -- | Idle |
What the matrix does not contain¶
Clause 7.5.1 calls this "the complete set of states, commands, and allowed transitions for each procedural element", and the completeness is the point. RUNNING leaves through COMPLETE on normal completion and through STOPPING, HOLDING, PAUSING or ABORTING on command, and there is no other exit. Clause 7.5.2 describes the two that terminate a run as acting states that execute logic: STOPPING performs "a controlled normal stop", ABORTING "an immediate abnormal stop".
Deleting the CR of a running procedural element takes it out of RUNNING with no acting state having run at all, which is a transition the model does not contain. Until 2026-08-14 every delete handler in the gateway permitted it. The Batch, Procedure, UnitProcedure, Operation and Phase handlers now read the target's state first and answer 409 with a Rejected AuditRecord unless it is in one of Clause 7.5.2's three Final States (COMPLETE, STOPPED, ABORTED), in the Initial State (IDLE), or has not begun. Both sets are written as allow-lists, so a status value neither of them names refuses. The alternative is that a new enum member becomes silently deletable mid-run. See ADR 0060 for the ruling and the incident record for how the gap shipped and survived four reviews.
The same transition is reachable through the equipment, and by a shorter path.
A Unit owns its runtime Pod, so deleting the Unit of a running batch collects
the process that executes the function-block network and drives the equipment's
outputs. The element leaves RUNNING with no acting state having run, exactly as
if its own CR had been deleted. What the batch does afterwards is detection, not
prevention. It can no longer read the unit, so it holds. No runtime is left
under the equipment to hold it. The physical handlers now ask the same
question from the equipment side. A Unit that is executing refuses deletion, and
so do the ControlModule, IOModule, Controller, ProcessCell, Area, Site and
Enterprise through which a running Unit is reached. A Unit counts as executing
when its own state is outside the four resting states above, when
status.allocatedTo names a batch holding it, or when status.activeWork names
a procedural element that has not finished. Each of those three is the only
signal available in some window, and the allocation is the one that covers a
batch still allocating against a unit that is still IDLE.
Two rows of the matrix are on camera.
Mode Management¶
ISA-88 Clause 7.3.2 defines modes for procedural elements and equipment entities.
| Requirement | Status | Implementation |
|---|---|---|
| Procedural modes (Automatic/Semi-Automatic/Manual) | Implemented | Mode field on all procedural Specs and ProceduralStatus; dcs.io/mode annotation for runtime changes |
| Equipment modes (Automatic/Manual) | Implemented | Mode field on EquipmentStatus, UnitSpec, and ControlModuleSpec |
| Semi-Automatic pause-at-transitions | Implemented | Sequencing controllers auto-pause (Running to Pausing) when a child completes in Semi-Automatic mode; operator resumes to advance |
| Manual operator-directed execution | Implemented | In Manual mode, sequencing controllers do not auto-start idle children; operator must issue Start command to each child explicitly |
| Mode propagation (spec to status) | Implemented | Unit defaults Spec.Mode to Status.Mode on first reconcile; runtime override via the dcs.io/mode annotation takes precedence. The annotation→Status.Mode consumer runs in the unit, equipment-module, and control-module reconcilers (#952) — before that, only the unit consumed it, so equipment/control modules stamped the annotation but never persisted it and POST …/mode reported success while the switch silently never happened |
| Gateway mode API | Implemented | POST /api/v1/sites/{site}/mode endpoint with kind-specific validation; rejects mode changes during transitional states (HTTP 409) |
| CLI mode command | Implemented | dcs mode <Kind> <name> <Mode> with kind-aware validation |
| Mode switching enforcement | Implemented | Mode changes rejected during transitional states (Pausing, Holding, Restarting, Stopping, Aborting, Resetting); gateway returns 409 Conflict, controllers defer annotation processing until state settles |
| Mode-aware command filtering | Implemented | Gateway filters validCommands by mode: Automatic/Semi-Automatic hide Start (system-initiated); Manual shows all commands |
| Table 1 Automatic (Basic Control): equipment not manipulable by the operator | Implemented | The gateway refuses a direct tag write with HTTP 403 while the ControlModule resolves to Automatic, on both write routes (POST …/controlmodules/{cm}/tags/{tag} and a CM-qualified address on POST …/units/{unit}/tags/{address}). An unset mode reads as Automatic, matching the ControlModuleSpec.Mode default. The refusal is audited as Rejected with reason AutomaticMode. Control-algorithm writes are unaffected: the SFC engine and FB scan write in-process via adapter.WriteTag and never traverse the gateway. Before #1255 the barrier existed only in hmi-faceplate.js, so an operator-tier API caller could move a final control element while the module declared its control algorithm owned it |
| Gateway UI mode control | Implemented | Mode selector dropdown in Unit, Procedure, UnitProcedure, Operation, Phase detail views; calls POST /api/v1/sites/{site}/mode |
Exception Handling (Clause 7.4)¶
ISA-88 Clause 7.4 requires automatic exception detection, severity-based response, and escalation for equipment faults, process deviations, and safety conditions.
The "interlocking" basic-control category (Clause 3.1.5 / Clause 5.2) is
implemented at three layers per
ADR 0007: device-level interlocks
in the output function blocks (DO/AO ILCK input / interlockAddress
parameter, evaluated every scan in the unit runtime, the control module
level where Clause 3.1.13 places basic control), phase SFC transition
guards for procedural safe-state sequencing, and
AlarmDefinition.exceptionAction for the audit-traceable batch response.
The project terminology, the BPCS-vs-SIS boundary, and the worked
patterns are documented in
docs/library/alarms-and-interlocks.md.
Interlocks here are BPCS-layer protection functions. Under IEC 61511 Clause 9.2
they can be credited as independent protection layers, and whether a hazard
also requires an independent SIS (a separate, hardwired or certified safety PLC
trip system) is a per-process PHA/LOPA decision (see
docs/architecture.md and
ADR 0011).
| Requirement | Status | Implementation |
|---|---|---|
| Interlocking as basic control (Clause 3.1.5, 5.2.1) | Implemented | ILCK input + safeValue/interlockAddress/interlockInvert params on DO/AO blocks (pkg/fbruntime/blocks/); forced safe state evaluated every scan on the edge; ILCK_ACTIVE output for annunciation; fail-safe trip on unreadable signal. A single block's runtime error no longer halts the scan — interlocks keep evaluating for every healthy output while the program runs Degraded (ADR 0009) |
| Basic control at the control module level (Clause 3.1.13) | Implemented | Library ControlModuleTemplates pass the interlock parameters through to their output blocks; instances opt in via spec.parameters.interlockAddress |
| Interlocking is authored where equipment is driven (Clause 5.2.1, 5.2.2.3, 6.6.3) | Implemented | Clause 5.2.1 places interlocking inside basic control, Clause 5.2.2.3 says basic control in a unit is performed by the equipment and control modules within it, and Clause 6.6.3 separates the recipe from the equipment that carries it out. So a transitions[].interlock guard is authored on a phase chart or the Unit's armed safe-state chart and refused above them, on create and on update, by the procedural.dcs.io admission webhooks (ADR 0069). A step above the phase names a child procedural element and drives no equipment, so the flag stored and armed nothing while the change-control diff rendered it as a safety claim |
| A step above the phase names a child procedural element (Clause 5.3) | Implemented | The procedural control model nests procedure, unit procedure, operation and phase, so a step on a chart above the phase stands for the child element one level down and does no work of its own. proceduralv1alpha1.ValidateAbovePhaseChart requires every step on an OperationTemplate, UnitProcedureTemplate or ProcedureTemplate chart to name that child, refused on create and on update by the procedural.dcs.io admission webhook and by the three gateway save paths (ADR 0070). The batch instantiator is the oracle: TestShippedProceduralTemplatesInstantiate drives every shipped template through CreateProceduralTree |
| Basic control modifiable by operator/procedural control (Clause 5.2.1) | Implemented | Trip-source overrides travel the audited tag-write path (interlock bypass, recorded); first-class gated bypass tracked as follow-up |
| Requirement | Status | Implementation |
|---|---|---|
Multi-stage exception escalation (ProcessException) |
Not implemented (removed in #863) | The dedicated Clause 7.4 escalation kind (alarm-linked scope/severity/escalation-timer policies) was runtime-generated in theory but unfed and invisible in practice — purged; re-add on customer demand. The live exception path is AlarmDefinition.spec.exceptionAction below |
| Alarm audit trail | Implemented | All alarm controllers create AuditRecords for: activation, acknowledgment, shelve expiry, auto-clear, and re-activation events (21 CFR Part 11 compliance) |
| Reactive auto-HOLD (AlarmDefinition) | Implemented | AlarmDefinition.spec.exceptionAction (Hold/Stop/Abort) issues command to affected batch procedures on alarm trigger |
| Reactive auto-HOLD (runtime crash) | Implemented | Unit controller detects RuntimeReady true-to-false while Running+Allocated and issues Hold |
| Reactive auto-HOLD (IOModule fault) | Implemented | IOModule controller detects Online-to-Fault/Offline transition, annotates affected units with Hold command. Unknown is deliberately outside the set: it means the io-probe did not answer, so nothing measured the equipment, and the probe is a monitoring sidecar the runtime driving the unit does not depend on (ADR 0053). A device that drops mid-batch still reads Fault from a working probe and still holds the unit |
| FB program halt / single-block fault → fail-safe outputs | Implemented | A single block's runtime error degrades the scan without halting it, so basic control for the equipment module does not collapse on one faulting output; deliberate program removal drives every output to its safeValue; ungraceful process death is backstopped by the remote-I/O comm-loss watchdog. See ADR 0009 and HA and Failure Modes |
| Control equipment malfunction → Hold → Restart (Clause 7.4 canonical exception) | Implemented | Edge-runtime loss is the canonical Clause 7.4 control-equipment-malfunction event. The unit lands in Held (watchdog/self-hold or runtime-crash auto-HOLD above); edge-runtime failover (ADR 0006) re-binds the unit's runtime to a designated standby node — manual fenced failover (Autonomy) or automatic lease-based (Failover) — and recovery is the standard ISA-88 Restart whose recipe-defined restarting logic re-establishes process conditions. Under a control-plane partition the held unit drives a deliberate sequenced safe state, and never freezes at last value (ADR 0008): an SFC hold program armed at the edge — the running phase's holdingChart, else the UnitSpec.safeStateChart baseline — runs in an SFC engine embedded in the runtime when the control-plane heartbeat watchdog fires, writing the unit's own control-module tag space (the FB scan stays the sole I/O writer). Autonomy holds and keeps holding; Failover holds for a bounded sequence then self-fences. The "outputs hold last value / device fail-safe" path now narrows to the dead-node or no-armed-chart case (no software runs to sequence). The running phase survives in Phase.Status; on reconnect the runtime's self-held status reconciles to Held and recovery requires an explicit Restart. A dead runtime is the case the armed chart does not cover, because arming is in-process state that goes with the pod, and a phase whose holdingChart could not run for the whole Holding transition settles Held with its declared posture never commanded. That deferral is recorded in Phase.status.holdingActionDeferredSince and annunciated as a distinct hold-deferred alarm, and the first reconnect commands the deferred holding action and re-arms the phase chart at the edge (ADR 0048, #1397). The run is driven against a state machine positioned in HOLDING, so it is the interrupted transition completing late: Clause 7.5.1 makes HELD a waiting state that has "previously achieved a defined set of conditions" and "is not permitted to direct any immediate actions", and the deferral is a Clause 7.4 sub-state of HELD, where a plain HELD is what the platform could not substantiate. Recovery is unchanged — still an explicit Restart. Failover decisions, fence confirmations, and edge hold-lifecycle events are audit-logged, and a failover during a running batch is additionally materialised into that batch's production record with its control and data gaps bounded separately (#1308, ADR 0041). See HA and Failure Modes, ADR 0006, and ADR 0008 |
| Control-system absence during a timed dwell (Clause 7.4) | Implemented | A step's elapsed time (StepName.T) is wall clock from its activation, and the activation time is part of the persisted position that lets a phase resume where it was. An interval in which no scan ran therefore counted as dwell time, so an eleven-minute power loss satisfied a 120-second hold in two scans and the batch ran to Complete unattended (#1685, found by the #1349 cold-boot drill). A timed dwell is where a recipe asserts the product spent N minutes under stated conditions, so counting an interval nothing supervised falsifies that assertion in the batch record. A dwell now counts only time in which a scan of its chart ran. The SFC engine stamps each scan into status.sfcStatus.lastScanAt, and a run resuming from that position measures the distance before re-entering the chart; a distance longer than the one-minute runtime-outage grace is a control gap. steps[].onControlGap carries the verdict, defaulting to Hold: Hold takes the interval back out of the dwell and self-holds for an operator ruling, Extend takes it back out and carries on, Fail aborts the phase, and Count counts it for a dwell anchored to wall clock. The gap is recorded in Phase.status.controlGap, audited, alarmed and observed as dcs_phase_control_gap_seconds for every verdict. This is distinct from the Clause 7.4 control-equipment-malfunction row above, which covers the equipment losing its controller; here the controller is what went away, and ADR 0008's armed hold cannot cover it because the armed program needs a live process. See ADR 0067 |
| Edge-armable hold-chart constraint (Clause 7.4 safe-state placement) | Implemented | An SFC chart staged as a partition-triggered local-hold program may only READ/WRITE the unit's own control-module tag space (no cross-unit, no control-plane-only data) — the runtime cannot reach those tags during a partition. It must also call only builtins that execute with no control plane: PROMPT*, MODE, COMMAND, STEP_ACTIVE, CALL_SERVICE and the three MTP builtins each need an operator, the apiserver or another machine, so a chart calling one ran correctly on the control-plane path and errored mid-hold only during the partition it was armed for (#1406, ADR 0008 amendment). ValidateEdgeArmableChart (api/procedural/v1alpha1/edge_armable.go) is the single validator; the runtime applies the builtin half when a chart is staged and backstops the tag half at resolution time (internal/adapter/hold.go), and make lint-edge-armable enforces both statically over the example corpus. A chart that fails the constraint falls back to the unit baseline and the operator surfaces a warning. An armed unit baseline chart is where a future exception-policy safe state (the purged ProcessException.safeState idea, #863) could actually run against equipment |
| Tag-based alarm conditions | Implemented | AlarmDefinition evaluates TagHigh, TagLow, TagHighHigh, TagLowLow, TagDeviation conditions by reading live process values from unit runtime HTTP API; deadband hysteresis prevents chatter; supports exceptionAction for auto-HOLD/Stop/Abort |
| Phase-scoped (state-based) alarm suppression | Implemented | AlarmDefinition.spec.armingGate (issue #232) suppresses evaluation unless a Phase matching phaseTemplateRef/unitRef is in one of the gate's allowed ISA-88 states (default [Running]; Idle/Complete/Stopped/Aborted rejected by enum). Surfaces status.armed, status.gatedBy, and the Suppressed condition (Disabled/NotArmed) — distinct from out-of-service (spec.enabled) and operator shelving per ISA-18.2. Design: phase-scoped alarm arming. |
| Operator alarm shelving (ISA-18.2 annunciation suppression) | Implemented | POST /alarms/{name}/shelve (CLI dcs alarm shelve, PermOperate) sets Alarm.status.shelvedUntil; gated to ActiveAcknowledged only (cannot suppress an unseen alarm or pre-shelve a cleared/idle one — 409 otherwise). Suppression is system-wide: the shelve is carried on the retained MQTT alarm event (shelved/shelvedUntil), and annunciation consumers (HMI nav icon/card border/faceplate strip; external annunciators/SCADA) drop it from annunciation while keeping it in list/summary views. The AlarmReconciler auto-unshelves on expiry and re-announces. Temporary, audited, auto-returning — the operator-grade tier distinct from out-of-service (spec.enabled) and state-based suppression (armingGate). See ADR 0014. |
Recipe Transformation (ISA-88.00.03-2003 Clause 8)¶
Not implemented (removed in #858/#859, originally surface-reduced in #480). The Clause 8 transformation engine and the transform-component library kind were purged together with the General/Site recipe levels they transformed. Re-add on customer demand, since git history has the engine, CRD, and controller.
Production Planning & Scheduling (Clause 8.4 / Part 2 Clause 4.5)¶
Implemented (no UI surface). Batch-schedule CRD
(batchschedules.batch.dcs.io) with a reconciler
(internal/controller/batch/batchschedule_controller.go, registered in
cmd/batch-operator/main.go) that drives the schedule-entry lifecycle
(Proposed > Planned > Committed > Started > Completed), performs equipment
conflict detection (internal/controller/batch/schedule_conflict.go), and
auto-creates Batch CRs at scheduled times. Campaign CRD
(campaigns.batch.dcs.io) with a reconciler
(internal/controller/batch/campaign_controller.go) groups schedules and
aggregates their status. The UI/API/CLI surface was removed in #480.
Schedules and campaigns are authored via kubectl/GitOps against the retained
CRDs.
Coordination Control (Clause 7.2)¶
ISA-88 Part 1 Clause 7.2 defines coordination control as distinct from basic and procedural control. It directs, initiates, and/or modifies the execution of procedural control and the use of equipment entities.
| Requirement | Status | Implementation |
|---|---|---|
| Signal-based cross-unit coordination | Implemented | CoordinationBlock CRD (Mode=Signal) — producer raises signal when phase reaches Running; consumers are notified via annotation and resume from Held state |
| Phase coordination integration | Implemented | Phase controller checks for dcs.io/coordination-signal/* annotations; enters Held state when CoordinationBlockRef is set, resumes on signal |
| Timeout handling | Implemented | Configurable timeoutSeconds with automatic transition to TimedOut phase |
| Audit trail | Implemented | AuditRecord created for signal satisfaction and timeout events |
Control Module Continuous Observability¶
ISA-88 Part 1 defines Control Modules as the smallest addressable equipment
entities providing continuous regulatory control. Their process values
(PVs, valve positions, sensor readings) must remain observable to operators
independent of procedural execution. A transmitter's temperature is a fact
of the plant whether or not a phase is currently reading it. Issue #192
traced a gap where CM PVs rendered as silent -- whenever the CM's FB
network was not running (deployment race, compile error, runtime crash),
violating this principle.
| Requirement | Status | Implementation |
|---|---|---|
| Continuous CM PV publishing | Implemented | Runtime raw-I/O fallback loop (internal/adapter/raw_fallback.go) publishes driver-direct reads at 1 Hz with quality: "Raw" whenever a CM's FB network is not Running; strict precedence prevents overriding live engineering-scaled values |
| Per-CM FB-network health signal | Implemented | Retained MQTT topic dcs/{ns}/equipment/controlmodule/{cm}/health with states Running/Down/CompileError/Unknown; published on transitions by the runtime (internal/adapter/cm_health.go) |
| HMI three-state rendering | Implemented | Gateway forwards health to HMI via cmhealth WS channel; faceplate shows CM health banner and RAW quality chip on affected tag rows (internal/gateway/static/js/hmi/hmi-faceplate.js) |
| Health transition audit trail | Implemented | Historian persists every transition to cm_health_events hypertable (pkg/historian/migrations/003_cm_health_events.sql) with batch correlation |
| Commissioning visibility (pre-deploy) | Implemented | Raw fallback activates whenever CM FB network is Unknown/Down, so PVs are visible from unit-runtime startup — before any ControlProgram exists |
| Pharma-appropriate labeling | Implemented | Driver-direct reads always carry quality: "Raw" and are surfaced as a distinct amber HMI chip; operators can never confuse unscaled driver output with an engineering-unit PV (21 CFR Part 11 data-integrity principle) |
Gap Analysis¶
| Gap | Priority | Description |
|---|---|---|
| Equipment property unit-of-measure conversion | Low | Property constraint validation uses raw numeric comparison; no engineering unit conversion (both sides must use consistent units) |
| ~~Recipe versioning~~ | ~~Low~~ | ~~Resolved: RecipeRevision CRD (reciperevisions.recipe.dcs.io) snapshots the master recipe spec on approval (ensureRevision in internal/controller/recipe/masterrecipe_controller.go); gateway exposes list/get/diff/revert revision endpoints (/api/v1/sites/{site}/recipes/master/{name}/revisions); dcs diff renders the same semantic diff between two CRD YAML revisions from the CLI (cmd/dcs/internal/cmd/diff.go)~~ |