PQ)¶

Cloud-Native DCS ships executable Installation, Operational, and Performance Qualification protocols as first-class CLI subcommands: dcs qualify iq, dcs qualify oq, and dcs qualify pq. These replace the traditional blank markdown templates that every vendor ships and that exercise nothing in the running system.

The thesis is simple: a cloud-native DCS can qualify itself in a way traditional DCS cannot. Kubernetes already has authoritative state for most IQ checks, the state machine already drives OQ, and the historian already captures the data PQ needs. Turning qualification into code makes it re-runnable after every deploy, retiring the one-time pre-production paperwork exercise.

Authority¶

21 CFR Part 11 §11.10(a): "Validation of systems to ensure accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records."

This document is referenced from the 21 CFR Part 11 traceability matrix. Within a GAMP 5 computerized-system-validation lifecycle, these protocols are the executable right-leg (IQ/OQ/PQ) verification artifacts.

Protocol Overview¶

Protocol	What it verifies	How	Status
IQ (Installation)	The system is correctly installed and configured	Reads live cluster state, compares to approved baseline	Implemented
OQ (Operational)	The system operates per spec across its state machine	Drives canned batches through ISA-88 commands, asserts AuditRecord per transition	Implemented
PQ (Performance)	The system produces consistent output over repeated runs	Runs N batches on the target site, computes Cpk on a CQA	Implemented
DR (Disaster Recovery)	A post-restore cluster matches a known-good baseline	Reuses the IQ snapshot and evaluator against a stricter baseline (full CRD set, all operators); records evidence as `dr-*` AuditRecords	Implemented

The suite is on camera.

dcs qualify iq runs against a live cluster and its PASS report lands in the adjacent audit-trail pane as a signed Qualification record; dcs qualify oq drives the ISA-88 state machine while oq-batch StateTransition records accumulate in the trail, ending on the Command Coverage table; dcs qualify pq computes Mean, StdDev, and Cpk across three batches to a CAPABLE verdict; a second IQ run and dcs audit export list the run history as ordinary audited records — Three protocols against a live deployment: `dcs qualify iq` passing against a quality-approved baseline with its report landing as a signed Qualification record, `dcs qualify oq` driving the state machine to its Command Coverage table, and `dcs qualify pq` computing Cpk across three batches to a CAPABLE verdict.

IQ — Installation Qualification¶

Command: dcs qualify iq [flags]

What gets checked¶

IQ reads a point-in-time snapshot of the cluster and evaluates it against an approved baseline. Checks cover:

Kubernetes version — must meet the baseline's minVersion floor
CRDs — every CRD listed in requiredCRDs must be present and served
Helm release — release name, chart version, lifecycle status, and a sha256 digest of user-supplied values (drift in the digest is a change-control red flag)
Required deployments — pods matching the baseline's name-prefixes must be Ready in their target namespaces, with container image digests captured
RBAC — a named ClusterRole's rules are canonicalized and hashed. Drift from the baseline hash indicates out-of-band edits. Cloud-Native DCS ships one ClusterRole per operator (cloud-native-dcs-gateway-role, cloud-native-dcs-physical-operator-role, etc.). The baseline therefore selects which role to pin by name. The default baseline leaves this category disabled. Customers opt in by setting clusterRole.name to the role they want under change control
PVCs — all listed PVCs must be Bound
ServiceAccounts — presence check for the platform SAs

Each check produces a PASS / FAIL / WARN row with an expected vs. actual comparison and a remediation hint on failure.

Baseline¶

Without --baseline, dcs qualify iq uses an embedded default suitable for internal smoke testing. For formal pharma validation, customers are expected to copy the default, edit it to match their approved configuration, sign it off with quality, and pass it explicitly:

dcs qualify iq --baseline ./validated-iq-baseline.yaml

The default baseline omits the clusterRole block entirely (skipping the RBAC category) and leaves the Helm values digest blank. To enable RBAC drift detection, copy the baseline, pick one role to pin, and paste the observed hash from a known-good cluster:

clusterRole:
  name: cloud-native-dcs-gateway-role
  expectedHash: "sha256:..."

# capture the hash from a running cluster
dcs qualify iq --format json | jq '.snapshot.clusterRole.hash'
dcs qualify iq --format json | jq '.snapshot.helm.valuesDigest'

Once pinned, any drift in these values between IQ runs fails the check. If you installed the chart with a non-default Helm release name, also rewrite the requiredPods[].namePrefix entries and clusterRole.name to match your release prefix.

Output and retention¶

Reports are rendered to stdout (or --output <file>) as either Markdown (the default, suitable for regulatory submission) or JSON (for CI/CD consumption). In addition, every run writes an AuditRecord to the dcs-system namespace with:

Target.Kind = Qualification
Target.APIGroup = compliance.dcs.io
Target.Name = iq-<runID>
CorrelationID = <runID>
Spec.Changes[0].NewValue = gzipped + base64-encoded full report JSON

The full report is therefore retained in the audit trail and reconstructible via dcs audit. Past qualification runs are queryable the same way any other audited event is, and the audit-archiver CronJob moves old records from etcd to PostgreSQL on the normal retention schedule. The figure under Protocol Overview ends on exactly this query: a second IQ run followed by the export listing the run history.

# list all past qualification runs
dcs audit export --target-kind Qualification

# drill into one run
dcs audit trace <runID>

CI/CD gating¶

dcs qualify iq --format json --fail-on-deviation exits non-zero on any failed check, making it a drop-in post-rollout smoke test. Wire it into the deployment pipeline immediately after a Helm upgrade to catch regressions before they hit production:

helm upgrade cloud-native-dcs ... --wait
dcs qualify iq --fail-on-deviation

This is the value-add that traditional DCS can't replicate. IQ stops being a piece of paperwork signed once at commissioning and becomes a check that runs after every deploy.

OQ — Operational Qualification¶

Command: dcs qualify oq [flags]

OQ drives canned batches through the ISA-88 state machine and records what happened. Each invocation runs a scenario set (a list of named command/expected-state sequences) against a target unit and capability, creating a fresh Batch per scenario so correlation IDs stay disjoint.

Default scenarios¶

The shipped scenario set covers 8 of the 8 external ISA-88 commands across five complementary flows. Every flow ends at a terminal state (Running, Aborted, Stopped, or Complete) and threads an AuditRecord per transition under a per-scenario correlationID.

Scenario	Steps	Commands exercised
`happy-path`	Idle → Running → Held → Running	Start, Hold, Restart
`abort-path`	Idle → Running → Aborted	Start, Abort
`pause-resume-path`	Idle → Running → Paused → Running → Complete	Start, Pause, Resume
`stop-path`	Idle → Running → Stopped	Start, Stop
`reset-path`	Idle → Running → Stopped → Reset rejected	Start, Stop, Reset

reset-path chains a Stop step internally to set up the Stopped state that Reset targets. This duplicates the stop-path command sequence by design, so each scenario is self-contained for auditor review. The Reset step qualifies the documented Batch command surface, where no state transition exists to drive: batches are one-time execution instances (re-executing under the same batch ID would corrupt the batch production record), so a terminal batch refuses every command and writes an Action=Command, Result=Rejected AuditRecord. The step passes only when that rejection record lands and the batch state is unchanged. The ISA-88 Part 1 example state model's Reset → Idle transition applies to procedural elements and equipment (units auto-reset and are re-allocated). It does not apply to the Batch entity (#1016).

The OQ Markdown report now opens with a Command Coverage table that lists every external ISA-88 command, the scenarios that exercise it, and the pass/fail/warn breakdown. Auditors can confirm 8-of-8 coverage at a glance without walking each scenario's per-step table.

Customer-defined scenarios¶

DefaultOQScenarios is the shipped baseline. Customers with site-specific state-transition requirements (e.g., a pharmaceutical line that requires Pause to land within a fixed wall-clock window per Annex 11 §6) can extend the driver with additional scenarios via the public OQDriver.Scenarios field:

import "github.com/cloud-native-dcs/cloud-native-dcs/pkg/qualify"

driver := &qualify.OQDriver{
    // ...site/unit/capability...
    Scenarios: append(qualify.DefaultOQScenarios,
        qualify.OQScenarioDefinition{
            Name:        "pause-within-window",
            Description: "Pause must land within 2s (Annex 11 §6)",
            Steps: []qualify.OQScenarioStep{
                {Command: "Start", ExpectedState: "Running"},
                {Command: "Pause", ExpectedState: "Paused"},
                // ...site-specific timing assertions...
            },
        },
    ),
}

Custom scenarios appear in the same Markdown report and Command Coverage table as the defaults, and are persisted into the Qualification AuditRecord identically. There is no second-class path for customer extensions. The Scenarios field is a wholesale replacement when set. To keep the defaults, prepend or append them as shown above.

Fixture¶

OQ creates a minimal PhaseTemplate per run with an ActionChart that pauses in its first step for PhaseHoldSeconds seconds (default 10), using a timed transition (wait.T > T#10s). The hold window gives Hold, Restart, and Abort commands a real window to land before natural phase completion fires. Without it, an ActionChart-less phase auto-completes before the command can propagate through the Batch → Procedure → UnitProcedure → Operation → Phase hierarchy.

The MasterRecipe and Batch are scoped to the user's selected site namespace and cleaned up after the run unless --keep-artifacts is set.

What each step records¶

For every step the driver:

Sends the ISA-88 command via the existing POST /api/v1/sites/{site}/command endpoint
Polls the batch state until it matches the expected state. On a scenario's final step an accepted transitional state (Holding for Held, Restarting for Running, Aborting for Aborted, etc.) also satisfies the check. A transitional observation is legitimate evidence that the state machine is progressing correctly, since the procedural hierarchy can take tens of seconds to settle an SC (state-complete) transition. Mid-scenario steps wait for the resting state itself, because the next command is issued from whatever state the driver broke out on and ISA-88 permits it only from the resting state (#1011)
Queries the AuditRecord API for a record whose status.phase change lands in the accepted-states set for the target, and captures its name in the report
Records the observed state, the transition latency in milliseconds, and a Pass/Fail/Warn status

Serialized scenario execution¶

Every scenario runs against the same unit, requesting one unit of the target capability. Before creating each scenario's Batch, the driver waits for that unit to be allocatable again (Idle and claimed by no batch), and only then submits the next batch.

The barrier is load-bearing because releasing a unit is not a single write. The batch controller Stops the allocated unit, deallocates, and clears both the claim annotation and status.allocatedTo. The unit controller (a separate operator, on a later reconcile) then auto-Resets the unit from Stopped/Complete/Aborted back to Idle, and deliberately holds at Stopped while the unit is still allocated. That chain spans at least three reconciles across two operators, while the driver's per-scenario cleanup (DeleteBatch) is fire-and-forget. Without the barrier, the next scenario's Start lands against a unit that is still held, the batch sits in Allocating, and the wait is charged against the step budget, surfacing as an intermittent, position-dependent transition timeout (#996).

The wait has its own budget (60s, separate from --timeout) precisely so unit release is never charged against transition time. If it expires, the scenario fails with a message naming the unit and its current holder, where the old failure was an ambiguous state-transition timeout. Scenario 0 pays the same check, which catches a cluster left dirty by an earlier interrupted run.

PQ applies the identical barrier between replicates, for the same reason.

When a batch does time out in Allocating, the driver reads the batch's allocation-block diagnostic and appends it to the failure detail, so contention ("capability X is held … by batch Y") reads differently from genuine no-capacity ("no unit is available with capability X … this is a capacity problem, not contention"). Both previously rendered as the same bare timeout message.

Target discovery¶

By default OQ auto-discovers the first unit with at least one capability on the selected site. --unit, --capability, --process-cell, and --namespace override any or all of these. A fresh MasterRecipe, PhaseTemplate, and Batch are created per run in namespace and cleaned up on exit. Pass --keep-artifacts to leave them behind for post-mortem debugging.

Output and CI gating¶

--format md (default) emits a human-readable Markdown report with a per-step table and a summary block. --format json emits the raw OQReport struct. --fail-on-deviation exits non-zero if any step failed, so OQ can run as a deploy-gate check the same way IQ does:

helm upgrade cloud-native-dcs ... --wait
dcs qualify iq --fail-on-deviation
dcs qualify oq --fail-on-deviation

Audit persistence¶

Every OQ run is persisted as a Qualification AuditRecord via the POST /api/v1/qualify/oq/record endpoint. The wire format matches IQ. The full OQReport is carried as gzipped+base64 JSON in the record's Changes[0].NewValue, Target.Kind=Qualification, and Target.Name=oq-{runID}. This means dcs audit and the audit archiver both handle OQ records with no code paths added. The record shape is uniform across IQ, OQ, and (eventually) PQ. Pass --skip-record to bypass persistence during debugging.

PQ — Performance Qualification¶

Command: dcs qualify pq [flags]

PQ measures whether the system produces consistent, capable output over repeated runs against a Critical Quality Attribute (CQA). It reuses OQ's fixture primitives (same minimal PhaseTemplate + MasterRecipe, same Batch apply path) and adds two PQ-specific concerns: per-batch CQA querying from the historian, and process-capability statistics across the run.

Flow per batch¶

Apply a fresh PQ-prefixed Batch referencing the shared MasterRecipe.
Send Start and wait for the batch to reach Running.
Hold the batch alive for a configurable measurement window (default 15s). The historian's tag-ingestion loop maps namespace → active batchID once per collection cycle (typically every 5–10s), so the window must span at least one cycle to capture tagged samples.
Query GET /api/v1/historian/batch/{batchID}/tags?address=...&agg=last for the CQA value. Poll up to --historian-poll-timeout (default 15s) until a value appears.
Tear down the batch (unless --keep-artifacts).

PQ deliberately does not wait for the batch to reach Complete, for two reasons. Auto-completing phases finish faster than the historian's collection cycle, so a fast Complete leaves the historian with no tagged samples. And phases that depend on the unit-runtime SFC bridge self-hold on bridge errors (TLS expiry, network partition) and never reach Complete at all. Measuring during Running decouples PQ from both failure modes.

Statistics¶

Across all batches that produced a CQA value:

Mean — arithmetic average
StdDev — sample standard deviation (n-1 denominator)
Min / Max
Cpk — min((USL - mean) / (3σ), (mean - LSL) / (3σ)) per Montgomery's Introduction to Statistical Quality Control. One-sided specs are supported by passing only --lsl or only --usl.

Verdict¶

When --lsl and/or --usl are configured, PQ assigns a verdict:

CAPABLE when Cpk ≥ --cpk-min (default 1.33 per FDA guidance for new processes)
NOT_CAPABLE when Cpk < --cpk-min
UNKNOWN when no spec limits are configured (the report still computes mean/stddev for trending, but cannot conclude capability)

--fail-on-deviation exits non-zero on NOT_CAPABLE, making PQ a deploy-gate check the same way IQ and OQ are.

Output¶

--format md (default) renders a Markdown report with:

CQA spec table (address, target, LSL, USL, Cpk minimum)
Statistics table (N, mean, stddev, min, max, Cpk)
ASCII trend chart with horizontal reference lines for target and spec limits (no chart-rendering dependency added). The chart embeds in a fenced code block and diffs cleanly between runs
Per-batch table (batchID, CQA value, status)
Deviation log for any batches that failed to produce a value
BPR pointer (dcs batch record <batchName>) for the first batch

--format json emits the raw PQReport struct for CI/CD consumption.

Audit persistence¶

Every PQ run is persisted as a Qualification AuditRecord via POST /api/v1/qualify/pq/record. Same wire format as IQ and OQ (Target.Kind=Qualification, Target.Name=pq-{runID}, gzipped report in Changes[0].NewValue). Historical capability is therefore queryable via dcs audit export --target-kind Qualification, and the audit archiver handles PostgreSQL retention with no PQ-specific code paths.

Operational requirements¶

PQ requires the historian to be receiving tag samples for the target unit during the measurement window. On clusters where the unit-runtime or its tag-publishing path is broken (expired runtime mTLS, MQTT broker down, etc.) the PQ batches will reach Running and produce a complete report shape, but every batch row will be FAIL: no historian value. This is the expected behavior: PQ surfaces upstream observability gaps and fabricates no capability data.

DR — Disaster-Recovery Restore Qualification¶

dcs qualify dr validates that a post-restore cluster matches a known-good baseline. It reuses the IQ snapshot endpoint and evaluator against a stricter baseline (the full CRD set, all operators required) and prefixes its AuditRecord Target.Name with dr- so drill evidence is discoverable separately from installation qualifications. The drill procedure, cadence, and recorded runs live in the DR runbook. The 2026-04-22 cert-expiry and archive-corruption drills both seal with a dcs qualify dr --fail-on-deviation run.

Design notes¶

Why executable? Blank markdown templates were removed in commit ab00bd3. They exercised nothing in the running system and tracked a feature we didn't yet provide. Executable protocols re-raise §11.10(a) with something we can actually defend in an audit: the protocol is the code, and the code runs on the live system.

Why gateway-mediated? The dcs CLI is a thin HTTP client by design. Every operational action already flows through the gateway. IQ needs low-level Kubernetes reads (discovery, CRDs, ClusterRoles). The gateway exposes them as dedicated endpoints, and the CLI stays free of a kubeconfig dependency.

Why reuse AuditActionCreate with a free-form Target.Kind? Adding a new AuditActionQualify enum value would force CRD regeneration, webhook schema updates, and archiver changes. Using the existing Create action with a synthetic target kind (compliance.dcs.io/Qualification) achieves the same filtering and retention semantics at zero schema-churn cost.

Why gzip the report? IQ reports are ~2-5 KB after gzip, well under the 64 KB etcd soft cap and small enough to carry inline in AuditRecord.Spec.Changes. The round-trip is verified by qualify.DecompressReportBase64, and the audit archiver handles second-stage PostgreSQL retention automatically.