Skip to content

21 CFR Part 11 Compliance Traceability

Standard: FDA 21 CFR Part 11, "Electronic Records; Electronic Signatures" Applicability: Pharmaceutical batch manufacturing records, recipe approval, control logic changes, equipment configuration Overall Status: Substantially Implemented (see Gap Analysis for the remaining items)

EU GMP Annex 11 and PIC/S Annex 11

The controls in this matrix also substantially cover EU GMP Annex 11 (Computerised Systems) requirements on electronic records, audit trails, electronic signatures, access control, and change management. This matrix is written against 21 CFR Part 11 wording. An Annex-11-specific traceability matrix can be derived from the same implementation evidence on request. PIC/S Annex 11 uses the same wording as EU Annex 11.

Status Legend

Indicator Meaning
Implemented Requirement is fully implemented with tests and evidence
Partial Core functionality exists but gaps remain
Planned Architecture is designed but implementation has not started
N/A Requirement does not apply to this system

GAMP 5 Categorization

Cloud-Native DCS is designed for deployment as a GAMP 5 Category 4 (configured product) system in regulated environments. Customer-authored recipes, equipment hierarchies, alarm definitions, and phase templates are configuration applied via declarative Kubernetes resources. They are not custom code. The full per-layer categorization rationale (including the Category 5 boundary for forked images) and the mapping of product artifacts to the GAMP 5 V-model validation lifecycle live in the dedicated GAMP 5 Computerized System Validation document.

Subpart B -- Electronic Records

Section 11.10 -- Controls for Closed Systems

Requirement Description Status Implementation
§11.10(a) System validation for accuracy, reliability, and consistent performance Implemented CI pipeline with unit tests, e2e tests in Kind cluster; full executable IQ/OQ/PQ protocol suite shipped (dcs qualify iq, dcs qualify oq, dcs qualify pq) — see validation.md. OQ covers all 8 external ISA-88 commands (Start, Hold, Restart, Abort, Pause, Resume, Stop, Reset) across five scenarios (happy-path, abort-path, pause-resume-path, stop-path, reset-path) and threads every state transition to an immutable StateTransition AuditRecord. PQ measures CQA values during a configurable measurement window, computes Mean/StdDev/Cpk across N batches, and emits a CAPABLE/NOT_CAPABLE verdict against --cpk-min (default 1.33 per FDA guidance)
§11.10(b) Generate accurate and complete copies of records in human-readable and electronic form Implemented AuditRecord CRD stores immutable records; Gateway REST API returns JSON; CLI dcs audit trace displays chronological trail. At batch terminal the BatchRecord reconciler materialises phase history, operator actions, input-parameter values with recipe-layer source attribution, process outputs from chart snapshots, and a frozen control-recipe snapshot into the CRD spec, so the entire Part 4 Clause 5 record (not just the batch metadata) is covered by the HMAC-SHA256 finalization signature. Outputs are declarative-only: every value the BPR records must be declared on spec.outputs[]; the chart's ST is the only producer, and outputs the chart never assigned to are omitted from the snapshot — the BPR cannot disguise a setpoint as a measurement (issues #247, #248, #250). BatchRecordParameter.Source carries recipe-layer provenance (formula-scaled / override / default) for input rows; output rows live on a separate Outputs list with no provenance enum. Gateway consolidated BPR endpoint exports JSON or BatchML XML. Export completeness (#1299): "complete copies" is a claim a backup file has to be able to substantiate, so every dcs backup crds / dcs backup audit export ends with a machine-readable completeness footer (dcs-export-complete, dcs-export-skipped) naming any kind the gateway could not read and why (crd-not-installed / forbidden / error). Before this, a missing RBAC rule made the omission indistinguishable from a cluster that held no such objects — the gateway ClusterRole had no coordinationblocks rule, so every backup silently omitted the kind (#1222). A non-benign gap fails the CLI, is copied into the sidecar manifest, and is recorded in the audit trail as a failed export. Failover gaps are explained in the record (#1308): a controller failover during a running batch materialises into spec.failoverEvents[] at batch-terminal time, with the control gap (Hold → explicit Restart) and the data gap (lease fence → re-establishment) bounded separately and each bound referencing its evidencing AuditRecord, so the copy a QA reviewer reads discloses the gap without recourse to operator logs — distinct from the #1299 export-completeness footer: that footer discloses a failed read at export time, while failoverEvents disclose a gap in the underlying record itself (ADR 0041). The record's audit half is correlated by batch ID (#1460): operator actions, change history and failover evidence are matched by AuditRecord.spec.correlationID == Batch.spec.batchID, so accuracy depends on that ID naming one batch. It does — a Batch whose batchID is already held by another Batch in the site is refused at admission (ADR 0051). Before that was enforced, a work order resubmitted after a lost response created a second Batch under the same ID, and each twin's record then aggregated the other's operator entries. The record carries the conditions the batch did not choose (#1686): the batch's significant-event log is frozen into spec.executionEvents[] at batch-terminal time, because the consolidated view used to read it from the live Batch resource and lost every hold, abort and rejected command the moment that resource was garbage collected, which is exactly when the copy has to stand on its own
§11.10(c) Protection of records to enable accurate and ready retrieval throughout retention period Implemented and drilled (archival + tamper-evidence + scheduled verification + restore) Archival path — implemented and covered by unit/integration tests: Audit records stored in etcd for active period (default 90 days), then archived to historian PostgreSQL/TimescaleDB (default 3-year retention per 21 CFR Part 211). Archival CronJob moves records idempotently. Gateway queries both active (K8s) and archived (historian) in one merged list, and says which halves it could read: every audit list response carries an archive verdict of ok, unreachable or not-configured (#1648). Before that verdict existed the merge was silent, so a historian outage served an auditor a shorter trail with a 200 and a fully-archived range came back empty and unremarkable. The single-record route answers 503 for a record it could not look for, where it used to answer 404, dcs audit export exits non-zero on an export the gateway called short unless --allow-partial is passed, and the Audit Trail view paints the incompleteness above the table and above the empty state. CLI dcs audit export provides regulatory export in JSON/CSV. Continuous process data stored in historian with configurable retention (default 365 days). Archive tamper-evidence (issue #216): every archival batch produces one audit_archive_manifest row with a SHA-256 digest over the canonical-JSON concatenation of the batch's records and an HMAC-SHA256 signature over {namespace, batch range, record count, digest} using the rotating dcs-signing-key key store. Each record carries a manifest_id FK so the exact covered set is recoverable. CLI dcs audit verify --archived [--since --until] re-computes the digest, verifies the signature, and emits per-manifest pass/fail with specific reasons (digest_mismatch, signature_invalid, unknown_key, count_mismatch, missing_records); non-zero exit on any failure. Scheduled verification (Phase B): the gateway runs the same verify pipeline on a configurable interval (default 6h, floor 15m enforced by the scheduler) and writes one AuditRecord per run with Target.Kind=ArchiveIntegrityCheck and Actor.Component=gateway-archive-verifier; the failing manifest set is serialized into the record body so auditors can reconstruct the exact failure from the trail without re-running verification. Read-only "Archive Integrity" panel in the gateway UI surfaces the latest run plus history at /api/v1/audit/archive-integrity/{latest,history}. Immutable S3 mirror (Phase C, opt-in): with historian.audit.archival.immutable.enabled=true, every signed batch is additionally written to an S3-compatible bucket with Object Lock in Compliance mode and a 7-year default retention — no identity (root included) can overwrite or destroy that object version until the lock expires. Retention protects a version and not a name (#1455): an unversioned DeleteObject is still accepted and appends a delete marker that hides the bundle from any name-addressed read, so restore procedures list versions and read the newest non-marker one, and the bucket policy keeping s3:DeleteObject* away from the archiver's credentials is a stated requirement of the deployment. Mirror is in addition to PostgreSQL, and PG remains the query path. A failed upload blocks deletion of the source CR so the next archiver run retries idempotently. Bucket lifecycle + legal-hold policy documented in backup-recovery.md. Regression coverage (issue #222): chart-level E2E spec test/e2e/audit_mirror_test.go installs the full Helm chart against an in-cluster minio fixture with Object Lock enabled, runs the archiver CronJob end-to-end, asserts Retention-Mode=COMPLIANCE on the mirrored object, and verifies a version-targeted DELETE is rejected before expiry. Nightly via .github/workflows/e2e-audit-mirror.yml; opt-in locally via make test-e2e-audit-mirror. Restore path — drilled 2026-04-22: Scenario: Audit Archive Corruption walked end-to-end on a two-droplet k3s with in-cluster minio Object Lock. One manifest (3 records, key v1) corrupted via direct UPDATE, dcs audit verify --archived reported digest_mismatch in <1 s, quarantine AuditRecord audit-quarantine-zc8pl stamped, bundle pulled from minio via mc cp (X-Amz-Object-Lock-Mode: COMPLIANCE, retention 2027-04-22), transactional BEGIN/UPDATE×3/COMMIT restored canonical bytes, re-verify green. Full recovery RTO 124 s wall clock; signed qualification qual-20260422-192940-443c0887; deviation report audit-deviation-lzcxh. See drill log. Primary mitigation for DR row "Audit records (archived, in historian)" in the RTO/RPO table. Also covers the gateway archiveIntegrity scheduler's detection role in Phase B
§11.10(d) Limit system access to authorized individuals Implemented OIDC authentication on the gateway REST API and Web UI maps each request to an individual human identity; group-claim based authorization enforces the cumulative DCS role hierarchy (dcs-admin, dcs-supervisor, dcs-engineer, dcs-lead-operator, dcs-operator, dcs-viewer). Kubernetes RBAC with per-operator ClusterRoles backs the control plane; AuditRecords restricted to create-only. Refused access is evidence too: when the gateway's route gate denies a mutating action it writes a Rejected AuditRecord (category authorization-denied) naming the action, the deciding policy layer, and the caller, before the 403 reaches that caller, so an inspector reading the trail sees attempted actions an individual was not authorized for and not merely the ones that succeeded. Refused read-shaped actions — those any hyphen-separated segment of whose verb is read/list/browse/poll, e.g. site:read and the HMI's three-second-polled controlmodule:read-tag — are logged, and the trail omits them: a bounded, documented scope adopted so a misconfigured role against a polled view cannot flood the trail; see Security Operations § Authorization Denials (#1296)
§11.10(e) Secure, computer-generated, time-stamped audit trails that independently record operator entries and actions Implemented AuditRecord CRD captures: who (Actor — including the originating client sourceIP), what (Action, Target, Changes), when (Timestamp), why (Reason). The gateway resolves the client IP from forwarded headers (Cloudflare's CF-Connecting-IP, then X-Forwarded-For / X-Real-IP) and stashes it once per request, so attribution records the real client even when the gateway sits behind a trusted proxy, where the naive answer would be the reverse-proxy / tunnel-connector pod IP. Immutability enforced via RBAC (create-only) and validating webhook (rejects UPDATE/DELETE in production namespaces). The webhook is not optional in production: chart rendering fails if gateway.mode=production is combined with webhook.enabled=false or batchOperator.enabled=false (templates/webhook-configuration.yaml), so a production install cannot silently ship without the enforcement point. Free-text fields are size-capped at the schema level as defense-in-depth (message/errorMessage/reason 4096 characters, per-change values 1024 characters, 64 changes per record — far above the ~1.5 KB records observed in practice); a pathological producer is truncated with an explicit ...[truncated] marker before the apiserver, so an oversized field can never cause a lost record. Out-of-band mutations of compiled control-plane state are independently recorded: when the ControlModule compiler detects that a ControlProgram was changed behind its back (spec hash no longer matches the sealed compiled hash, #789), or the Unit reconciler detects the same for a unit's runtime ConfigMaps — the I/O driver wiring and the CM-scoped tag-access map (#796) — it emits a DriftCorrected AuditRecord — carrying the reverted fields and the foreign field manager(s) recovered from metadata.managedFields — before restoring the declared state, and withholds the correction if the record cannot be persisted (same fail-closed contract as change-control transitions). A Warning Event (OutOfBandChangeReverted) on the parent ControlModule or Unit surfaces the same fact to kubectl describe. See Archive-attested deletion model for the exact deletion contract
§11.10(e) Audit trail retained for at least as long as the electronic record Implemented AuditRecords kept in etcd for active period, then archived to PostgreSQL with configurable retention (default 1095 days = 3 years). Archival CronJob preserves all fields including electronic signatures. Etcd deletion is cryptographically gated on proof of durable archival (see archive-attested deletion model). Historian API exposes archived audit records for unified query
§11.10(f) Operational system checks to enforce permitted sequencing of steps and events Implemented ISA-88 state machine enforces valid command sequences (13 states, 8 commands); procedural controllers enforce step ordering; recipe derivation chain validated; batch size validated against recipe header min/max bounds
§11.10(g) Authority checks ensuring only authorized individuals can use the system, sign records, or alter records Implemented Gateway middleware enforces group-claim checks before every state-changing endpoint; recipe approval, rejection, and batch-record finalization require the appropriate dcs-supervisor / dcs-admin group membership and persist signer identity in the resulting ElectronicSignature. External-ref signature minting (POST /api/v1/signatures, the GitOps/ticket-bridge signer in change-control.md) is gated on Supervise; both mints and refused attempts are written to the audit trail with the external ref, and the change-control admission webhook independently re-verifies the HMAC against the applied object's external-ref annotation
§11.10(h) Device checks to determine validity of data input or operational instruction source Partial Device identity is established at the deployment layer (cluster node join, e.g. Talos machine identity — ADR 0004) and bound to a Controller resource by name; runtime health checks verify device availability
§11.10(i) Training for persons developing, maintaining, or using electronic record systems N/A Organizational policy; outside system scope
§11.10(j) Written policies for accountability of individuals using electronic signatures N/A Organizational policy; outside system scope
§11.10(k) Use of appropriate controls over systems documentation Implemented (deployment-opt-in) Four complementary controllers cover the §11.10(k) surface. Enforcement is operator-enabled and off by default: the chart ships changeControl.enforceOnProduction: false and the webhook gates only namespaces labelled dcs.io/production=true, so a deployment must flip the value and label its production namespaces for these controls to bind — in contrast to AuditRecord immutability (§11.10(e)), which the chart forces unconditionally. (1) Change-control webhook — mutating webhook gates ControlProgram / ControlModule mutations in namespaces labelled dcs.io/production=true: writes must arrive through an allowlisted ServiceAccount, carry a configured external-reference annotation (git commit SHA, ticket ID, etc.), and carry at least one HMAC electronic signature with meaning="approved". Every admission synchronously writes an AuditRecord with {sourceServiceAccount, externalRef, resourceSHA, signature} — no admit without audit. The webhook is self-referential (#643): it also gates ChangeControlPolicy CREATE/UPDATE/DELETE — cluster-scoped, so gated unconditionally, with no namespace label to scope it — under the current policy's allowlist + e-signature, so an actor with policy-edit RBAC cannot empty the allowlist, flip requireDistinctApprover, or enable break-glass without an approved, signed change. The single exemption is the CREATE that seeds the configured policy name, which cannot require a policy of its own; it is carved out in the rule's matchConditions so the apiserver never routes it, which is what lets a fresh install seed the policy before the webhook's own pod is ready. A CREATE under any other name is gated like an edit (#1550): such a policy is read by nothing only until changeControl.policy.name names it, so an ungated CREATE would let an unsigned policy be planted and wait for a rename to activate it. The gate admits an approved CREATE, which keeps the rename itself reachable as an ordinary approved change. See change-control.md. (2) GitOps-enforcement admission policy (#288) — ValidatingAdmissionPolicy that gates engineering-class CRD writes (plant structure, recipe definitions, alarm setpoints, control programs) when gitopsEnforcement.enabled=true. The policy admits writes from the configured Flux ServiceAccount, in-cluster DCS operator ServiceAccounts in the chart's install namespace (so reconcile loops can write Unit annotations and Phase status during normal operation — issue #372), or members of the cluster's break-glass ClusterRoleBinding; everything else is rejected with a message pointing at the ChangeRequest workflow. (3) ChangeRequest reconciler (#289) — drives the propose-approve-merge state machine for changecontrol.dcs.io/v1alpha1/ChangeRequest objects. Required HMAC signatures (approved plus deployment-configured extras like reviewed) are verified against the same key store used by the webhook before the request can advance from Proposed to Approved. Segregation of duties is enforced in-product across all backends: the gateway refuses to stamp an approved signature for the user who signed proposed (HTTP 403), and the reconciler independently refuses to count a self-approval toward the required meanings — governed by ChangeControlPolicy.spec.requireDistinctApprover (default true, enforced even when no policy object exists). Once approved, one of four pluggable backends realises the change: direct-apply (SSA against the API server), git-pr (open a PR against the configured repository, advance to Reconciled on merge with the merge commit SHA recorded), webhook (open a record in the customer's external quality system, poll it for a decision, and hand delivery to a configured direct-apply or git-pr delegate once it approves — #1424, ADR 0049), or in-cluster-queue (hold for an external change-management system). The webhook backend deliberately layers the external decision on top of the in-product e-signature, which it never replaces: backends run downstream of Proposed to Approved, so the required signatures are already collected and HMAC-verified before any external record is opened, and the §11.50 attribution therefore remains a MAC binding a named identity to the exact change content in a key store the deployment controls. The alternative it forecloses is a claim by a machine account about a decision made elsewhere. The record opened externally carries the verified signer identities (meaning, signer, signing time) and not the digests, which bind to that key store and prove nothing to a third party. An external record that closes as rejected, cancelled, voided or superseded moves the ChangeRequest to the terminal Rejected phase audited with Reject/Rejected. Failed is reserved for infrastructure faults, so a quality decision is never recorded as one; the closing state is preserved verbatim in status.resolution.externalRecordState. That rejection record carries no ElectronicSignature (#1483). The only signature the ChangeRequest holds by then is the approver's, whose meaning is approved, and stamping it onto a Rejected record would attribute the rejection under §11.50 to a person who did not make it; the Approved, Applied and Reconciled records carry the signature because those transitions are the approved change being carried out. For git-pr, the author-of-record on the resulting Git history is controlled explicitly, where the default would leave it to whoever minted the credential (#819, #830): every commit carries the ChangeRequest identity chain as machine-readable git trailers (Change-Request, Change-Request-UID, Proposed-by, Approved-by), so the human proposer and approver(s) survive in merged history regardless of the byline; gitPR.commitAuthor optionally pins a stable service identity as commit author+committer; and gitPR.authMode=github-app goes further — commits and pull requests attribute to the App's [bot] identity with GitHub's verified badge, authenticated by short-lived installation tokens with no standing human-owned credential, so staff turnover and token rotation never change the author of record. Scope note: under the default authMode=token, the pull request author remains the token owner even when commitAuthor is set — only App auth covers PR attribution. See ChangeRequest Backends § Commit identity. Every state transition emits an AuditRecord linking the original signature, the desired-spec SHA, and (for git-pr) the resulting commit; the Approved and Applied transitions are withheld if that record fails to persist (the reconcile-path analogue of the webhook's "no admit without audit", issue #642). (4) Audit-linkage receiver (#291) — the audit-flux-bridge Deployment receives reconcile events from Flux's notification-controller and emits one AuditRecord (spec.action="Reconcile") per applied object, written into that object's own namespace with the apply verb in the message, plus a wrapper record for the Kustomization itself; the per-object changeset is captured from the kustomize-controller's Progressing events and emitted when the subsequent ReconciliationSucceeded event supplies the authoritative revision (#955). Each record carries spec.gitCommitSHA, spec.gitPullRequestURL, and spec.changeRequestRef (when an originating ChangeRequest is found by matching status.resolution.gitCommitSHA + target GVK + namespace + name). Together with the e-signature recorded as a dcs.io/esig-approved annotation on the target object at admission time, this closes the inspector-traceable chain from cluster state to Git commit to approving signature, scriptable from either direction with kubectl get auditrecord. Idempotence on at-least-once webhook re-delivery is provided by an in-process (target, SHA) dedupe with a 30-minute TTL. Together the four layers ensure systems documentation cannot be modified outside the cluster's GitOps contract regardless of which CRD shape carries it, and any change in cluster state can be traced back to a specific Git commit and its approving signature. See Turning On GitOps Enforcement, ChangeRequest Backends, Audit Linkage, and GitOps Enforcement RFC

Several of the controls in this table are filmed operating in the product. Each figure names the clause row it evidences.

§11.10(a): dcs qualify iq run against the live cluster, its PASS report persisting as a signed Qualification AuditRecord, and OQ's per-command StateTransition records accumulating in the trail as the suite runs.
§11.10(b): the batch production record read section by section as one human-readable copy, then the §11.50 ceremony executed: decision, required reason, Approve & Finalize, and the record's Finalized status.
§11.10(e) and §11.10(c): a Hold captured through its reason ceremony and read back seconds later with actor and typed reason, then Archive Integrity reading Clean over the signed manifests and dcs audit verify --archived re-proving the chain. The same clip carries the §11.70 evidence: the signature located in the trail with its signer, meaning, and HMAC digest.
§11.10(e), the refusals: a record's spec edit and its deletion both denied by the webhook with the clause in the message, a hand-forged archival attestation denied on its recomputed HMAC, and the archiver's attested path retiring every record, the forged one included once the archiver recomputes the same HMAC and re-archives it under a genuine signature. The closing beat re-proves the archive per §11.10(c).
§11.10(e), the out-of-band half: a ControlProgram changed behind the operator's back, reverted to the declared value, and the DriftCorrected AuditRecord naming the foreign field manager.
§11.10(d): four directory personas working the same screens, each stopped at the next rung of the role hierarchy, one by a withheld affordance and the others by a server-side refusal naming the action.
§11.10(f): the command bar re-deriving itself at every ISA-88 state transition, and disappearing once the batch is terminal, so an out-of-sequence command is never offered.
§11.10(g): every recipe lifecycle action taken by an authenticated supervisor through its own reason-gated ceremony, with Approve's confirm held disabled until the reason satisfies the deployment's reason policy.
§11.10(k): an approved recipe change leaving the cluster as a real pull request: e-signature, rendered YAML diff, merge as the deployment gate, and the ChangeRequest closing out as Reconciled.

Section 11.30 -- Controls for Open Systems

Requirement Description Status Implementation
§11.30 Open system controls including encryption of records N/A Cloud-Native DCS deployments are closed systems per §11.3(b)(4): single-tenant, on-premise Kubernetes clusters where system access is controlled by the persons responsible for the records; there is no hosted or multi-tenant offering. Should a site nonetheless operate components across an open network, the §11.10 controls already in place (mTLS/TLS transport encryption, HMAC-SHA256 record-bound signatures per §11.70) form the basis for the additional §11.30 measures, assessed per deployment

Section 11.50 -- Signature Manifestations

Requirement Description Status Implementation
§11.50(a) Signed records shall display the printed name, date/time of signing, and meaning of the signature Implemented ElectronicSignature struct captures SignerID, SignerName, SignedAt, Meaning, and Digest. Gateway approval/rejection handlers populate all fields. Batch record finalization uses the same e-signature mechanism with meaning "finalization". REST API and CLI display signer name, timestamp, and meaning. The signing ceremony itself also manifests the signature's meaning before the act (#688): the Review & Sign panel shows an attestation statement that tracks the Approved/Rejected decision, what is being signed, and a "Signing as" line with the signer's name, account, and signature meaning (quality-review). The panel expands at the foot of the production record, so the record itself is what is being signed on that surface. The batch detail links to the record and does not render it, so a panel opened there carries the summary hydrated from the BatchRecord API instead (recipe, execution window, outcome, phase/alarm/operator-action counts)
§11.50(b) Signature information subject to same controls as electronic records Implemented Electronic signature stored within immutable AuditRecord (write-once, webhook-protected). Recipe status fields (ApprovedBy, ApprovedAt, RejectedBy, RejectedAt) updated atomically with audit record creation

Every BatchRecord in PendingReview exposes a Review & Sign action in the Data → Batch Records detail view. The panel expands at the foot of the record, so the reviewer scrolls the record itself and attests underneath it. It carries an attestation statement that tracks the Approved/Rejected decision and a "Signing as" line manifesting the reviewer identity and the signature meaning (quality-review). It collects the decision, comments, and a required reason, then issues POST /batchrecords/{name}/review which attaches an ElectronicSignature to the record. Once approved, the record badge flips to Finalized and the validating webhook rejects any further modification.

Review & Sign: the 21 CFR Part 11 electronic signature panel at the foot of the production record, carrying the decision (signature meaning), comments, and required reason. The signer identity is taken from the authenticated session

§11.50(a): the Review & Sign panel manifesting the signature's meaning before the act, the attestation statement rewriting as the decision toggles, above the "Signing as" identity line. The executed ceremony is in the §11.10(b) figure above.

Section 11.70 -- Signature/Record Linking

Requirement Description Status Implementation
§11.70 Electronic signatures linked to respective records so signatures cannot be transferred Implemented HMAC-SHA256 digest computed over canonical JSON of recipe spec (or batch record spec) + signerID + timestamp + meaning + keyID. For batch records, spec includes the materialised phase history, operator actions, parameter values, control-recipe snapshot, and change history (issue #194), so the signature covers the full production record and not merely its metadata. Digest stored in ElectronicSignature.Digest within the AuditRecord and BatchRecordFinalization.Digest, cryptographically binding the signature to the specific record content. Versioned key store (dcs-signing-key Secret in kube-system) supports key-ID chaining: each signature carries a keyID identifying the key version used, and rotation adds a new key without removing old ones, so a rotation never strands the key version a historical signature was made under. Legacy (pre-rotation) signatures carry an empty keyID and are reported for manual review. Two verification paths exist and they prove different things. For archived records the stored batch is re-fetched, its canonical digest recomputed, and the manifest signature re-verified against the key named by archive-key-id (pkg/audit/archive.VerifyManifest, surfaced by dcs audit verify --archived); the same recomputation gates each record's archival attestation. For active records in etcd, dcs audit verify checks that each signature's keyID resolves in the current key store and does not recompute the digest, because the AuditRecord does not retain the payload that was signed (internal/gateway.handleVerifySignatures). Active-record integrity therefore rests on admission-time immutability (§11.10(e)) plus the archive re-verification every record is eventually retired into. Re-signing the live object after the fact plays no part. Signatures are also refused where there is no live record to bind to: an operator prompt belongs to the runtime episode that raised it, so once the phase reaches a terminal state the procedural operator clears status.pendingPrompt and the gateway rejects a late prompt-response with 409 Conflict, refusing to sign an acknowledgement nothing will consume (#992)

Subpart C -- Electronic Signatures

Section 11.100 -- General Requirements

Requirement Description Status Implementation
§11.100(a) Each electronic signature unique to one individual Implemented OIDC authentication provides unique subject claim per user. Gateway approval handlers require authenticated user and record SignerID (OIDC subject) in ElectronicSignature. Each signature is bound to a unique individual via OIDC identity
§11.100(b) Identity verified before establishing electronic signature Implemented OIDC token verification before any approval/rejection. Gateway handlers explicitly check authenticated user (401) and required group membership (403) before signing
§11.100(c) Electronic signatures based on at least two components (e.g., ID + password) Partial Relies on OIDC provider for multi-factor authentication. System enforces Bearer token verification but does not independently enforce two-factor -- delegated to identity provider policy

Section 11.200 -- Electronic Signature Components and Controls

Requirement Description Status Implementation
§11.200(a)(1) Non-biometric signatures: at least two distinct identification components (ID + password) Partial OIDC authentication requires credentials at the identity provider (ID + password or MFA). System validates Bearer token. Two-factor enforcement delegated to IdP configuration
§11.200(a)(2) Only first signing in continuous session requires both components Partial OIDC token has an expiry window; within that window, the session is continuous. Re-authentication required when token expires. Session continuity managed by OIDC token lifetime
§11.200(a)(3) Non-continuous sessions require both components each time Partial Token expiry forces re-authentication. CLI stores token but it expires per OIDC provider policy. Each new session requires full OIDC login flow
§11.200(b) Biometric signatures designed to be usable only by genuine owner N/A No biometric support planned

Standing decision (#688): the signing credential is the live authenticated session. Electronic signatures (batch record review/finalization, recipe approval/rejection via the promote flow) deliberately do not re-collect a password or other credential inside the signing ceremony. The two identification components of §11.200(a)(1) are executed at the identity provider during OIDC login (ID + password, plus MFA where the IdP enforces it). Within the resulting session, §11.200(a)(2) permits subsequent signings on the continuous-session component alone. The gateway bounds "continuous" independently of general token lifetime: all e-signature endpoints enforce a signature-specific token max-age and return reauthentication_required when the session token is older, forcing a fresh full login before the signature can execute (requireFreshToken, surfaced in the UI by the Review & Sign and Promote panels). Re-collecting a static credential in the ceremony was considered and rejected: the gateway never sees IdP passwords by design, and a gateway-local secondary password would be a weaker, separately managed credential, exactly what §11.300 controls exist to avoid. Both signing surfaces (review-sign.js, promote.js) share this property and manifest the credential basis to the signer ("via authenticated OIDC session").

Standing decision (ADR 0055, #1498): a machine identity never signs. Data arriving from one is recorded as a measurement. An API-key identity carries no iat claim, so requireFreshToken refuses it unconditionally on every e-signature endpoint. That refusal is correct and permanent: §11.200 requires a signature to be made by an identified individual, and a machine account is not one. It also meant that until #1498 a laboratory result could only reach a running phase by a person reading it off one screen and typing it into another.

The resolution separates two acts that had shared one mechanism. A decision is a human act and carries a signature. A measurement is data and carries provenance: the delivering system, the credential that presented it, and the sample or analysis id, recorded on an AuditRecord under the ExternalResult action with no ElectronicSignature attached. That absence is the assertion. A reviewer asking who vouched for the number is told nobody, which is the truth.

The delivery endpoint is fenced to integration identities for the same reason (#1515). It sits at the operate tier, which every human role holds. Without the fence, a person could have reached the one route in the product that files a value the recipe branches on with no signature against it. A caller holding a human role is refused with 403 and directed to the signed prompt path.

Two properties make the distinction durable. The delivered value lands in batchRecord.spec.externalResults[] and is never joined into operatorActions[], whose every field (acknowledgedBy, acknowledgedAt, a signature behind auditRecordRef) asserts a person. No reader has to consult a discriminator field to tell a machine delivery from a sign-off. And the value is validated against the bounds the chart author declared, through the same code path an operator's entry uses. An integration therefore cannot steer a transition to a value the recipe excluded.

When the external system cannot deliver, an operator may satisfy the same wait through the normal prompt-response endpoint (fresh token, signature, required reason), and that entry is recorded as an operator action carrying the signature meaning external-result-manual-entry. Every value that took the fallback is therefore retrievable as a population for review.

Section 11.300 -- Controls for Identification Codes/Passwords

Requirement Description Status Implementation
§11.300(a) Uniqueness of identification codes Planned Kubernetes ServiceAccount names are unique per namespace
§11.300(b) Periodically checked, recalled, or revised Planned No credential rotation mechanism
§11.300(c) Loss management procedures for tokens/cards N/A No physical tokens used
§11.300(d) Transaction safeguards to prevent unauthorized use Partial RBAC prevents unauthorized API access; no application-level session management
§11.300(e) Testing of devices for proper function Partial Health probes on all operator pods; health check on runtimes

ALCOA+ Data Integrity

The FDA Data Integrity Guidance (2018) and MHRA GxP Data Integrity Guidance define nine ALCOA+ properties. Each maps to concrete Cloud-Native DCS features. Several of the mechanisms cited below are the same ones filmed in the §11.10 and §11.50 figures above. The rows cite the mechanism, and the footage stays in those sections:

Property How Cloud-Native DCS satisfies it
Attributable Every AuditRecord carries Actor fields populated from the authenticated OIDC identity (sub, name, email, groups). Gateway middleware rejects state-changing requests that lack an attributable identity.
Legible Records are JSON/YAML under CRD schemas with OpenAPI validation. dcs audit trace renders chronological trails in human-readable form; the REST API returns structured JSON.
Contemporaneous AuditRecords are created synchronously with the action that generated them. Timestamps use RFC3339 with millisecond precision. NTP/chrony on cluster and edge nodes is a deployment prerequisite.
Original AuditRecord CRs are immutable: create-only RBAC plus a ValidatingWebhookConfiguration that rejects UPDATE and DELETE. BatchRecord finalization is write-once.
Accurate HMAC-SHA256 digests bind electronic signatures to the exact record content. dcs qualify iq/oq/pq provide executable validation protocols. CRD OpenAPI schema validation rejects malformed input at the API server.
Complete All ISA-88 state transitions on Batch, Procedure, UnitProcedure, Operation, Phase, and Unit emit AuditRecords. The archiver CronJob moves records from etcd to PostgreSQL idempotently without dropping fields. A batch reaches its terminal phase before its last phase has written its own terminal status, so a record frozen at that instant can hold a phase entry with no end time; the reconciler repairs the record until every entry rests, and a PhaseHistorySettled condition on every record says whether it got there (#1686).
Consistent Two-tier retention (etcd active + PostgreSQL archive) with idempotent archival. Unified query across both tiers through the gateway REST API presents one logical stream to downstream consumers. The principle forbids unexplained gaps, and the one bounded gap the architecture produces — a controller failover during a running batch (ADR 0041) — is explained inside the batch record itself: BatchRecord.spec.failoverEvents[] carries each failover with its control gap and data gap bounded separately, every bound referencing its evidencing AuditRecord (#1308).
Enduring Active etcd retention defaults to 90 days; archive retention defaults to 1095 days (3 years) and is customer-configurable with no hard upper bound. Archive format is stable JSON, portable across PostgreSQL versions.
Available GET /api/v1/sites/{site}/audit merges active and archived records and declares whether it reached both, so a trail shortened by a historian outage reads as short (#1648). dcs audit export produces JSON or CSV for regulatory submission. Audit queryability is exercised by the IQ/OQ/PQ executable protocols on every deploy.

Archive-attested deletion model

Problem. A naive "webhook forbids all DELETE on AuditRecord" policy satisfies §11.10(e) but makes Kubernetes garbage collection impossible: etcd grows unboundedly and namespace teardown blocks forever on a stuck-on-Terminating finalizer. The archiver must be able to move records out of etcd once they are durably archived.

Contract. The audit-immutability.dcs.io validating webhook admits DELETE only when one of two mutually-exclusive tracks clears it. The compliance guarantee (no AuditRecord is lost from durable storage) is preserved in both branches.

Track 1 — production: cryptographically-attested deletion

Deletion is admitted only when the record carries a complete archival attestation that verifies under the cluster's HMAC signing key (dcs-signing-key Secret in kube-system). The archiver stamps six annotations onto a record after the enclosing batch has been inserted into PostgreSQL and (when configured) mirrored to an S3 Object-Lock bucket:

Annotation Meaning
audit.dcs.io/archived-to Backend identifier (always historian). Presence marker.
audit.dcs.io/archive-manifest-id UUID of the row in audit_archive_manifest that covers this record.
audit.dcs.io/archive-manifest-digest Hex SHA-256 of the canonical record set in the manifest.
audit.dcs.io/archive-signed-at RFC 3339 timestamp used in the signed payload.
audit.dcs.io/archive-key-id Key-store ID of the HMAC key used (rotation support per §11.70).
audit.dcs.io/archive-signature Hex HMAC-SHA256 over the canonical JSON {namespace, recordUID, recordName, manifestID, digestSHA256}.

The webhook verifies the signature independently: it reconstructs the canonical payload from the record itself plus the annotations and re-computes the HMAC under the declared key-id. Any tampered field (manifest ID, digest, key ID, signer ID, meaning, timestamp) produces a non-matching HMAC, and DELETE is denied.

Invariants.

  • The archiver only stamps attestation annotations after a successful PG InsertAuditBatch and (when enabled) a successful S3 Object-Lock write, so the annotation cannot exist without a durable copy.
  • The archiver verifies an attestation it finds, using the same check the webhook applies (archive.VerifyAttestation). Mere presence of the annotations is never trusted. A record whose attestation does not verify is re-archived and re-stamped. Without this the two halves disagreed: the archiver skipped any annotated record as already archived while the webhook refused its DELETE on the bad signature, so a record carrying fabricated annotations could be archived by nobody and deleted by nobody, permanently (#1193). No record was ever lost, so §11.10(e) held throughout. The record still could not complete its retention lifecycle. The re-stamp itself has to survive admission, which is what the attestation-overwrite rule below permits.
  • The HMAC key is held in a kube-system Secret accessible only to the batch-operator (webhook verification) and the audit-archiver and gateway (signing). No service account with etcd-level delete authority on AuditRecord can obtain the key.
  • The signature binds namespace + recordUID + recordName + manifestID + digestSHA256 together. An attacker who steals annotations from one record cannot transplant them onto another. A different UID changes the signed payload, so the HMAC no longer matches.
  • The UPDATE branch accepts only spec-identical updates whose new annotations are drawn from a fixed allow-list (the six above). An attacker with update permission alone cannot smuggle non-archival metadata onto the record.
  • One overwrite of those six keys is accepted, and only one: an update that leaves every other annotation untouched and whose resulting record carries an attestation that verifies under the cluster's key store. That is the archiver's re-stamp of a record whose attestation was fabricated. Only a holder of the signing key can produce it, which is the same bar DELETE applies. Swapping one unverifiable attestation for another is denied, and so is a re-stamp that also rewrites an annotation the archiver does not own. Without this exception the heal above was denied at admission for exactly the records it exists to rescue, leaving them wedged and the archiver CronJob permanently red.

Key rotation. The dcs-signing-key store supports key-ID chaining (§11.70). Rotating the key does not invalidate historical attestations: verify selects the key identified by the stored archive-key-id.

Track 2 — dev/demo: namespace opt-out

A namespace carrying label dcs.io/retention-policy=none is outside the compliance perimeter entirely. The webhook admits UPDATE and DELETE in that namespace with an explicit Warnings: header:

AuditRecord "<ns>"/"<name>" deletion allowed via dcs.io/retention-policy=none: compliance enforcement is disabled in this namespace

This label is applied exclusively by hack/demo-provision.sh when setting up sandbox demo sites. Production namespaces must never carry this label. Operators who apply it manually are explicitly exempting the target namespace from 21 CFR Part 11 §11.10(e) and must be prepared to justify that choice to an auditor.

The label is observable: kubectl get ns -L dcs.io/retention-policy lists every namespace currently outside compliance, so the opt-out population is never hidden.

Why this preserves §11.10(e)

The §11.10(e) guarantee is about the audit record itself. Its storage location is free to change. Track 1 delivers a stronger guarantee than the original "etcd only, webhook-denied" policy: the record is in PostgreSQL + S3 Object-Lock (WORM) with a cryptographic proof binding its deletion from etcd to its presence in durable storage. Track 2 applies only to namespaces that by policy do not contain compliance-regulated records.

Audit Trail Architecture

The audit system follows a write-once, read-many architecture:

flowchart TD
    CR[Controller Reconcile Loop] --> Rec[Audit Recorder]
    API[Gateway REST API] --> Rec
    CLI[CLI - future] --> Rec
    Rec --> AR[AuditRecord CR<br/>immutable, write-once<br/>RBAC: create-only<br/>webhook rejects UPDATE/DELETE]
    AR -->|after active retention| Arc[audit-archiver CronJob]
    Arc -->|1. insert idempotent| PG[(PostgreSQL / TimescaleDB<br/>audit_records table<br/>retention: 1095 days)]
    Arc -->|2. annotate archived-to: historian| AR
    Arc -->|3. delete from etcd| AR

Recording points: Batch creation, state transitions, recipe approval/rejection, equipment allocation (including re-assertion of a claim a batch had lost, #1314), operator prompt acknowledgements (PromptAcknowledge), measurements delivered by an external system (ExternalResult, #1498), out-of-band change reversion (DriftCorrected, #789/#796), failures. (InvalidateSetupToken records exist on clusters bootstrapped before the setup-token mechanism was retired with ADR 0004. The action value is retained so those historical records remain schema-valid.)

Query paths: Gateway REST API (/api/v1/sites/{site}/audit) transparently merges active (K8s) and archived (historian) records. CLI (dcs audit trace, dcs audit export).

Write-failure handling: the recorder retries a transient apiserver failure with a short bounded backoff before giving up. A failure that survives the retries is always logged and counted on the dcs_audit_record_failures_total metric (labels action, target_kind, component). A dropped record is therefore never silent. Alert on any non-zero rate (see Monitoring Metrics). Beyond that visibility floor, two write paths carry different guarantees, by design:

  • Admission path (strongest). The change-control admission webhook denies the mutation if its AuditRecord cannot be written ("no admit without audit"). The regulated change never happens without its record.
  • Approval/apply transitions (strong). The ChangeRequest reconciler withholds the Approved and Applied state transitions when the AuditRecord (the §11.10(k)/§11.50 compliance artifact for the e-signed action) fails to persist: the request stays in its prior phase and requeues, refusing to advance un-auditably (issue #642). This mirrors "no admit without audit" for the reconcile-driven transitions. The same fail-closed posture covers the diff base of the record (issue #895): the reconciler snapshots the target's review-time state into status.baseSnapshot before signatures can advance the request, and post-apply phases render the semantic diff from that recorded base. "What was approved" therefore stays answerable from a terminal ChangeRequest even after the live target has converged on the approved change.
  • Other reconcile/bridge paths (best-effort). Generic controller status transitions and the audit-flux-bridge reconcile records do not block their control loop on an audit-write failure. The record is dropped after retries, and the drop is visible via the metric above. These records are operational trail. The gating §11.10(k) artifact for an engineering change is produced on the admission or approval/apply path.

Filesystem authoring mode and the review-of-record

Filesystem authoring mode (dcs edit, topology F) lets an engineer author CRD YAML against a local Git working tree with no cluster. This is a dev/offline authoring facility. It does not change where the Part 11 review-of-record happens.

  • The review-of-record stays at Promote. The reviewed-and-signed change (the one that carries the §11.50 signature manifestation, the §11.70 record/signature linkage, and the immutable AuditRecord) is the connected Promote / ChangeRequest flow, unchanged, in every authoring topology. A change authored as files reaches that flow when it is pushed to a cluster and Promoted.
  • File-mode review is developer review and not the regulated review. When an engineer reviews a file-mode change in their own Git tooling, that is ordinary source review. The optional dcs diff command renders the semantic diff there for convenience, but it creates no AuditRecord and no electronic signature, and is explicitly not the review-of-record.
  • No control is weakened. Because approval, signing, and the audit trail remain server-side in the gateway, moving authoring off-cluster does not affect §11.10(k), §11.50, or §11.70. The compliance carrier is exactly where it was.

See Change Control for the contract this preserves.

Gap Analysis

Gap §11 Reference Priority Description
~~User authentication~~ ~~§11.100, §11.200, §11.300~~ ~~High~~ ~~Resolved: OIDC identity provider integration implemented; user actions attributed to individual authenticated users~~
Electronic signatures §11.50, §11.100, §11.200 Resolved HMAC-SHA256 electronic signatures with signer identity, timestamp, meaning, and tamper-evident digest. Gateway API + CLI for recipe approval/rejection. Remaining gap: MFA enforcement delegated to IdP
~~Session management~~ ~~§11.200(a)(2-3)~~ ~~Medium~~ ~~Resolved: OIDC token-based sessions with configurable lifetime; session continuity enforced via token expiry~~
~~Change control~~ ~~§11.10(k)~~ ~~Resolved~~ ~~Two-layer control: (1) control-operator defers ControlProgram hot-swap (spec replace) while any Batch on the target unit is Running/Holding/Held/Aborting — the SAFETY gate. (2) Mutating admission webhook on ControlProgram/ControlModule requires, in namespaces labelled dcs.io/production=true, an allowlisted ServiceAccount + configured external-reference annotation + HMAC electronic signature with meaning="approved". Every admission writes an AuditRecord with {sourceServiceAccount, externalRef, resourceSHA, signature}. Optional break-glass path requires a meaning="break-glass" signature plus a reason annotation. See change-control.md.~~
~~Record retention~~ §11.10(c), §11.10(e) ~~Medium~~ Resolved: audit-archiver CronJob moves records from etcd to PostgreSQL; gateway queries both active and archived; CLI export for regulatory submissions
~~Webhook deployment~~ ~~§11.10(e)~~ ~~Low~~ ~~Resolved: ValidatingWebhookConfiguration for AuditRecord, Batch, and BatchRecord immutability is shipped by the Helm chart with webhook.enabled: true as the default and cert-manager CA injection wired in templates/webhook-configuration.yaml~~
~~HMAC signing key rotation~~ ~~§11.70~~ ~~Medium~~ ~~Resolved: Versioned key store with key-ID chaining. Each signature carries a keyID; rotation adds a new key version without removing old ones. dcs audit verify confirms post-rotation that every historical signature's key version still resolves in the store; see the §11.70 row for what that check does and does not recompute. See rotation runbook~~
~~Formal validation~~ ~~§11.10(a)~~ ~~Low~~ ~~Resolved: dcs qualify iq, dcs qualify oq, and dcs qualify pq implemented as the full executable validation suite. IQ reads live cluster state against an approved baseline. OQ drives canned batches through all 8 external ISA-88 commands (Start, Hold, Restart, Abort, Pause, Resume, Stop, Reset) and links each transition to a StateTransition AuditRecord. PQ runs N batches, queries the historian for a CQA per batch, and computes Mean/StdDev/Cpk with a CAPABLE/NOT_CAPABLE verdict against a configurable Cpk minimum (default 1.33 per FDA guidance). All three emit signed reports persisted as AuditRecords (Target.Kind=Qualification) and support --fail-on-deviation for CI/CD gating. See validation.md.~~

Roadmap

  1. ~~Identity provider integration~~ -- Implemented: OIDC authentication for the Gateway, mapping user identity to AuditActor fields
  2. ~~Electronic signature service~~ -- Implemented: HMAC-SHA256 signing of recipe approval/rejection via gateway API and CLI
  3. ~~Webhook activation~~ -- Implemented: cert-manager-injected ValidatingWebhookConfiguration enforces AuditRecord, Batch, and BatchRecord immutability (chart default webhook.enabled: true)
  4. ~~Change control gates~~ -- Implemented: two-layer control. (1) Control-operator defers ControlProgram hot-swaps while a Batch is active on the target unit (safety gate). (2) Mutating admission webhook requires allowlisted ServiceAccount + external-reference annotation + HMAC meaning="approved" signature for every ControlProgram / ControlModule write in namespaces labelled dcs.io/production=true (change-control gate). Every admission writes an AuditRecord with source SA, external ref, spec SHA, and signature. Break-glass path requires a separate meaning="break-glass" signature. See change-control.md
  5. ~~Record archival~~ -- Implemented: audit-archiver CronJob archives AuditRecord CRs from etcd to historian PostgreSQL/TimescaleDB. Active retention configurable (default 90 days), archive retention default 3 years (21 CFR Part 211). Gateway transparently queries both active and archived records. CLI dcs audit export provides JSON/CSV export for regulatory review
  6. ~~ChangeRequest propose-approve UI~~ -- Implemented: gateway endpoints + Promote button submit ChangeRequest CRs with HMAC-bound proposed and approval e-signatures. The reconciler from #289 verifies the signatures before transitioning the request to Approved and dispatching to the configured backend. See Promote a Recipe for the user flow and ChangeRequest backends for the operator-side configuration.