Change Control for Production ControlPrograms¶

This guide explains how Cloud-Native DCS enforces 21 CFR Part 11 §11.10(k) change control over ControlProgram and ControlModule edits in production namespaces, and how to wire your own change-control system (GitOps, a ticket system, or a time-bounded human apply window) into the Kubernetes contract.

The compliance contract is opinionated. The implementation is not. Pick any carrier you like. The contract that satisfies §11.10(k) stays the same.

Why this exists¶

The ISA-88 state machine, hot-swap safety gate, and electronic-signature plumbing together cover most of 21 CFR Part 11. The remaining §11.10(k) gap was segregation of duties on control-logic edits: in a production namespace, a single engineer could still author, apply, and deploy a ControlProgram change with no second-party review.

Pharma QA asks about this during vendor qualification. The answer you want is:

Production mutations of ControlProgram and ControlModule are rejected by an admission webhook unless they arrive through an allowlisted service account, carry a reference to our change-management system, and are cryptographically signed by an authorised approver. Every admission writes an immutable AuditRecord.

That contract is the subject of this document.

How it works¶

The compliance contract (opinionated)¶

A validating/mutating admission webhook runs in batch-operator and guards control.dcs.io/controlprograms and physical.dcs.io/controlmodules CREATE/UPDATE/DELETE requests. It also guards changecontrol.dcs.io/changecontrolpolicies CREATE/UPDATE/DELETE requests, exempting only the CREATE that seeds the configured policy (see Gating the policy itself below).

In any namespace labelled dcs.io/production=true, the webhook requires:

Allowlisted ServiceAccount — req.UserInfo.Username must match a spec.allowlist[].serviceAccount entry in the cluster-scoped ChangeControlPolicy.
External reference annotation — the object must carry the annotation key named by the SA's policy entry (a git commit SHA, a ServiceNow ticket ID, etc.). The value is opaque to the DCS and meaningful to your change-control system.
Electronic signature(s) — at least one dcs.io/esig-<role> annotation whose JSON payload parses as ElectronicSignature with meaning="approved" and verifies against the gateway HMAC key store. The signature is cryptographically bound to the external-reference value.

On admit the webhook stamps three annotations onto the object (dcs.io/cc-source-sa, dcs.io/cc-admit-path, dcs.io/cc-external-ref) and synchronously writes an AuditRecord with {sourceServiceAccount, externalRef, resourceSHA, signature}. If the AuditRecord write fails the admission is denied: no admit without audit.

In non-production namespaces (no dcs.io/production=true label) the webhook allows the request unchanged.

Gating the policy itself¶

ChangeControlPolicy has no UI, CLI, or REST surface, and that is a design decision. It is declarative configuration, managed via git like the rest of the plant model, seeded from the Helm chart's changeControl.policy values or applied as a manifest (see examples/changecontrolpolicy.yaml).

The ChangeControlPolicy configures the very controls the webhook enforces: the allowlist, requireDistinctApprover, and the break-glass path. If the policy itself were not gated, an actor with update/delete RBAC on changecontrolpolicies could empty the allowlist, flip requireDistinctApprover to false, or enable break-glass, turning off every control silently and with no signed or approved change. That would be a self-referential integrity gap.

So the webhook gates the policy too. Because ChangeControlPolicy is cluster-scoped it carries no namespace and therefore no dcs.io/production label, so its mutations are gated unconditionally (not via the production-namespace check) whenever changeControl.enforceOnProduction: true. An UPDATE or DELETE of the policy must come from a ServiceAccount that the current policy allowlists, carry that SA's external-reference annotation, and carry a verified meaning="approved" signature, exactly like a ControlProgram edit. The resulting AuditRecord is written into the operator's own namespace (the policy has none of its own).

CREATE is gated as well, with exactly one exemption. Seeding the policy whose name the operator is configured to read is the bootstrap. Gating that would require a policy to approve creating one, so it is admitted without a check. The apiserver evaluates that exemption itself, through a matchConditions expression on the rule. The webhook is never consulted for the request. That placement matters on a fresh install. There the chart creates the webhook configuration before the policy it seeds, while the operator pod behind the webhook is still starting.

A policy under any other name takes the same approval an edit takes. It is read by nothing while the configured name is something else, which looks at first like a reason to leave its CREATE alone. That inertness describes the current value of changeControl.policy.name. It is not a property of the object. The moment that value names the policy, it governs the cluster. An ungated CREATE would let an unsigned policy be planted and wait for a rename, which is the same self-referential gap the UPDATE and DELETE gates close one operation over.

The gate admits an approved CREATE, and that is what keeps a rename reachable. Moving the cluster onto a new policy name means creating the new policy first, and that is a change-controlled act like any other. It needs an allowlisted carrier, an external reference, and a verified approved signature. A flat refusal would have forced the operator flag to move first, which leaves the webhook fail-closed on every production mutation until the new policy lands.

The flag itself is not change-controlled. It is a Helm value, and cluster RBAC on the operator Deployment is what protects it. With CREATE gated there is nothing unapproved for it to be pointed at. A flag naming a policy that does not exist fails closed, because the webhook denies every gated mutation whose policy it cannot load.

Two operational consequences follow. A rename applied by a person running helm upgrade is refused, because a human is not a ServiceAccount and cannot be on the allowlist. That was already true of any policy edit. The rename now behaves like the edit it resembles, and a GitOps install applies both through its allowlisted reconciler as usual.

The second consequence is the boundary of the gate. None of this applies while changeControl.enforceOnProduction is false, because the webhook is not installed at all. A policy authored before enforcement was switched on was therefore never reviewed by this contract. Audit the changecontrolpolicies already in the cluster when you turn enforcement on.

The implementation (not opinionated)¶

Anything can drive an allowlisted ServiceAccount. We ship a GitOps reference implementation because it is what we run. There is also a worked integrator example for customers whose qualified SOP is ServiceNow, TrackWise, or another ticket-driven system.

Where the YAML was authored does not matter¶

The contract above is enforced at admission into a production namespace, on the content of the request. How the YAML was produced plays no part in it. Authoring a ControlProgram by hand, through the connected UI, or in filesystem authoring mode (dcs edit, topology F) makes no difference: the object still cannot enter a dcs.io/production=true namespace without the allowlisted ServiceAccount, the external reference, the verified electronic signature, and the synchronous AuditRecord.

dcs diff lets an engineer preview a file-mode change semantically in their own Git review, but it signs nothing and admits nothing. The electronic signature and the review-of-record are applied here, at the production boundary.

Enabling the feature¶

The webhook, RBAC, and CRD are installed but disabled by default. Enable it with one Helm value:

# values.yaml
changeControl:
  enforceOnProduction: true

  webhookFailurePolicy: Fail  # deny when the webhook is down (recommended)

  policy:
    name: default
    allowlist:
      - serviceAccount:
          namespace: flux-system
          name: kustomize-controller
        externalRefAnnotation: dcs.io/git-commit-sha
        requiredSignatureMeanings:
          - approved

    # Segregation of duties on the ChangeRequest propose/approve workflow:
    # the proposer of a ChangeRequest cannot also approve it. On by default —
    # and also enforced when no ChangeControlPolicy exists at all — so this
    # is only ever an explicit opt-out.
    requireDistinctApprover: true

    breakGlass:
      enabled: false

Then label each production namespace:

kubectl label namespace site-newark-plant dcs.io/production=true

From that moment, every ControlProgram / ControlModule write in site-newark-plant must arrive through the flux-system/kustomize-controller ServiceAccount carrying a dcs.io/git-commit-sha annotation and a verified approval signature. Any other writer (human user, other SA, unlabeled tooling) is denied.

Bootstrap ordering

Install the chart with changeControl.enforceOnProduction: true and the populated allowlist before adding the dcs.io/production=true label to any namespace. A failure-policy of Fail combined with a missing ChangeControlPolicy will brick the namespace. See Bootstrap runbook below.

GitOps reference implementation¶

What we run in our own reference environment.

Topology¶

flowchart TD
    Repo["GitHub config repo<br/>protected branch"]
    CO[CODEOWNERS: author != reviewer != approver]
    BP[Branch protection: signed commits + approving review]
    Flux["Flux kustomize-controller<br/>cluster: dcs-prod<br/>runs as allowlisted SA flux-system/kustomize-controller"]
    Ann["annotates each applied object with<br/>dcs.io/git-commit-sha = merged commit SHA<br/>dcs.io/esig-approver = HMAC signed by approver"]
    Adm[Admission webhook admits]
    Audit[AuditRecord written]
    Recon[Reconcilers proceed]

    Repo --> CO
    Repo --> BP
    Repo --> Flux
    Flux --> Ann
    Ann --> Adm
    Adm --> Audit
    Audit --> Recon

Set up¶

Create the config repo with a Flux Kustomization targeting the production cluster's namespaces.
Protect the main branch with:
- "Require signed commits" on
- "Require pull request reviews before merging" (reviewer ≠ author)
- CODEOWNERS mapping api/control/ and config/controlmodules/ to the dcs-supervisor team (approver ≠ reviewer)
Wire the signer — a small pre-commit or CI step that, once the PR has the required supervisor approval, POSTs the approver's identity to the gateway's /api/v1/signatures endpoint (see the request/response shape in the API reference), receives back an ElectronicSignature JSON, and annotates the merged objects with dcs.io/esig-approved: {...}. The signer binds the signature to the PR's merged commit SHA (the externalRef in the mint request) so the webhook accepts it.
Allowlist the Flux SA in the ChangeControlPolicy (see YAML above).
Label the production namespace last.

Signature lifecycle¶

The gateway is the only signer. It holds the HMAC key store that the webhook verifies against. The CI signer script calls POST /api/v1/signatures with supervisor-group OIDC credentials (§11.10(g)). The gateway checks the caller's group claim, produces an ElectronicSignature bound to the requested external-ref, audits the mint (refusals included), and returns it for embedding as an annotation.

This means the cryptographic authority never leaves the gateway pod, even though the artefact (the signed annotation) travels through git.

Integrator guide (non-GitOps change control)¶

If your qualified SOP is ServiceNow, TrackWise, or another ticket-driven system, do not replace it with git. Bridge it instead.

Pattern:

flowchart TD
    Req[ServiceNow Change Request approved]
    Hook["ServiceNow webhook -> in-cluster controller<br/>runs as allowlisted SA dcs-changecontrol/servicenow-bridge"]
    Fetch[fetches approved ControlProgram/ControlModule spec<br/>from ticket payload or blob store]
    Sign["calls gateway /api/v1/signatures<br/>returns ElectronicSignature bound to ticket ID"]
    Apply["applies object with annotations<br/>dcs.io/change-ticket = CHG0001234<br/>dcs.io/esig-approved = signed payload"]
    Adm[admission webhook admits]
    Audit[AuditRecord carries ExternalRef=CHG0001234]

    Req --> Hook
    Hook --> Fetch
    Fetch --> Sign
    Sign --> Apply
    Apply --> Adm
    Adm --> Audit

Alternative: bridge through ChangeRequests

A ticket bridge that prefers not to handle signatures itself can instead create a ChangeRequest via the gateway API (POST /api/v1/changerequests) and drive the approval there. The ChangeRequest ceremony mints the required signatures internally.

Allowlist configuration:

changeControl:
  policy:
    allowlist:
      - serviceAccount:
          namespace: dcs-changecontrol
          name: servicenow-bridge
        externalRefAnnotation: dcs.io/change-ticket
        requiredSignatureMeanings:
          - approved

Everything else (the webhook, audit schema, CLI, UI panel) works identically. Auditors inspecting an AuditRecord see externalRef: CHG0001234 and follow the link back into ServiceNow. The dcs change-control verify CLI treats git-backed and ticket-backed chains the same way.

Break-glass¶

Sometimes a real incident requires bypassing change control. Enable the break-glass path only for sites where the on-call rota has been briefed:

changeControl:
  policy:
    breakGlass:
      enabled: true
      # Default annotation key: dcs.io/change-control-break-glass-reason

A break-glass mutation:

Carries a non-empty dcs.io/change-control-break-glass-reason: "prod-outage-2026-04-16" annotation explaining why normal change control was bypassed.
Carries an ElectronicSignature annotation with meaning="break-glass", signed by the on-call engineer (ties the bypass to a specific human per §11.200).
Is admitted without an allowlist check — any authenticated SA may apply it once the reason + signature are present.

Every break-glass admission writes an AuditRecord with admitPath=break-glass and the reason in externalRef. Add a Prometheus alert on that annotation so a break-glass is impossible to hide.

Observability¶

Three surfaces expose the change-control chain:

Surface	Use case
Gateway UI → `/system` → ControlModule detail → Change History	Operator/engineer inspects a specific ControlModule's admission chain
`dcs change-control history <Kind> <Name>`	CLI equivalent, scriptable
`dcs change-control verify`	Periodic cluster-wide integrity check; non-zero exit if any admitted record is missing §11.10(k) fields

The verify subcommand is intended for a scheduled job or CI pipeline. Run it nightly and page if the exit code is non-zero.

Bootstrap runbook¶

The first install, in order:

Deploy the chart with changeControl.enforceOnProduction: false.
Confirm the dcs-signing-key Secret exists (created automatically by the gateway on first start).
Populate changeControl.policy.allowlist in your values.yaml with the ServiceAccount(s) that will carry production mutations.
Upgrade the chart with changeControl.enforceOnProduction: true. At this point the MutatingWebhookConfiguration exists but no namespace is labelled, so nothing is gated yet.
Verify the ChangeControlPolicy resource was seeded: kubectl get changecontrolpolicies.
Label each production namespace: kubectl label namespace site-<name> dcs.io/production=true.
Run dcs change-control verify to confirm no stale non-webhook-sourced records exist and the chain is clean.

Rollback¶

To disable the feature in a live cluster (e.g., during a qualified incident response):

kubectl label namespace site-newark-plant dcs.io/production-
# or, more broadly:
helm upgrade dcs charts/cloud-native-dcs -f values.yaml \
  --set changeControl.enforceOnProduction=false

The ChangeControlPolicy CR remains in place. Only the webhook binding and namespace label are removed. Re-enabling is reversible.

Compliance mapping¶

§11.10 clause	How this feature satisfies it
(a) Validation	Admission webhook + sync AuditRecord write is covered by `dcs qualify oq` when the test harness exercises a prod-labeled namespace
(e) Secure retention of records	AuditRecords are immutable (existing AuditRecord webhook), archived to PostgreSQL (audit-archiver), and never rewritten
(k) Appropriate controls over systems documentation	Webhook + SA allowlist + external-ref + e-signature — the subject of this doc. The `ChangeControlPolicy` that configures these controls is itself gated (CREATE/UPDATE/DELETE, exempting only the CREATE that seeds the configured policy), so the controls cannot be disabled in-band without an approved, signed change, and no unapproved policy can be planted for a later rename to activate; only an operator-level `helm upgrade` (`enforceOnProduction=false`) removes the gate
(g) Authority checks	Signatures are produced only by the gateway, which enforces group-claim checks (§11.100) before signing

See 21cfr11.md for the full traceability matrix.

Segregation of duties is enforced at two independent layers. In the GitOps reference implementation above, GitHub enforces author ≠ reviewer ≠ approver via CODEOWNERS and branch protection. Independently of any backend, the in-product ChangeRequest workflow refuses self-approval: the gateway returns 403 when the proposer calls approve, and the reconciler will not count an approved signature from the proposer toward the required meanings. Both checks honour ChangeControlPolicy.spec.requireDistinctApprover (default true, enforced even when no policy object exists).

For the user-facing dev → prod recipe-promotion workflow built on the ChangeRequest CRD, see Promote a Recipe.