GitOps Enforcement RFC¶

Status: Implemented. Helm values + policy template (#285, #288), ChangeRequest CRD and reconciler (#289) shipped Issue: #285 Umbrella: #283 Scope: changecontrol.dcs.io/v1alpha1 ChangeRequest CRD and the Helm values that configure GitOps enforcement.

Motivation¶

The narrative motivation lives in two existing explanation docs:

GitOps for Automation Engineers: why the UI must be the front door for changes, and how it bridges to a GitOps substrate underneath.
UI Deployment Topologies: the spectrum of supported topologies and the product-vs-deployment split.

This RFC commits to the configuration surface and CRD schema that make those documents implementable. Two surfaces are intertwined and are designed together here:

The Helm values shape — the contract between the product and the deployer.
The ChangeRequest CRD schema — the contract between the UI and the cluster for propose-approve-merge round-trips.

Locking both before any controller code lands prevents churn on the load-bearing API, and gives the laptop-mode UI (#286) and semantic-diff framework (#287) something concrete to target.

Relationship to existing types¶

Two related types already exist in changecontrol.dcs.io/v1alpha1:

ChangeControlPolicy — configures the admission webhook that gates ControlProgram and ControlModule mutations in production namespaces (21 CFR Part 11 §11.10(k)). Its Allowlist names the service accounts that may write controlled resources, with required external-reference and e-signature annotations.

This RFC does not change ChangeControlPolicy. The ChangeRequest CRD introduced here uses the existing policy: when a ChangeRequest is approved and a backend writes to the cluster, the writer's service account must satisfy the active ChangeControlPolicy. The two types are complementary. ChangeControlPolicy is the who-may-write admission gate, and ChangeRequest is the propose-approve workflow the UI drives to satisfy that gate without forcing an engineer into a Git client.

Helm values shape¶

All GitOps enforcement is gated by a single top-level Helm values key. Default is off to preserve current install behavior:

gitopsEnforcement:
  # Master switch. When false, no admission policy is installed and the
  # ChangeRequest controller (when shipped) acts in direct-apply-only
  # mode. Default: false.
  enabled: false

  # Identity of the Flux service account permitted to write engineering
  # CRDs in enforced namespaces. Required when enabled=true unless the
  # deployment uses a different reconciler — see backends.
  flux:
    serviceAccount:
      namespace: flux-system
      name: kustomize-controller

  # Break-glass override path for incident response. Holders of this
  # role can write engineering CRDs directly; every such write is
  # audited at elevated severity.
  breakGlassRole: "dcs:admin:break-glass"

  # Per-class enforcement mode. The product ships ownership-class
  # annotations on every CRD (see #288). The deployment chooses what
  # the admission policy does for each class in enforced namespaces.
  #
  #   read-only      — reject all writes from non-Flux SAs (the in-cluster
  #                    operator SAs in the chart's install namespace are
  #                    allowlisted; see #372).
  #   propose-only   — reject direct writes; require a ChangeRequest.
  #   direct-write   — allow direct writes (no enforcement).
  #
  # Operational classes are always direct-write; this stanza only
  # affects engineering and hybrid classes.
  classes:
    engineering:
      mode: propose-only
    hybrid:
      mode: direct-write   # only annotations/named fields are mutable

  # Default backend for ChangeRequests that do not name one explicitly.
  # Per-Site overrides are configured on the Site CR.
  changeRequest:
    defaultBackend: in-cluster-queue   # direct-apply | git-pr | webhook | in-cluster-queue

    # Configuration for the git-pr backend. Required only when any
    # ChangeRequest selects backend=git-pr.
    gitPR:
      # Default repository URL. Sites may override via Site.spec.deployRepo.
      repository: ""
      # Base branch to PR against (e.g., main).
      branch: main
      # Kubernetes Secret holding a GitHub/Gitea API token. Must have
      # repo:write scope.
      tokenSecretRef:
        name: ""
        namespace: ""
      # Provider hint, used to build the API URL. Supported: github, gitea.
      provider: github

  # Approval routing. The product ships single-approver and two-person
  # primitives; richer routing is layered by the deployment via custom
  # ChangeControlPolicy entries.
  approval:
    # E-signature meanings required on every ChangeRequest before it
    # transitions Proposed → Approved. At minimum, "approved" is
    # always required. Add "reviewed" for two-person rule.
    requiredSignatureMeanings: [approved]

  # Audit linkage (#291, IMPLEMENTED). When enabled, the audit-flux-bridge
  # Deployment receives reconcile events from Flux's notification-controller
  # and writes an AuditRecord carrying spec.gitCommitSHA,
  # spec.gitPullRequestURL, and spec.changeRequestRef. See
  # docs/platform-administration/audit-linkage.md for the operator-facing
  # walk-through. The receiver is a separate component rather than folded
  # into the historian: the audit-emit pathway has no Postgres or MQTT
  # dependency and the bridge can be scaled or restarted independently.
  notification:
    flux:
      enabled: false
      # Empty = compute from chart fullname + Release.Namespace + listenPort.
      webhookEndpoint: ""
      listenPort: 61153
      secretRef:
        name: ""
      auditNamespace: dcs-system

What the deployment owns vs. what the product owns¶

Per the UI Deployment Topologies contract, the values above are deployment decisions. The product decisions baked into the chart are:

The annotation / label the admission policy reads to find a CRD's ownership class.
The set of classes (engineering, operational, hybrid).
The set of backends (direct-apply, git-pr, webhook, in-cluster-queue) and their interfaces.
The set of e-signature meanings recognised (approved, reviewed, verified, responsible).

If a deployment needs anything beyond the values surface, the product should grow a new value or a new policy CRD. A fork is never the answer.

ChangeRequest CRD¶

ChangeRequest is a namespaced CR in changecontrol.dcs.io/v1alpha1. It is the in-cluster handle the UI opens for any propose-approve round-trip. The full type sketch lives in api/changecontrol/v1alpha1/changerequest_types.go. The conceptual shape is:

ChangeRequest
  metadata: standard
  spec:
    target:                 # what is being changed
      apiVersion: recipe.dcs.io/v1alpha1
      kind: MasterRecipe
      namespace: site-newark-plant
      name: ipa-master-v3
    operation: Update       # Create | Update | Delete
    desiredObject: <YAML>   # full SSA-style spec at proposed state
    reason: "Increase Mash Hold time per CCR-0421"
    backend: ""             # "" → use deployment default
    signatures:             # collected as the request progresses
      - meaning: approved
        signerRef: ...
        signedAt: ...
        annotation: dcs.io/esig-approved
    requiredSignatureMeanings: [approved]   # may extend deployment default
  status:
    phase: Proposed         # see state machine below
    conditions: [...]
    resolution:
      backend: in-cluster-queue
      gitCommitSHA: ""
      gitPullRequestURL: ""
      auditRecordRef: ...
    observedGeneration: 1

Field rationale¶

spec.target uses an explicit GVK + namespace + name reference. A Kubernetes TypedLocalObjectReference cannot serve here, because the target may live in a different namespace from the ChangeRequest itself (e.g., a recipe in dcs-prod proposed from a ChangeRequest authored in dcs-dev).
spec.desiredObject carries the full proposed spec, with no JSON Patch form. The semantic-diff framework (#287) computes the human-readable diff at preview time by snapshotting the current object and rendering against desiredObject. Apply semantics are SSA, and the field manager is dcs.io/change-request:<name>.
spec.signatures records collected e-sigs inline for queryability and audit. The signature payload itself is HMAC-verified against the gateway key store at admission.
spec.backend allows per-CR override of the deployment default. A single deployment can then mix flows (e.g., recipes via git-pr for the audit trail, and phase parameter overrides via direct-apply for iteration speed).
status.resolution is the audit anchor: it links the cluster state back to the Git commit and the AuditRecord that recorded the approval.

State machine¶

stateDiagram-v2
    [*] --> Proposed
    Proposed --> Approved: required signatures collected
    Proposed --> Rejected: reviewer rejects
    Proposed --> Withdrawn: author withdraws
    Approved --> Applied: backend executes (direct apply / PR merge)
    Approved --> Failed: backend error
    Applied --> Reconciled: target controller observes change
    Applied --> Failed: reconcile error
    Reconciled --> [*]
    Rejected --> [*]
    Withdrawn --> [*]
    Failed --> Proposed: retry by author
    Failed --> Withdrawn: author gives up

Legal terminal states: Reconciled, Rejected, Withdrawn. Failed is recoverable: an author can fix and retry (returning to Proposed) or withdraw.

Backends¶

The reconciler (#289) exposes a small Go interface, and four concrete implementations ship:

Backend	When `Approved` → `Applied` means	Use case
`direct-apply`	Reconciler SSA-applies `desiredObject` to the API server	Single-cluster deployments without GitOps; iteration in dev namespaces
`git-pr`	Reconciler renders `desiredObject` to YAML and opens a PR against the configured repo. `Applied` fires when the PR merges	Strict GitOps shops; the canonical Promote flow
`webhook`	Reconciler opens a record in an external quality system and holds in `Approved` until it is decided; the configured delegate backend then delivers, so `Applied` means what it means for that delegate (#1424, ADR 0049)	GMP plants whose eQMS (TrackWise, Veeva Vault QMS, MasterControl, ETQ) is the system of record for change control
`in-cluster-queue` (experimental)	Holds in `Approved` until a deployer-supplied controller (custom reconciler, ServiceNow webhook, etc.) flips the phase. No reference integration ships with the product (ADR 0013, #649)	Integrations with existing change-management systems, where the deployer builds the external consumer

For the git-pr backend, the merge commit SHA is recorded in status.resolution.gitCommitSHA once the polling loop observes the PR merge. The audit-flux-bridge (#291, implemented) consumes the same SHA on the Flux side: when Flux's notification-controller posts a ReconciliationSucceeded event, the bridge correlates the event's metadata.revision to a ChangeRequest whose status.resolution.gitCommitSHA matches and emits an AuditRecord with spec.action="Reconcile", spec.gitCommitSHA, spec.gitPullRequestURL, and spec.changeRequestRef populated. This closes the §11.10(k) traceability loop for the git-pr flow without requiring the reconciler itself to learn anything about Flux's wire protocol (see Audit Linkage).

Backends never bypass ChangeControlPolicy. The direct-apply backend runs as a service account that itself appears in the active policy's Allowlist. The git-pr backend produces a commit that Flux later applies under the Flux SA, which the policy already allowlists. The webhook backend writes nothing itself. It gates, and its delegate is one of the two above. The policy applies to the delegate unchanged. The in-cluster-queue backend defers to a deployer-supplied flow whose final writer must satisfy the policy.

Configuration surface checklist¶

Before a deployment can turn on gitopsEnforcement.enabled=true:

[ ] flux.serviceAccount matches the cluster's actual Flux SA.
[ ] breakGlassRole is bound to a real ClusterRole (typically cluster-admin plus an audit-elevation annotation).
[ ] classes.engineering.mode is chosen.
[ ] changeRequest.defaultBackend is chosen, with required sub-config (e.g., gitPR.repository for git-pr).
[ ] At least one requiredSignatureMeanings entry is set.
[ ] An active ChangeControlPolicy exists matching the flux.serviceAccount.

These values are real and wired as of #288. The chart's templates/gitops-enforcement-policy.yaml translates each into a concrete ValidatingAdmissionPolicy, ValidatingAdmissionPolicyBinding, and break-glass ClusterRole. See Turning On GitOps Enforcement for the operator-facing walk-through.

A new dcs admin gitops-preflight CLI command (filed as a follow-up to #288) will check this list before the deployer commits.

CRD ownership taxonomy¶

Every shipped CRD carries a dcs.io/ownership-class label (engineering, operational, or hybrid) on the CustomResourceDefinition object itself. The label is the contract the admission policy reads to decide whether to gate a given write. The policy matches by GVR + resource-name whitelist (since matchConstraints cannot select on the target CRD's labels), and the whitelist is generated from the same taxonomy that drives the labels.

Class	Meaning	Example
`engineering`	Plant structure, recipe definitions, alarm setpoints, control programs	`MasterRecipe`, `Site`, `ControlProgram`
`operational`	Running-state and audit objects	`Batch`, `Phase`, `AuditRecord`
`hybrid`	Has both Flux-owned spec fields and UI-mutable annotations / fields	`Batch` (parameterOverrides + `dcs.io/command` annotation)

Batch is currently the only hybrid resource, and field-level enforcement is a follow-up (the MVP gates the whole spec). New CRDs default to operational if not classified, since that is the safer default (see the assignment map in test/integration/ownership_class_test.go).

Semantic-diff renderer framework¶

The Promote / ChangeRequest preview UI consumes spec.desiredObject through a kind-keyed plugin registry that produces structured, domain-language diff entries (e.g. Mash Hold: hold time 30 min -> 45 min) in place of a raw YAML text diff. This is the semantic-diff renderer framework tracked in #287 and described in detail at Semantic-diff renderer framework.

Three load-bearing points for this RFC:

The diff renders against spec.desiredObject, and a JSON Patch gives it nothing to render. That is the rationale for keeping the SSA-style payload in the schema over a JSON Patch form (see Open question 1 below).
The framework is a registry, one renderer per CRD kind. The long-tail of engineering CRDs (#292) can be picked up by a contributor without touching the ChangeRequest controller.
Path entries are keyed by natural identifier (name) inside arrays (spec.procedureSteps[mash-hold].timeoutSeconds, with no bare [2]-style index). A reorder therefore collapses to a single Step order entry with no sea of +/- lines.

The first concrete renderer covers MasterRecipe (and the cluster-scoped recipe-template kinds when their UI flows come back on) and ships with this RFC update. It is the validation that the framework shape holds for the recipe family that the Promote UI work in #290 will exercise first.

Open questions — resolutions from #289¶

The four questions deferred from the schema RFC have been resolved by the reconciler implementation:

JSON Patch vs full SSA payload for spec.desiredObject. Resolved: SSA payload. The direct-apply backend uses server-side apply with field manager dcs.io/change-request:<namespace>/<name>. The diff renderer (#287) consumes the same SSA payload and produces audit-friendly structured diffs.
Conflict handling at Apply time when desiredObject was computed against a now-stale source. Resolved: SSA's field-manager conflict semantics, surfaced as BackendOutcomeFailed. A conflict moves the CR to Failed with a message asking the author to rebase against the latest target and re-sign. This is a permanent failure with no automatic retry, because the new desired object must reflect the current state.
Multi-target ChangeRequests. Resolved: deferred. The reconciler is single-target as written. Multi-target wraps a future ChangeRequestSet and is filed as a follow-up. Recipe propose-approve continues to work with one CR per target CR, and the UI groups them under a common batch ID. The Promote UI MVP (#290) follows the same single-target shape (one Promote click produces one ChangeRequest), so this question stays deferred until a real multi-resource workflow lands.
Cross-cluster target. Resolved: deferred to the laptop-mode UI work in #286. The reconciler operates against the cluster where the CR lives. Cross-cluster targets are not supported in the MVP.

Additional decisions made in #289¶

Operator binary: the reconciler runs in the existing batch-operator Deployment, since the change-control mutating webhook (#288) is already hosted there. Co-locating both halves of the §11.10(k) surface in one Deployment minimises moving parts.
GitHub SDK: the git-pr backend uses a small in-tree HTTP shim (internal/controller/changecontrol/backend/gitpr), with github.com/google/go-github deliberately left out. The 5-endpoint surface (branch lookup, ref create, file PUT, PR list, PR get) is small enough that the dependency tax of a full SDK isn't worth paying. A Provider interface is exposed so a Gitea (or other host) adapter can be added without touching the backend logic.
Withdrawal signal: dcs.io/withdraw=true annotation on the ChangeRequest. This is least surprising and matches the dcs.io/reject=true rejection annotation already used in webhook-land.
Failed retry: editing the spec returns a Failed CR to Proposed automatically (observedGeneration < generation). An explicit dcs.io/retry=true annotation forces the same transition without a spec change, useful when the original failure was an external-service outage.
Reject signature: the rejection path requires a valid meaning="rejected" HMAC signature so a rogue user cannot reject without leaving an attributable signature.

Acceptance for #285¶

This RFC merged.
api/changecontrol/v1alpha1/changerequest_types.go skeleton compiles and make manifests generate produces a valid CRD.
A commented Helm values stub exists in deploy/helm/cloud-native-dcs/values.yaml, default off.
make verify passes.

The reconciler, semantic-diff framework, admission policy templates, and Promote UI are explicitly out of scope and live in their own issues. The Promote UI MVP (recipe-only) shipped in #290. See Promote a Recipe for the user-facing how-to.