Skip to content

Turning On GitOps Enforcement

This guide is for the platform administrator who runs helm upgrade on a Cloud-Native DCS cluster and wants to flip on gitopsEnforcement.enabled without breaking the people already using the system.

For the why of the design, see GitOps for Automation Engineers. For the what of the contract, see the GitOps Enforcement RFC.

What enforcement does

When gitopsEnforcement.enabled=true, the chart installs a ValidatingAdmissionPolicy (CEL, GA in Kubernetes 1.30+) that rejects writes to engineering-class CRDs unless the requester is one of:

  • the GitOps reconciler ServiceAccount configured under gitopsEnforcement.reconciler.serviceAccount
  • a ServiceAccount in the chart's install namespace (system:serviceaccounts:<release-namespace>), which covers all in-cluster DCS operator reconciles (batch / physical / procedural / control)
  • a member of the dcs.io:break-glass group bound to the gitopsEnforcement.breakGlassRole ClusterRole
  • system:masters (the cluster's built-in admin escape)

The in-cluster operator carve-out exists because reconcile loops legitimately write to engineering CRDs (e.g. batch-operator annotating a Unit during allocation). Without it, batches stall in Allocating as soon as enforcement is turned on (see #372). The carve-out is namespace-scoped: only SAs in the chart's install namespace qualify, and an arbitrary SA elsewhere in the cluster stays outside it.

Operational-class CRDs (Batch, Phase, AuditRecord, etc.) accept writes from any authenticated caller, exactly as they did before enforcement was on.

The classification is carried on every CRD as a label (dcs.io/ownership-class=engineering|operational|hybrid). See #283 for the full taxonomy.

What the policy is protecting: the same signed ceremony gates a routine change and a production one, whichever route the change takes.

Reconcile engine

The admission policy is engine-neutral: it allowlists a ServiceAccount identity (gitopsEnforcement.reconciler.serviceAccount) and does not care what drives it. Point it at Flux's kustomize-controller, ArgoCD's argocd-application-controller, a ticket-system bridge SA, or a human's time-bounded apply identity, and enforcement behaves identically.

Declare the engine so the chart can reason about the other half of the compliance story, audit linkage:

gitopsEnforcement:
  reconciler:
    engine: flux          # flux | argocd | servicenow | other

Audit linkage, the §11.10(k) loop that stamps each reconcile with its Git commit SHA, is currently implemented for Flux only (#816). To keep neutrality honest on the record, the chart fails the render if you enable enforcement with a non-flux engine:

gitopsEnforcement: reconciler.engine="argocd" has no audit-linkage adapter.
The 21 CFR 11 §11.10(k) traceability loop … will NOT close on this engine …

You have two ways forward:

  • set reconciler.engine=flux and configure audit linkage, or
  • set reconciler.acknowledgeNoAuditLinkage=true to run enforcement without linkage. You get the admission gate on any engine, and explicitly accept that no SHA-stamped AuditRecords are produced until an adapter ships.

Pre-flight checklist

Before you flip the switch:

Check Why
Kubernetes >= 1.30 ValidatingAdmissionPolicy is GA at v1; older clusters reject the resources
Flux (or equivalent) deployed The reconciler SA is the only writer the policy admits — without it, all engineering writes are blocked
gitopsEnforcement.reconciler.serviceAccount.{namespace,name} matches reality Verify with kubectl get sa -n flux-system (or your engine's namespace)
gitopsEnforcement.breakGlassRole exists or is acceptable as auto-created The chart creates it if absent. To use a custom name, set the value before upgrade
Real humans bound to the break-glass role Bind via ClusterRoleBinding whose subjects include both the user and the dcs.io:break-glass group
gitopsEnforcement.classes.engineering.mode chosen propose-only is the default; read-only is stricter; direct-write installs no policy
dcs admin gitops-preflight passes Tracked as follow-up to #288; when it lands, run it before every enable

Step-by-step

  1. Stage the change in a values file.

    # values-prod-gitops.yaml
gitopsEnforcement:
  enabled: true
  reconciler:
    engine: flux
    serviceAccount:
      namespace: flux-system
      name: kustomize-controller
  breakGlassRole: dcs:admin:break-glass
  classes:
    engineering:
      mode: propose-only

  2. Bind a real human to the break-glass role.

    # break-glass-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dcs-break-glass-oncall
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dcs:admin:break-glass
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: oncall@example.com
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: dcs.io:break-glass

    The dcs.io:break-glass group must be present, because the CEL policy reads it out of request.userInfo.groups to grant the override.

  3. Helm upgrade.

    helm upgrade --reuse-values --install cloud-native-dcs \
  oci://ghcr.io/cloud-native-dcs/charts/cloud-native-dcs \
  --version <pinned-version> \
  -f values-prod-gitops.yaml

    Confirm the installed resources:

    kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding \
  -l app.kubernetes.io/instance=cloud-native-dcs
kubectl get clusterrole dcs:admin:break-glass
Steps 4 and 5 performed against a live policy. RBAC says the engineer may write, the policy redirects her anyway, and the reconciler still gets through.

  1. Smoke-test the deny path.

    Apply an engineering CRD as a non-Flux user. The expected outcome is a Forbidden from the admission policy with a propose-only message pointing at the ChangeRequest workflow.

    kubectl --as=alice apply -f some-master-recipe.yaml
# Error from server (Forbidden): Engineering CRD writes require a
# ChangeRequest in propose-only mode...

    Impersonate someone who is actually authorised. The user you test as must already hold RBAC to write the CRD, or the request never reaches admission and you are testing the wrong thing:

    # alice with no RBAC — this is NOT an enforcement result
Error from server (Forbidden): masterrecipes.recipe.dcs.io is forbidden:
  User "alice" cannot get resource "masterrecipes" in API group "recipe.dcs.io"

    Both messages say Forbidden, and only the second names the policy. A plain RBAC refusal looks like a passing test and proves nothing about enforcement, so read the message itself. The exit code cannot tell the two refusals apart. Test as an engineer who could write the CRD before enforcement was enabled: that is the person the policy exists to redirect.

  2. Smoke-test the allow path.

    Verify Flux's existing reconcile still admits writes. Flux logs should show successful applies on the next reconcile cycle. If they fail, the most likely cause is reconciler.serviceAccount.namespace or name not matching the real reconciler SA.

Hybrid-mode notes

For the MVP, hybrid mode rejects all spec writes from non-Flux SAs: it treats hybrid CRDs identically to engineering. Field-level enforcement (allow the dcs.io/command annotation and named UI-mutable fields while gating the spec) is a follow-up, tracked in #288.

If you have UI-driven workflows that mutate Batch.spec.parameterOverrides or set the dcs.io/command annotation directly, leave classes.hybrid.mode=direct-write (the default) until the field-level policy ships.

Rollback

Set gitopsEnforcement.enabled=false and helm upgrade. The ValidatingAdmissionPolicy, its binding, and the break-glass ClusterRole are removed by Helm. Any humans bound to the break-glass role retain their binding (Helm does not own those ClusterRoleBinding resources). That is intentional, because the on-call rotation should not be coupled to the chart release lifecycle.