GitOps Operations Runbook

Status: Each entry below is a real incident captured from a production or pre-production deploy of the cndcs-deploy-demo Flux setup. The entries are written so you can recognise the failure mode from a symptom and apply the safe pattern before reproducing the same outage.

If you are running Cloud-Native DCS on a Flux GitOps stack, these are the operating gotchas you need to know about. That covers the worked example in cndcs-deploy-demo/ and any deploy that uses the flux-system Kustomization with prune: true (the flux bootstrap default).

The GitOps loop this runbook is about: a merged pull request reconciled into the running plant, audited with its commit SHA.

For procedures that are not GitOps-specific (cert rotation, password rotation, DR), see Secret and Cert Rotation and the DR Runbook.

Identity changes under prune: true cascade-delete

Symptom. You restructured the repo (renamed a flux/clusters/<x> directory, moved manifests between sub-Kustomization paths, renamed a sub-Kustomization), and the next reconcile deleted everything the sub-Kustomization owned.

What that means concretely depends on what you gave Flux to own, and the two shapes look nothing alike:

• Flux owns the platform. A sub-Kustomization holds the DCS HelmRelease and its dependencies, as in cndcs-deploy-demo. The entire dcs-system namespace clears out: gateway, operators, MQTT, historian Postgres including TimescaleDB data. HelmReleases, Deployments, StatefulSets and PVCs all go.
• Flux owns the plant model. A sub-Kustomization holds only the ISA-88 site tree: Enterprise, Site, Areas, ProcessCells, Units, ControlModules, phase templates, recipes. dcs-system is untouched and the gateway stays up, so nothing looks broken from the outside, while every unit disappears from the HMI and running batches lose the equipment they were bound to.

Both are the same failure. Recognise it by what the pruned Kustomization owned. The namespace that emptied is the wrong key.

Why this happens. flux bootstrap writes the parent flux-system Kustomization with prune: true, and that parent owns the sub-Kustomizations declared under its spec.path. The kustomize-controller records an inventory of the objects it applied and, on the next reconcile, deletes every inventory entry the new build no longer produces. Inventory entries are identified by the object's namespace, name, group and kind. What makes a sub-Kustomization "the same object" across reconciles is therefore its name. When the parent prunes a sub-Kustomization that itself carries prune: true, deleting it garbage-collects everything in the child's own inventory. That second step is the blast radius.

A sub-Kustomization leaves the parent's build when, and only when:

• its metadata.name changes.
• the file declaring it moves out of the parent's spec.path, or off the parent's kustomization.yaml resource list.
• it is deleted from the tree.

What does not trip it, and why that is the trap. Changing a sub-Kustomization's spec.path is an in-place update: the object keeps its identity, the parent re-applies it, nothing is pruned. So renaming a cluster directory and rewriting every spec.path inside it is harmless on its own. That is exactly why the pattern reads as safe right up until the restructure also carries a name with it. Cluster-prefixed child names (demo-apps → production-apps) are the usual way that happens, and dropping a child off the parent's resource list mid-move is the other.

The safe pattern and the cascade, performed in that order against a live parent/child topology. The silent watch pane is the safe rename. The storm is the same rename without the guard.

Reproduce on a test cluster (don't try on prod). Change an identity, not a path:

# In your deploy-ref repo on main — rename the CHILD OBJECT:
sed -i 's|^  name: apps$|  name: apps-v2|' flux/clusters/demo/apps.yaml
git commit -am "rename the apps Kustomization"
git push

# Watch it fire on the next flux-system reconcile: `apps` is in the
# parent's previous inventory and absent from the new build, so the
# parent deletes it, deleting it garbage-collects every object `apps`
# applied, and only then does `apps-v2` build them again.
kubectl -n flux-system get kustomizations -w
kubectl -n dcs-system get helmreleases,deploy -w

Safe patterns for restructuring.

1. Rename with pruning disabled on the object you are renaming. This is the only pattern that actually prevents the cascade, and it is the procedure Flux itself documents for renames. Set prune: false on the sub-Kustomization, land that first and confirm the live object carries it, then rename. The parent still deletes the old object, but with pruning off that delete orphans its resources, and the new object adopts them. Re-enable pruning afterwards.

flux reconcile kustomization flux-system -n flux-system  # fresh window (see below)
kubectl -n flux-system patch kustomization/apps \
  --type=merge -p '{"spec":{"prune":false}}'
flux export kustomization apps -n flux-system   # confirm prune: false is LIVE
git commit -am "rename apps -> apps-v2" && git push
# once apps-v2 is Ready and owns the resources, restore prune: true in git

Mind the parent's clock: the live prune: false is an out-of-band patch on an object the parent re-applies from git, so a periodic parent reconcile landing between the patch and the push silently re-arms pruning underneath you. Syncing the parent first resets its interval timer, which gives the whole patch-confirm-rename-push sequence one quiet window. Re-run the flux export confirm as the last thing before the push if there is any doubt.

Recent Flux versions express the same thing declaratively as spec.deletionPolicy: Orphan.

1. Re-bootstrap when the parent's own path moves. flux bootstrap is idempotent and updates the GitRepository + parent Kustomization spec in one transaction, so the parent never builds from a tree that no longer matches its path. This protects against a failed or flapping parent build. It does not protect a child whose name changed.

flux bootstrap github \
  --owner=<org> --repository=<deploy-ref> \
  --branch=main --path=./flux/clusters/<new> \
  --personal=false --token-auth

1. Suspend → push → patch → resume, for sequencing only. Stopping reconciliation around a multi-commit restructure keeps the parent from building any of the intermediate states.

flux suspend kustomization flux-system -n flux-system
git push                                   # land the rename / restructure
kubectl -n flux-system patch kustomization/flux-system \
  --type=merge -p '{"spec":{"path":"./flux/clusters/<new>"}}'
flux resume kustomization flux-system -n flux-system

Be clear about what this buys: on resume the parent still compares its old inventory against the new build, so an identity change is pruned just the same. Suspending is not a substitute for (1).

What is safe under prune: true. Pure content changes inside an existing path (adding a manifest, editing chart values, bumping chart.spec.version, adding a new HelmRelease at the same path) never trip the cascade, because no object's identity changes and nothing leaves the parent's build. The make sync-deploy-ref chart-pin workflow is safe.

Recovery if the cascade fires. Flux rebuilds from git on the next successful reconcile, and how long that takes is set by what was pruned:

• Manifests only (the plant-model shape): a single reconcile, on the order of seconds. Custom resources carry no volumes, so nothing is lost and the operators rebuild status from the restored spec. Any batch that was running when the Units vanished does not come back with them.
• A HelmRelease (the platform shape): allow roughly ten minutes for the chart to reinstall, and only on a cluster where the data is disposable. Stateless components (gateway, operators, ingress-nginx) recover transparently. Stateful ones do not: TimescaleDB hypertables, CNPG WAL, and any PVC whose reclaimPolicy: Delete are gone. Restore from historian.backup if enabled (Backup and Recovery).

Who has hit this: the cndcs-deploy-demo cutover collapse on 2026-05-06 (cloud-native-dcs#262). The collapse renamed flux/clusters/demo-new/ → flux/clusters/demo/ in the same commit that updated the live cluster's spec.path. The apps Kustomization went through delete-and-recreate. Its inventory was therefore collected before the replacement could re-establish ownership, and cnpg-system and dcs-system were both pruned. The incident record does not preserve which of the three triggers above fired. This page used to attribute it to the spec.path change alone, which cannot be the cause. The data was disposable, so the collapse was an annoyance and no user data was lost. The incident is documented in cndcs-deploy-demo/flux/README.md under "Operating notes" and is the reason this runbook exists.