ChangeRequest Backends

A ChangeRequest (changecontrol.dcs.io/v1alpha1) is the in-cluster handle for a propose-approve-merge round-trip. Once the required electronic signatures are collected and verified, the reconciler dispatches the mutation to a pluggable backend. Three are production backends:

• direct-apply — server-side apply against the API server.
• git-pr — render to YAML, commit on a branch, open a pull request.
• webhook — open a record in an external quality system, poll it for a decision, and hand delivery to one of the two above once it approves.

A fourth is an experimental integration seam:

• in-cluster-queue — wait for an external system (ServiceNow, Jira) to advance the phase. Experimental. No reference integration ships with the product (see below). The DCS side of the contract is defined, and the external consumer that completes it is the deployer's to build.

This guide explains when to pick each backend and how to wire them.

Regardless of backend, the workflow enforces segregation of duties: the user who proposed a ChangeRequest cannot approve it. The gateway rejects the proposer's approve call with 403, and the reconciler refuses to count an approved signature from the proposer when verifying the required meanings before Proposed → Approved. Deployments can opt out with ChangeControlPolicy.spec.requireDistinctApprover: false (default true, enforced even when no policy object exists).

Choosing a backend

Backend	When Approved -> Applied means	Use this when
direct-apply	Reconciler SSA-applies the desired object.	Single-cluster deployments without GitOps; iteration in dev namespaces; the reconciler's SA is allowlisted on the active ChangeControlPolicy.
git-pr	Reconciler edits the target's declaring file on a branch and opens a PR against the configured repository. Applied fires on PR open; Reconciled on PR merge — it attests the merge, and delivery of the merged change belongs to the deployment's reconciler over the site tree.	Strict GitOps shops; canonical Promote flow. Flux (or equivalent) is the only writer admitted by the GitOps-enforcement admission policy.
webhook	Reconciler opens a record in an external quality system and holds in Approved until that record is decided. On approval the configured delegate backend (direct-apply or git-pr) does the actual delivery, so Applied and Reconciled mean what they mean for the delegate.	GMP plants where the eQMS — TrackWise, Veeva Vault QMS, MasterControl, ETQ — is the system of record for change control and has to approve a change before it reaches the plant.
in-cluster-queue (experimental)	Holds in Approved until a deployer-supplied controller flips the phase. No reference integration ships with the product.	Integrations with existing change-management systems where the source of truth is outside the cluster — and you are prepared to build the external consumer yourself. Prefer webhook, which carries the whole round trip.

The deployment-level default is set on the chart values:

gitopsEnforcement:
  changeRequest:
    defaultBackend: in-cluster-queue   # direct-apply | git-pr | webhook | in-cluster-queue (experimental)

The shipped default is in-cluster-queue, the conservative choice. An approved ChangeRequest holds until something acts on it, and nothing auto-applies. Most deployments will want to set direct-apply or git-pr explicitly.

A ChangeRequest may override per-CR via spec.backend. Empty string resolves to the deployment default.

Wiring direct-apply

direct-apply requires no extra configuration. The reconciler SSA-applies spec.desiredObject using the field manager dcs.io/change-request:<namespace>/<name>. Two caveats:

1. The reconciler's ServiceAccount must satisfy the active ChangeControlPolicy for the target namespace. If the target namespace is labelled dcs.io/production=true, the writer must be on the policy's allowlist (see Change Control).
2. SSA conflicts (another field manager owns a field the ChangeRequest is trying to set) move the CR to Failed with a message asking the author to rebase. The UI re-renders the desired object against the latest target and the author re-signs.

Wiring git-pr

The git-pr backend opens pull requests against a configured repository.

The backend end to end: a signed promotion opening a real pull request, and reaching Reconciled when it merges.

Set the chart values:

gitopsEnforcement:
  changeRequest:
    defaultBackend: git-pr
    gitPR:
      repository: "owner/repo"             # or https://github.com/owner/repo
      branch: main                          # base branch PRs target
      provider: github                      # github | gitea
      authMode: token                       # token | github-app; see below
      tokenSecretRef:                       # authMode=token
        name: dcs-changerequest-git-token
        namespace: dcs-system
      githubApp:                            # authMode=github-app
        appID: 0
        installationID: 0
        privateKeySecretRef:
          name: ""
          namespace: ""
      pollInterval: ""                      # merge-poll cadence; empty = 60s
      commitAuthor:                         # optional; see "Commit identity"
        name: ""
        email: ""

Authentication modes

gitPR.authMode selects how the backend authenticates to the git host. The two modes are mutually exclusive and there is no fallback between them: the selected mode's fields must be complete, and a half-configured mode fails validation loudly. You get exactly what you configure.

	token (default)	github-app
Credential	standing personal access token	GitHub App private key
Token lifetime	until revoked	~1 hour (minted per operation, cached)
PR/commit attribution	the token's owner	<app-slug>[bot], verified badge
Survives staff turnover	no — rotation changes the author of record	yes
Setup effort	one Secret	GitHub App + installation + key Secret

authMode: token — Token Secret

Create a Kubernetes Secret in the namespace named in tokenSecretRef, with a single key token containing a personal access token with repo:write scope:

kubectl create secret generic dcs-changerequest-git-token \
    --namespace dcs-system \
    --from-literal=token='ghp_xxxxxxxxxxxxxxxxxxxx'

The token is read on every Apply call (no in-memory cache). Rotate it by patching the Secret, and the next reconcile picks up the new value.

authMode: github-app — GitHub App identity

For a first-class service identity, register a GitHub App (e.g. dcs-change-control) on the org that owns the config repository, grant it Contents: Read and write and Pull requests: Read and write repository permissions, install it on the config repository, and store its PEM private key in a Secret under the key private-key:

kubectl create secret generic dcs-changerequest-gh-app-key \
    --namespace dcs-system \
    --from-file=private-key=dcs-change-control.private-key.pem

Then point the chart at the app identity:

gitopsEnforcement:
  changeRequest:
    gitPR:
      authMode: github-app
      githubApp:
        appID: 12345              # App settings → "App ID"
        installationID: 67890     # …/settings/installations/<id>
        privateKeySecretRef:
          name: dcs-changerequest-gh-app-key
          namespace: dcs-system

Per operation the backend signs a short-lived JWT (RS256, ≤10 minutes) with the app key and exchanges it for an installation access token (~1 hour lifetime), which it caches until near expiry and re-mints transparently. The private key never leaves the cluster, and no standing human-owned credential exists. Commits and pull requests both attribute to <app-slug>[bot] with GitHub's verified-app badge, so staff turnover and key rotation never change the author of record.

Rotate the key by generating a new one in the app settings, updating the Secret, and revoking the old key. The next mint uses the new key. Note that github-app requires provider: github (GitHub or GHES). Gitea has no equivalent.

Provider support (MVP)

The backend ships with native GitHub support (also compatible with self-hosted GitHub Enterprise via the BaseURL configuration in code). Gitea is currently a stub. Set provider: gitea and the backend reports Failed on every reconcile with a pointer at the follow-up issue. Wire your own provider via the Provider interface in internal/controller/changecontrol/backend/gitpr/ if your deployment needs a different host.

What the PR looks like

The backend commits the proposed document into the file that already declares the target: an in-place edit of one document in a (possibly multi-document) site-tree file, with every other document preserved byte for byte. A Create appends the new document (or creates the file), and a Delete removes the document (and the file, when that empties it). The PR therefore diffs the site tree exactly the way a hand-authored change would, and the same Flux Kustomization that reconciles the site tree delivers the merge, with no extra wiring.

The document's text comes from spec.sourceDocument when the proposal carried it: the CLI cuts the proposed document out of the -f file verbatim, so the PR diff is exactly what the engineer edited: comments, key order, and quoting untouched (#1344). The backend refuses to commit carried text that does not parse to exactly spec.desiredObject, does not declare the target, or holds more than one document. The bytes on the branch can therefore never say anything the approval ceremony did not sign. The document count is taken with a YAML stream decoder, so it agrees with what Flux and kubectl would apply. A bare --- scan would count differently, because a separator carrying a comment starts a document like any other. Text that does not parse at all is refused outright (#1362). The gateway applies the same rule when the proposal is submitted, so an author learns about a second document before a reviewer spends a signature on it.

When the field is empty (the gateway Promote flow, whose proposals have no source file), the backend patches the declaring document in place. It walks the document's existing node tree and moves only the values that changed. The comments, the key order, and the quoting the author chose all survive, and a one-field change diffs as that field (#1352). The patched bytes are decoded again and have to read back as exactly spec.desiredObject before they are committed. That is the same post-condition the carried-text path enforces by refusing a mismatch, and for the same reason: formatting is a courtesy, but bytes that say something the approval did not sign are not a formatting problem. A document the patch cannot edit falls back to the full render, which carries the same object under a noisier diff. Two things the renderer cannot reproduce cost a diff line each. An inline comment padded out to a column comes back separated by a single space, and a block sequence written flush with its key comes back indented under it.

The declaring file is named by spec.gitFilePath (repo-relative, filled by the CLI from its -f path). When the field is empty (the gateway Promote flow), the backend searches the repository for the one file declaring the target, and fails loudly when it finds none or several, asking the proposer to supply the path. The path the backend edited is recorded in status.resolution.gitFilePath.

The per-CR canonical tree is retired

Earlier releases committed each approved change as a parallel file at changerequests/<ns>/<apiVersion>/<kind>/<name>.yaml. That produced a second declaration of an object whose authoritative declaration is a site-tree file, and two declarations of one object cannot both be reconciled: whichever tree applied last won, silently, so the site tree re-applied the unapproved baseline over the merged change every interval (#1325). A repository that still carries the tree keeps it as history. The backend neither writes to it nor matches it when resolving the declaring file. Do not point a Kustomization at it.

The edit lands on a deterministic branch cr/<changerequest-namespace>/<name>-<short-uid>. The branch name is stable across retries, so a reconciler restart never opens duplicate PRs for the same ChangeRequest. The PR title is the commit message, and the body explains which ChangeRequest opened it.

The commit message body carries the ChangeRequest identity chain as machine-readable git trailers, so the approving identities survive in merged history independently of who holds the token:

ChangeRequest dcs-dev/cr-abc123: Update MasterRecipe/ipa-master

Increase hold time for the maturation phase.

Change-Request: dcs-dev/cr-abc123
Change-Request-UID: 7f3c…
Proposed-by: Erin Engineer <erin@dcs.example.com>
Approved-by: Sam Supervisor <sam@dcs.example.com>

What that looks like merged, on the commit itself:

The PR can be wider than the approval ceremony

The Promote panel computes its semantic diff from the live cluster object: the current manifest is the before, and the form fields overlaid on a copy of it are the after. The approver therefore reviews exactly the edit that was just made, and the electronic signature binds to that.

This backend diffs a different baseline: the declaring file on the base branch. Anything the cluster carries that Git does not is therefore in the pull request, without having appeared in the ceremony that was signed. The most common source is an earlier change realised through direct-apply, which by design never touches Git.

Both are correct answers to different questions. The modal asks what am I approving, and the pull request asks what is Git behind on. The pull request is the wider of the two, so nothing lands unreviewed (the merge still requires a human to read the diff in front of them). A proposer who expects the PR to match the modal will still be surprised.

So the modal says it. When the proposal will run git-pr, a note under the semantic diff states that the pull request diffs the declaring file on the base branch and carries everything Git is behind on (#1383). It names the branch but never the file: resolving the declaring file needs the repository and a token, and the gateway is deliberately given neither. Only the reconciler holds the repository credential.

The gateway learns two inert facts to render the note, both from the chart values that already configure this backend:

Gateway environment variable	Chart value
CHANGEREQUEST_DEFAULT_BACKEND	gitopsEnforcement.changeRequest.defaultBackend
CHANGEREQUEST_GIT_BRANCH	gitopsEnforcement.changeRequest.gitPR.branch

The same two values render the reconciler's --changerequest-default-backend and --changerequest-git-branch, so the modal cannot describe a backend the reconciler is not running. A deployment that leaves the default unset gets no note on (deployment default), because the gateway does not guess a backend it was not told about. An explicit git-pr selection still gets one.

Commit identity

By default a commit is attributed to whoever owns the configured token, because GitHub bylines the commit against the authenticated identity. For a regulated deployment that means token rotation (a new engineer mints a new PAT) silently changes the author of record on compliance-relevant history, exactly the kind of drift an auditor flags.

Set gitPR.commitAuthor to pin a stable service byline instead:

gitopsEnforcement:
  changeRequest:
    gitPR:
      commitAuthor:
        name: "DCS Change Control"
        email: "change-control@dcs.example.com"

When both fields are set, every commit on the CR branch carries this identity as author and committer regardless of the token owner. A partial value (only one field) is treated as unset. Leaving it unset preserves the pre-existing token-owner attribution byte for byte.

Scope: what this does not cover. commitAuthor sets the commit byline only. The pull request author remains the authenticated identity: GitHub attributes PRs to whoever holds the token, full stop. The human proposer and approver(s) are still recorded, in the commit trailers above and in the AuditRecord written on each state transition. The identity-of-record is preserved regardless. For a first-class verified service identity where the PR itself is opened by <app-slug>[bot] with no standing PAT, use authMode: github-app instead. commitAuthor remains the lightweight option for deployments that don't want to operate a GitHub App.

From Applied to Reconciled

The state machine live: Applied while the pull request is open, Reconciled the poll after it merges, with the identity chain visible in both systems.

The backend reports Applied once the PR is open and polls the PR status every 60 seconds by default. gitPR.pollInterval (a Go duration, e.g. 30s, or the flag --changerequest-git-poll-interval on the batch-operator) tunes the cadence. Demo and capture stacks shorten it so a merge is reflected without a minute-long wait. When the PR merges, the backend reports Reconciled and writes the merge commit SHA into status.resolution.gitCommitSHA.

Reconciled on this backend attests the merge. In-cluster delivery is still ahead: the declaring file now carries the approved change, and the deployment's reconciler over the site tree (Flux) applies it on its next interval. The phase message says exactly that. The audit-linkage bridge (#291) closes the remaining gap: when Flux applies the merged commit, the per-object Reconcile AuditRecord it mints carries the merge SHA and a changeRequestRef back to the originating ChangeRequest.

Wiring webhook (external quality system)

In a GMP plant the system of record for change control is the customer's electronic quality management system. It is never this product. The webhook backend is the seam onto that system. It is deliberately generic: one HTTP contract that every eQMS in the class can be put behind, with no adapter carrying any vendor's name. The vendor-specific part is deployment configuration, or a thin service the customer already runs in front of their quality system.

The decision it settles is recorded in ADR 0049. Three points of it govern everything below.

The external approval layers on top of the electronic signature. It never replaces it. Backends run after Proposed → Approved, so by the time this one opens a record the required signatures have already been collected and HMAC-verified here. That ordering is structural, and no policy choice can invert it. Our signature binds a named identity to this exact change content in a key store we control, and an eQMS decision reaches us as a claim by a machine account about something that happened elsewhere.

The loop is polled. The operator exposes no inbound HTTP surface. A callback could only accelerate this poll, and it is not implemented.

A close that is not an approval is a rejection, and never a failure. The ChangeRequest ends in the terminal Rejected phase, audited as a rejection, with the quality system's own word for the close recorded in status.resolution.externalRecordState.

Chart values

gitopsEnforcement:
  changeRequest:
    defaultBackend: webhook
    webhook:
      endpoint: https://qms.example.com/api/v1/change-records
      delegate: git-pr            # direct-apply | git-pr
      tokenSecretRef:
        name: qms-api-token       # key "token"
        namespace: dcs-system
      tokenHeader: ""             # empty -> Authorization: Bearer <token>
      tokenPrefix: ""
      approvedStates:             # required
        - Approved
        - Closed-Approved
      rejectedStates:
        - Rejected
        - Cancelled
        - Void
        - Superseded
      pollInterval: 5m

delegate names the backend that performs the change once the record is approved. The webhook backend gates and never applies anything itself, so gating on a quality system does not cost a deployment its GitOps delivery. Naming a delegate the deployment has not configured (git-pr with no gitPR.repository) fails the operator at startup, before the first approval needs it.

approvedStates is required. Reading the record's state fails closed. A state on neither list means the record is still open, so a mismatched vocabulary delays changes. A mapping you get wrong is discovered by a change that does not move, and a change can never release itself through a wrong mapping.

If the quality system presents a certificate from a private CA, add that CA to the operator pod's trust store the usual Kubernetes way (mount a ConfigMap over /etc/ssl/certs/). The backend has no CA setting of its own.

The integration contract

Two calls, both JSON.

Open a record. POST <endpoint>, with the credential header and Idempotency-Key: <ChangeRequest UID>:

{
  "changeRequest": {
    "namespace": "dcs-dev",
    "name": "promote-mash-hold-via-eqms",
    "uid": "1ffa4595-0ab6-4fbc-96f2-dced3d7a1177"
  },
  "target": {
    "apiVersion": "recipe.dcs.io/v1alpha1",
    "kind": "MasterRecipe",
    "namespace": "site-newark",
    "name": "ipa-master-v3"
  },
  "operation": "Update",
  "reason": "CCR-4471: promote dev recipe revision to prod",
  "signatures": [
    {"meaning": "proposed", "user": "kbrewer", "displayName": "K. Brewer", "signedAt": "2026-08-11T09:00:00Z"},
    {"meaning": "approved", "user": "mqa", "displayName": "M. QA", "signedAt": "2026-08-11T09:14:00Z"}
  ],
  "document": "apiVersion: recipe.dcs.io/v1alpha1\nkind: MasterRecipe\n…",
  "baseDocument": "apiVersion: recipe.dcs.io/v1alpha1\nkind: MasterRecipe\n…"
}

document is the proposed object rendered as YAML, and baseDocument is the target as it stood when the change entered review. A reviewer in the quality system therefore sees the diff, and the result never arrives alone. A Create has no base and a Delete has no document. Both fields are omitted when empty.

The signatures array carries attribution and not the HMAC digests. The digest binds to our key store, so it proves nothing to a party that does not hold the key.

The response must name the record:

{"id": "QMS-1234", "url": "https://qms.example.com/records/QMS-1234", "state": "Under Review"}

id is required. Without it there is nothing to poll, and the ChangeRequest fails. url is optional, recorded for humans, and never requested by the backend. state is optional here, and an approving state on the create response releases the change on that same pass.

Poll the record. GET <endpoint>/<id>, with the credential header:

{"id": "QMS-1234", "url": "https://qms.example.com/records/QMS-1234", "state": "Approved"}

state is required on a poll. A record served without one carries no decision, so the ChangeRequest fails. It never parks forever against a contract the endpoint is not honouring.

Note what the contract does not contain. There is no field by which a response can say where to poll. The poll URL is always <endpoint>/<id>, so the credential you minted for your quality system can only ever be sent to the host you configured.

Idempotency is on your side. The backend polls the identifier it recorded and creates no second record, but a crash between the create call and the status write can still re-issue the create. Your endpoint must treat a repeated Idempotency-Key as a lookup of the record it already created for that key.

Non-2xx responses

Any non-2xx from either call is treated as an outage, and never as a decision. The reconciler stays in Approved and retries. A quality system that is down never rejects a change by being down.

What the operator sees

While the record is open the ChangeRequest stays in Approved with an Applied=False / BackendPending condition naming the record and its last observed state, and status.resolution carries:

Field	Meaning
externalRecordID	The quality system's identifier for the record.
externalRecordURL	Deep link for a reviewer, when the system supplied one.
externalRecordState	The last observed state, in the quality system's own vocabulary.

On approval the delegate runs and the request advances through the delegate's own phases, with the external record fields preserved alongside whatever the delegate resolved (a pull request URL, a commit SHA). On any other close the request ends in Rejected.

Wiring in-cluster-queue

Experimental — no reference integration

in-cluster-queue ships as a defined contract with no demonstrated consumer. The DCS side is ~5 lines (it reports Pending and waits). The external system that watches Approved requests and patches the phase back is the deployer's to build, and the product ships no reference adapter, no worked example of the external side, and no end-to-end test proving the reconciler resumes from an externally-written phase. Treat it as an integration point to prototype against, and never as a turnkey backend. Use direct-apply or git-pr for production change control until a customer SOP requires the ticket-driven path. (Tracking: #649, rationale in ADR 0013.)

in-cluster-queue is the integration seam for external change-management systems. Once a ChangeRequest reaches Approved, the backend reports Pending indefinitely. A deployer-supplied controller (or a human with kubectl patch) is expected to:

1. Watch ChangeRequest objects.
2. Push the request through the external system's workflow.

3. Once approved externally, patch the ChangeRequest status:

   status:
     phase: Reconciled       # or Applied if the writer hasn't yet observed it
     resolution:
       backend: in-cluster-queue
       auditRecordRef: SNOW-CR-12345   # or whatever your system uses
   

The reconciler still validates signatures and emits AuditRecords on every transition the DCS observes. This backend bypasses none of the propose-approve safety net. It only delays the Approved -> Applied step until the external system signals readiness.

State machine reference

                                +--> Withdrawn (terminal)
                                |
                +-> Proposed ---+--> Rejected (terminal)
                |               |         ^        ^
                |               |         |        |
                |               +--> Approved      |
                |                       |          |
   spec edit --+                        +--> Applied --> Reconciled (terminal)
                                        |       ^
                                        |       +-- backend observes target
                                        |
                                        +--> Failed --[author retries]--> Proposed
                                                +--[author gives up]--> Withdrawn

The two extra edges into Rejected are the webhook backend reporting that the external quality record closed as something other than an approval. They land in Rejected deliberately, and never in Failed. The Failed phase says the backend could not do its job and offers the author a re-stage, so recording a quality decision there would put a false statement into a Part 11 record. See ADR 0049.

Signalling decisions

The reconciler interprets these annotations on a ChangeRequest:

Annotation	Meaning
dcs.io/withdraw=true	Author withdraws (any non-terminal phase).
dcs.io/reject=true (paired with a meaning="rejected" HMAC signature)	Reviewer rejects. High-severity audit.
dcs.io/retry=true	Force a Failed CR back to Proposed without editing the spec (useful when the original failure was an external service outage).

Editing spec on a Failed CR also returns it to Proposed automatically on the next reconcile (status.observedGeneration < metadata.generation).

Audit trail

Every state transition emits an AuditRecord in the ChangeRequest's namespace, with:

• target — the resource the ChangeRequest is mutating.
• correlationID — the ChangeRequest's UID, so all transitions for one CR can be queried together.
• signature — the canonical "approved" signature when one is present.
• resourceSHA — the SHA-256 of the canonical desired-object spec.

Combined with the git-pr backend's commit SHA, this satisfies 21 CFR Part 11 §11.10(k) end to end: every change carries an attributable signature, a contemporaneous timestamp, and a tamper-evident link to the applied manifest.

Under the webhook backend the same records are written, and status.resolution additionally names the external quality record that gated the change. The attributable signature is still ours: the eQMS record is a second review, recorded alongside the signature. It never stands in place of it.

Related Documentation

• Promote a Recipe — the user-facing how-to whose Promote button submits the ChangeRequests these backends realise.
• Integration -- the routing index that sends a reader holding an eQMS product to this page.