Skip to content

ADR 0060: A mutating endpoint declares what it refuses

Status: Accepted Date: 2026-08-14 Issue: #1565 Related: #1561 (the delete guard), ADR 0059 (the ceremony tiers)

Context

For the product's whole life, DELETE /api/v1/sites/{site}/batches/{name} deleted a Batch in any state, including Running. The finalizer then deallocated the units and garbage-collected the live procedure tree, with none of the safe-state sequencing that Stop and Abort exist to provide. The only thing in the way was a dialog asking the operator to retype the batch name.

#1561 fixed the four handlers it found. This record is about the other question the founder asked, which is how a control system shipped that action for a year without anyone stopping on it.

The archaeology says it was never one decision. It was four layers, each correct on its own terms, that between them left the question unasked.

The endpoints were generated. All twenty DELETE handlers landed in one commit on 2026-02-17, titled "Add full CRUD for all resources across API, CLI, and GUI". The CLI half is one generic newDeleteSubCmd helper called nineteen times, differing only in the URL. Uniformity was the goal, and the shape has no slot for a per-kind precondition. A question that has nowhere to be written does not get asked.

The one guard on the route is about records. productionGuard refuses deletion of Batches, ControlRecipes and procedural instances when the gateway runs in ModeProduction, citing 21 CFR Part 11 record retention. It sits on the route in routes.go, where it reads like a safety check on a delete. It is no such thing. The default mode is ModeDevelopment, whose own comment says it "allows all operations including deleting batch records".

The ceremony was built for attribution and was read as authorization. The typed-name dialog arrived in April 2026 to replace window.confirm(). Its commit message gives the reason plainly: Part 11 traceability, meaning audit context, a reason field and name reconfirmation. It made the action attributable. Two months later, #687 re-ranked every confirmation in the product "by physical hazard", which is the right axis and the review that should have caught this. It did not, because the only thing that review could produce was a friction tier, and batch delete already wore the strongest tier in the product. It was ranked and passed over.

The tests asked the other question. TestDelete_AbsentTargetIs404 walks a table of DELETE routes and asks each what it answers when the target is absent. Somebody sat down, enumerated the destructive endpoints, and asked one question of every one of them, nine days before the incident. The question was what if it is not there. Nobody asked what if it is running.

Underneath all four is one thing. ANSI/ISA-88.00.01-2010 Clause 7.5.1 says the procedural state model is "the complete set of states, commands, and allowed transitions" for a procedural element, and Table 3 enumerates them. RUNNING leaves only through COMPLETE, STOPPING, HOLDING, PAUSING or ABORTING, and the two that terminate it are acting states that run logic to bring equipment somewhere safe. Delete is not in that matrix.

Our command handlers inherited the matrix and check state, because a state machine forces them to. Our CRUD handlers did not, because delete is not a command. Being outside the model was treated as being outside the model's concern, when what it actually means is that the model does not permit it.

Decision

Every mutating gateway route declares, in the tree, what execution state it refuses. The declaration lives in scripts/.mutation-preconditions.tsv, one row per DELETE, PUT and POST route, and make lint-mutation-preconditions fails when a route has no row or a row has no route.

A row carries one of three verdicts, and there is no fourth:

  • guarded — the handler reads the target's state and refuses when the action is unsafe. The row names a Go test that asserts the refusal, and the gate checks the function exists. A guard nothing asserts is the shape #1475 named, where code reads as a check and nothing goes red when it is dropped.
  • stateless — the target's state cannot make the action unsafe. The row carries the reason in writing, and it has to be a reason. "An integration credential; revoking it stops an MES from calling in, and the batches it started go on running under their own supervision" is a reason. "No state" is not.
  • gap — the action can be unsafe and is not refused. The row names an issue, and the gate requires that issue to be open. A gap row puts a defect on the record. It exempts nobody from the rule.

There is no allowlist. An allowlist would be a fourth verdict meaning "do not ask", and not asking is the entire defect.

Where a route is guarded on an execution state, the allow-list of permitted states is written as an allow-list, following ADR 0057. A state neither set names refuses. So adding a member to a status enum fails loudly at this gate, where the alternative is that it becomes silently mutable mid-run. For procedural elements the permitted set is not ours to invent. It is Clause 7.5.2's three Final States (COMPLETE, STOPPED, ABORTED) plus the Initial State (IDLE). Those four are exactly the states in which the element is directing no actions.

The gateway is where this is enforced, because the gateway is the operator API and there are no admission webhooks over the CRDs. Direct kubectl against the cluster is outside operator RBAC and is cluster-admin territory. That boundary is deliberate, and it is why the handler is the last line of several rather than one.

Alternatives Considered

Fix the four handlers and move on. This is what #1561 did, and it was right for the hazard in front of it. The sweep then found the Procedure unguarded, along with thirty-six other routes. The Procedure is the root of the very tree #1561 was protecting. Four layers of review had already passed over this class. A fifth pass with no mechanism behind it would be the same bet again.

An admission webhook over the CRDs. This would enforce against kubectl too, which the gateway guard cannot. It was rejected for now on two grounds. The product ships no webhooks over its own CRDs today, so this would be a new component with its own certificate lifecycle and failure mode. A webhook that fails closed on a control plane is a new way to lose the cluster. The threat it addresses is also narrow: a cluster-admin with direct API access, who sits outside the operator RBAC boundary by design and can remove the webhook as easily as the object. It stays on the table for the day a customer's platform team shares the cluster.

Require a refusal test per route with no declaration file. Simpler, and it was the first shape considered. It cannot express stateless, and that is the verdict most routes deserve. There is no test to write for "an auth lockout holds no equipment". The gate would then have had to guess which routes it applied to, and a gate that guesses its own scope is how a route slips out of it silently.

A stronger ceremony on the unsafe actions. Rejected on the founder's framing. Friction is not enforcement, and the incident is the proof. Batch delete wore the strongest ceremony in the product on the day it destroyed a running batch.

Consequences

A new mutating route cannot merge until somebody has written down which of the three verdicts it is. That is the cost, and it is the point: roughly one sentence of thought at the moment the endpoint is designed, in place of a year of a running batch being deletable.

The declaration file also states the size of the debt out loud. No route on the hundred-and-fifty-seven-route surface is a gap row today. The count was thirty-seven of fifty-six when this record was written. The sweeps closed those. Widening to POST then found four more, and those closed in turn. A green gate does not mean the surface is safe. It means every part of it has been looked at and the unsafe parts are named, which is the difference between this state of affairs and the one that produced the incident.

Recipe revert was one of those four, and it closed in #1581. It replaced a Released recipe's whole spec and cleared its approval attribution, which is the edit the PUT route refuses. Its guard is worth reading for one thing this record did not anticipate. The obvious fix was the phase set the PUT route already used, and that fix would have deleted the feature: a RecipeRevision is written only on approval, and revert is the only thing in the product that moves a recipe back to Draft, so a revert restricted to Draft could never have been called once. ISA-88.00.03-2003 settles it in the direction of naming the hazard instead. Its Table 7 is a list of life cycle states rather than a transition matrix, and its own annex says there is no standard state model and that reversals have to be considered. The two states the guard refuses are therefore the two the standard describes as out in the plant, Released and Effective, and #1569's rule that a refusal must leave the endpoint's purpose reachable is what picked them.

The other three were raw write paths that reached a live output without the equipment-mode barrier the modelled tag routes apply (#1580). Closing them settled a question the earlier sweeps had not had to ask. The runtime proxy's write route was the same act as the unit tag route. It carries the address in its body where the tag route carries it in the path, and it forwards the same payload to the same runtime endpoint. So it took the same three barriers, applied by the same code. The other two address a device below the tag model. A raw channel and a bare OPC UA node belong to no ControlModule the gateway can name, so there is no ISA-88 equipment mode to read for them at all. They ask what the plant model can always answer instead, which is whether a Unit is executing through this device. That is the question the delete and update guards already ask of the same object.

The consequence is that the raw roads are stricter than the modelled one. A control module in Manual admits an operator write while its batch runs, and a raw write to the same device is refused whatever the mode. That asymmetry is deliberate, and it follows the #1576 rule one layer down. The permissive answer needs a mode the gateway cannot read on these roads. A guard that granted it would be guessing. The refusal names the modelled road as the way to take control of a module, and commissioning happens on equipment that is not executing. The endpoint's purpose therefore stays reachable, which is what #1569 requires of a refusal.

Widening to POST also found six guards the product already carried and nothing asserted. They are an alarm acknowledgement's state gate, an ad-hoc unit procedure's idle-unit check, a finalized batch record's immutability, a terminal change request, and two recipe lifecycle transitions. A guarded verdict has to name a test the gate checks exists, so the file could not be filled in without writing them (internal/gateway/post_precondition_test.go). That is the same class the first sweep found in the recipe edit guard. It is worth naming as a second thing this gate does. It does not only find missing refusals. It finds refusals nothing would have noticed the loss of.

The PUT sweep in #1568 added one thing this record did not anticipate. On the delete side a verdict is a property of the route, because there is one action and it either happens or it does not. On the update side it is a property of the route and a set of fields. Renaming a unit's HMI symbol during a batch harms nothing, and rewriting the same unit's cmRoles rebinds the equipment the running phases resolve through. So a guarded PUT names two things: the state in which the target is executing, and the spec fields that may still move while it is. That second set is written as an allow-list for the reason the first one is. A field neither list names refuses, so a field added to a spec fails at this gate rather than becoming silently writable mid-run.

The last route the sweep answered, PUT /api/v1/sites/{site}/alarmdefinitions/{name}, added a third thing (#1576). A refusal has to be expressible in what the guard can actually read. An alarm limit that trips earlier and one that trips later are genuinely different in safety terms. ISA-18.2 puts a limit change under management of change rather than under prohibition. So a direction-aware verdict was the tempting answer, and the guard cannot tell the two directions apart honestly. The limits are strings. "Looser" inverts between a high alarm and a low one, and it means a wider band for a deviation alarm. Half of what the route can write has no direction at all, including a condition type change, a different tag, and an exception action added or removed. A guard that classifies safe from a parsed float and refuses everything else is a much larger surface reaching the same answer, and its safe branch can be silently wrong. So the verdict stays field-scoped and the nuance moves to where it belongs. The refusal names ISA-18.2 shelving, which is the product's answer to the mid-batch need and is reachable at PermOperate. A refusal that leaves the endpoint's purpose unreachable is a different defect, and the two alarm DELETE routes were guarded on that same principle a sweep earlier.

The gate also holds the delete-of-absent table complete. That table is the other enumeration named in the Context above. TestDelete_AbsentTargetIs404 says in its own comment that it covers every DELETE route deliberately. It covered fifteen of thirty-one for a year. One of the sixteen it never named was DELETE /api/v1/sites/{site}/recipes/control/{name}. That route was the exact delete-without-a-preceding-Get shape the comment describes, and it answered 500. Nothing compared the table to the route list, so it drifted the way any hand-kept copy of a generated set drifts. Each row now carries the route pattern as registered, and the gate requires every DELETE it finds in routes.go to appear there. Both contracts ask about the same enumeration. One route extraction therefore answers both, and there is no second parser (#1570).

A row in that table answers 404 unless it says in writing why it does not. Two do. Both are revocations. DELETE /api/v1/auth/sessions/{id} revokes an unknown session ID by planting a tombstone. That is what lets an administrator kill a session another replica issued, so a 404 would refuse the case the endpoint exists for. DELETE /api/v1/auth/lockouts/{username} clears the username's failure entry whether or not it had crossed the lockout threshold. Absence is therefore not a state the caller can observe. Both report what they found in the body.

One implementation note, because the gate failed its own first CI run on it. The gap rows are checked against one gh api listing of the repository's open issues. The obvious spelling, gh issue view <n> --json state per row, drives the GraphQL surface and resolves the repository from the git remote, and it answers nothing for a workflow's default github.token. Every row came back "could not be read". The gh api form is the house idiom for the same reason in ci-diff-base.sh, require-green-ci.sh and lint-ci-orphan-packages.sh. It also costs one round trip for the whole file, where the per-row form costs one per gap.

The rule covers DELETE, PUT and POST. It stopped at DELETE and PUT when this record was written, on the reasoning that POST is create or command. A create has no target to be in a state, and a command is already inside the state machine that carries its own preconditions. The reasoning was sound and its premise was already false.

POST /api/v1/apply is create-or-update. It dispatches thirty kind tokens to functions that each end in createOrUpdate. That helper Gets the object and Updates it when it is there, so the update half acts on an existing target outside the state machine. It reached the running unit's cmRoles, the chart the engine is walking, the compiled control logic and the live alarm limit. Those are every target the PUT sweep had just guarded. It answered 200 against the same cluster state where the REST route answers 409. The carve-out was reasoned about the verb. What the route does is the thing it needed to be reasoned about, and asking of the wrong thing is the same move that let a running batch be deletable (#1578).

The filter widened to the whole verb. The narrower scope on offer was "POST routes that act on an existing target", and that one is not computable from routes.go. A gate that guesses its own scope is how a route slips out of it silently. It is the shape the third alternative above was rejected for. GET and HEAD stay out because they write nothing at all. That is a property of the method, and it is never a claim about any handler. Most POST rows are stateless on the create-has-no-target reasoning, and writing them down is the point. The one that was not a create looked exactly like the others from outside.

Apply's own verdict is guarded, per kind, in internal/gateway/apply_exec_guard.go. Three things about it are worth carrying forward. The refusal reuses the allow-lists the PUT routes already carry, so one place decides a field's mid-execution writability. The check sits inside the client.Client the apply functions write through. That sits upstream of the write and downstream of the dispatch table. It therefore compares the spec the handler assembled, which is the #1568 rule. That includes applyUnit's preservation of the runtime ports and node selector a generic document omits, and a request-body comparison would have read those as a change. Third, a kind whose precondition is not the object's own status says so. A SyncBarrier reaches Waiting and a CoordinationBlock reaches Active on their first reconcile, with no batch involved. Refusing on those would have made both kinds uneditable from the second they were created. What makes an edit to either unsafe is the procedural elements they name, which is #1566's lesson a third time.

The widening also settled what a restore does about a refusal. dcs restore crds posts every document in a backup. Skipping the refused ones would leave a partial plant model written over a live one, which is its own hazard. It stops at the first 409 instead, reports it as REFUSED with the gateway's reason, and exits non-zero. The documents already applied are that backup's own content, and apply is an upsert. A re-run after the cause is cleared therefore lands the rest. An up-front pre-flight was considered and rejected. It could only ask about units. A check that claims to cover the whole surface while covering one part of it is #1570's shape.

This does not replace the ceremony tiers. ADR 0059 governs how much friction a permitted action carries, and it remains the right question for actions that are permitted. What #1565 established is the order: an action is refused or it is not, and only once it is permitted does the ceremony have anything to rank.