How a Running Batch Stayed Deletable¶

Status: Investigation complete (#1565) Issue: #1565 Ruling: ADR 0060 Scope: every mutating route on the gateway API

What happened¶

For the product's whole life, DELETE /api/v1/sites/{site}/batches/{name} deleted a Batch in any state. The finalizer then deallocated the units and garbage-collected the running procedure tree, destroying live phases with none of the safe-state sequencing that Stop and Abort exist to provide. The same held for the ad-hoc Phase, Operation and UnitProcedure delete endpoints. The only barrier was the UI's typed-name dialog, which is friction.

#1561 fixed those four handlers on 2026-08-14. This document answers the founder's other question, which is how the action shipped in the first place and stayed shipped through four reviews that each looked straight at it.

Root cause¶

Four layers, each defensible on its own terms, and one thing underneath all of them.

The endpoints were generated¶

Every DELETE handler in the gateway landed in commit 7cf1458d on 2026-02-17, titled "Add full CRUD for all resources across API, CLI, and GUI". Its own body counts the work: "20 DELETE handlers across all resource types", "19 delete subcommands via generic helper", "delete buttons on every table row".

The CLI half is literally one function. newDeleteSubCmd(resource, article, siteScoped, pathFn) is called nineteen times in cmd/dcs/internal/cmd/delete.go, and the only thing that varies between an alarm and a running batch is the URL. The gateway half is the same shape written out longhand: build an ObjectMeta from the path values, call client.Delete, record the audit, return 200.

Nine of the thirty-one DELETE handlers still never Get the object at all. A handler that does not read its target cannot have a precondition, so the question was not skipped. There was no place in the abstraction to put it.

The framing was CRUD completeness, which is a table with a row per kind and a column per verb. In that table every cell is the same question, and the answer is yes.

The guard on the route is about records¶

routes.go registers the batch delete as s.productionGuard("Batch", s.handleDeleteBatch), which reads at a glance like a safety check. It is a 21 CFR Part 11 record-retention check, and it fires only when the gateway runs in ModeProduction. The default is ModeDevelopment, whose declaration in server.go says so plainly:

ModeDevelopment allows all operations including deleting batch records.

So the one thing standing between the route and the handler was a control on whether an electronic record may be destroyed. Nothing on that path had an opinion about whether equipment was moving.

The ceremony was built for attribution and was read as authorization¶

The typed-name dialog arrived on 2026-04-15 in "HMI: replace window.confirm() with audited delete-confirm modal". Its stated reason is worth quoting, because it is the pivot of the whole story:

For 21 CFR Part 11 traceability, replace with a custom modal that requires retyping the resource name and entering a reason.

Audit context, a reason field, name reconfirmation. Every one of those makes the deletion attributable. None of them makes it refused. From that day the action wore the visual language of a guarded operation while being exactly as permitted as before.

Then, on 2026-07-02, #687 re-ranked every confirmation in the product. Its commit subject is "re-rank confirmation ladder by physical hazard", and its first line is "Friction now scales with physical consequence instead of per-dialog accident".

That is the correct axis and the correct instinct. It is the review that should have caught this, and it went through interlock bypass, abort, failover and diagnose-restart in detail. It passed over batch delete because the only output that review could produce was a friction tier, and batch delete already wore the top tier. It was ranked, found adequate, and left alone.

A hazard review whose only verb is "add friction" cannot conclude "refuse".

The tests asked the other question¶

internal/gateway/delete_notfound_test.go contains TestDelete_AbsentTargetIs404, added on 2026-08-05 for #1269. Its comment explains its own design:

The table is every DELETE route, deliberately, rather than only the eight that were broken. A handler that deletes without a preceding Get is the shape that regresses.

Nine days before the incident, somebody enumerated the destructive endpoints and asked one question of every one of them. The question was what the endpoint answers when the target is absent. The question of what it answers when the target is running was never in the frame, at the exact moment the enumeration existed to ask it.

The repository does have refusal-shaped tests, and their axes are revealing: TestAcknowledgeAlarm_NonOperatorForbidden is authorization, TestControlModule_CreateRejectsUnknownDeviceType is input validation, TestHandleArchiveIntegrityRun_RefusesConcurrentRun is concurrency. TestCommand_BatchStartBlockedOnRuntimeNotReady is a state precondition, and it is on a command route. Commands check state because a state machine forces them to. CRUD routes did not, and that is the last layer.

Underneath: delete is not in the transition matrix¶

ANSI/ISA-88.00.01-2010 Clause 7.5.1 defines the procedural state model as "the complete set of states, commands, and allowed transitions for each procedural element". Table 3 enumerates them. RUNNING appears with exactly five exits: COMPLETE on normal completion, and STOPPING, HOLDING, PAUSING or ABORTING on the corresponding command. Clause 7.5.2 describes the two that terminate a run as acting states that execute logic, STOPPING performing "a controlled normal stop" and ABORTING "an immediate abnormal stop".

Deletion is not a command in that model, and destruction is not one of RUNNING's exits. The product read that absence as "outside the model's concern" and implemented delete outside the safety model entirely. The standard's own wording says the opposite: the set is complete, so a transition it does not contain is one the model does not permit.

That single misreading is what all four layers sit on. Every one of them was about who is calling, whether the request is well-formed, whether the record may be destroyed, and how much the operator should be made to type. Not one of them was about whether the plant was moving.

The idea existed one layer down¶

The control-operator refuses a ControlProgram hot-swap while any Batch on the target unit is Running, Holding, Held or Aborting. It sets a HotSwapDeferred condition, emits a Rejected AuditRecord, and takes an override only with an explicit annotation and a written reason. That row has been in docs/compliance/isa88.md under "Operational safeguards: control-logic change while active" for a long time.

So the product already knew that changing something under a running batch is a thing to refuse, and already had the vocabulary for it: read the batch phase, answer no, record the rejection. It lived in a controller, scoped to one kind of change, and nothing carried it up to the API surface where every other change arrives. That is the sharpest evidence that this was a missing generalisation and not a missing idea.

What the security process had to say¶

Nothing, and the reason is structural. Nobody was asleep.

docs/sdl/secure-coding-standard.md has one rule for this surface: "Input validation on every state-changing gateway endpoint", satisfied by typed request DTOs, go-playground/validator and DisallowUnknownFields. That is a rule about the shape of the request.

The IEC 62443-4-1 SD-1 interface catalog in docs/sdl/secure-design.md lists the gateway's safeguards as TLS, OIDC/LDAP authentication, permission-gated routes, rate limits, typed-DTO validation, anti-replay nonces and audit attribution. Every entry answers who is calling and whether the request is well-formed. The catalog has no column for whether the action is safe against the target's current state, because that is a process-safety property and the catalog's vocabulary is a security vocabulary.

docs/compliance/isa88.md has the same gap from the other side. It maps implemented features to requirements, so it can record a capability the standard asks for. A prohibition the standard implies has no row shape, and a missing row is indistinguishable from one of the dozens of requirements not yet mapped.

The sweep¶

Fifty-six routes mutate an existing named target: thirty-one DELETE and twenty-five PUT. Fifty-four of them now carry a verdict of guarded or stateless. Two are gaps, recorded in scripts/.mutation-preconditions.tsv against two issues. The counts were nineteen and thirty-seven on the day the sweep closed.

Fixed here¶

DELETE /api/v1/sites/{site}/procedures/{name} was the root of the tree #1561 was protecting, and the one node the fix did not cover. It is worth stating what deleting it does, because it is worse than the case that started this.

The Procedure's children are owned by the Batch and not by it, so deleting it stops nothing. The UnitProcedures, Operations and Phases keep executing under the procedural operator, and the equipment keeps moving. What breaks is the batch's command path: forwardCommandToProcedure resolves status.procedureRef and answers NotFound, so Hold, Stop, Restart and Abort all fail. In abortBatchForUnitFailure the forward happens before the phase is set, so the batch does not even record that an abort was attempted.

The result is a plant running with the abort chart unreachable. The guard and TestDeleteProcedure_ActiveExecutionRefused land with this document.

Three recipe-edit guards that existed with no test asserting them now have one, in internal/gateway/recipe_edit_guard_test.go. requireEditablePhase refuses any recipe past Draft or Rejected. That refusal is also what makes the formula routes safe against live execution: a batch starts only on an Effective recipe, and Effective is refused.

Filed¶

Issue	Surface	The shape of it
#1566	Physical model	`DELETE /units/{name}` never `Get`s. The unit-runtime Pod is controller-owned by the Unit, so deleting a Unit a running batch holds garbage-collects the process driving the equipment. The batch detects it and holds, with no runtime left to run the holding chart. ControlModule, IOModule and Controller are the same family. Site, Area, ProcessCell and Enterprise are milder: they orphan the model while execution carries on, and they still need a written verdict.
#1567	Recipes	`reconcileRunning` `Get`s the MasterRecipe every pass and returns on NotFound, so deleting a Withdrawn recipe under a running batch silently switches off capability re-validation, allocation-drift detection and the ISA-88 §8.6 abort-on-unit-failure check while the tree keeps running. `DELETE /recipes/control/{name}` destroys the as-run recipe of a batch with no `Get` at all.
#1568	PUT	Twenty-two of twenty-five PUT routes consult nothing. The procedural controllers read `Spec.Chart` on every reconcile, so a PUT replaces the step list of a sequence that is mid-execution.
#1569	Alarms	An Active, Unacknowledged alarm deletes cleanly, which erases an annunciation nobody accepted.
#1570	Tests	The "every DELETE route" 404 table covers fifteen of thirty-one, and nothing holds it complete.

The process change¶

ADR 0060 carries the ruling. In short: every DELETE and PUT route declares in the tree what execution state it refuses, make lint-mutation-preconditions enforces that every route has a row and every row has a route, and the three verdicts are guarded (naming a test the gate checks exists), stateless (carrying a written reason) and gap (naming an issue the gate checks is open). There is no allowlist, because an allowlist is a fourth verdict meaning "do not ask".

The same pass holds the 404 table in the row above complete. Its rows carry the route pattern as registered, and the gate requires every DELETE it reads out of routes.go to appear there. The two contracts ask about the same enumeration. One asks what a route answers when the target is absent, the other what it answers when the target is running. A single route extraction therefore serves both, and there is no second parser. The sixteen routes the table never named are covered now. The one that failed the contract, DELETE /recipes/control/{name}, answers 404.

The gate is hermetic, runs as a step of the existing checks job, and has its own integration test in test/scripts/. That test plants each failure the gate is supposed to catch, which matters here for the reason #1481 taught: this tree's declaration file is complete today, so a gate that had quietly stopped firing would print exactly what a working one prints.

What this changes about how to read a green build¶

A green lint-mutation-preconditions does not mean the mutating surface is safe. Two of fifty-six routes are gaps, down from thirty-seven when this document was written. It means every route has been looked at, and the ones that are unsafe are named in a file anybody can read. The difference between that and the state of affairs that produced this incident is the only thing the gate is claiming.

What the sweep found next¶

The PUT sweep closed #1568 and left two new gaps behind it, which is what a sweep is for.

The first is worth naming for its shape. The control operator has refused a control-logic hot swap under an active batch for years, and that gate is the prior art this whole document credits. It keys on the ControlProgram's own generation, and a composite block type edit does not move it. The reconcile then takes the other branch, the unconditional pod-recovery redeploy, and swaps the running logic of every program that resolves the type. #1574 tracks it. The guard was not missing here. It was present, correct, and reachable by only one of the two roads to the hazard.

The second is the alarm definition. #1569 answered the alarm deletes on acknowledgement, which is the right axis for destroying an annunciation. Editing a definition turns on a different one: its condition is a live trip point, spec.enabled disarms the protection, and spec.exceptionAction is a standing instruction to Hold, Stop or Abort. Whether the answer is a refusal at all is an ISA-18.2 management-of-change question. That is too big to settle in passing, so it went to its own issue (#1576), which has since answered it. The route refuses while the equipment spec.source names is executing, and only spec.message may still move. The question that issue actually turned on was not whether to refuse. It was whether a guard could tell a tightened limit from a loosened one, and it cannot: the limits are strings, direction inverts between a high alarm and a low one, and a condition type change or an exception action removal has no direction to read. So the nuance moved into the refusal itself, which names the shelving that answers the mid-batch need.