Skip to content

ADR 0051: A batch ID identifies exactly one batch in a site, enforced at admission, and a repeat work order is answered rather than duplicated

Status: Accepted Date: 2026-08-11 Issue: #1460

Context

Batch.spec.batchID has always described itself as "a unique identifier for this batch execution", and the whole tree reads it as one. The MES API resolves a work order to a batch by it. GET /production-performance/{id} finds the production record by it. The BatchRecord controller correlates a batch's operator actions, its change history and its controller-failover evidence out of the site's audit trail by matching AuditRecord correlationID against it. The recipe instantiator stamps it as the batch.dcs.io/batch-id label on every resource a batch owns, and the procedural controllers and the gateway's prompt routes read that label back to find out which batch a phase belongs to.

Nothing enforced it. Every create path — the MES work-order endpoint, the gateway's own batch create, the CLI, plain kubectl — accepted a second Batch under an ID another Batch already held.

The way that happens in practice is not exotic. It is a client retrying a write it never got an answer to, which is the most ordinary thing an MES does. The anti-replay nonce does not cover the case and cannot: a replayed identical request is refused with 409 duplicate request nonce, but a correct retry carries a fresh nonce, and docs/mes-integration.md tells integrators so explicitly. The documented retry procedure was the thing that forked the work order.

Nothing surfaced the fork either. Both submissions answered 201, the batch list showed two rows that looked alike, and the site then held two Batches for one work order:

batch-n8qxx   WO-2026-0042   Pending
batch-vxh92   WO-2026-0042   Complete

Every read resolved by taking the first spec.batchID match out of a List. List order is the API server's, so which batch answered was arbitrary with respect to which one the client meant, and it could differ between two calls. In the case that was found, the new empty batch answered, and the completed run's production record — which still existed — was no longer reachable through the only handle the MES has.

The cost does not stop at a confused lookup. Because the BatchRecord controller correlates by batchID alone, a forked pair gives each twin's 21 CFR Part 11 production record the other twin's operator actions, change history and failover events. A record that is signed and retained as evidence of one execution can carry entries from a different one.

Decision

spec.batchID identifies exactly one Batch in a site. A Batch whose batchID is already held by another Batch in the same namespace is refused, and so is an update that renames a Batch onto an ID another one holds.

The check lives at admission, in the Batch validating webhook the batch operator serves, which now admits CREATE as well as UPDATE. Placement follows the same reasoning as ADR 0050's write barrier: an invariant every reader depends on has to hold for every writer, and the gateway is only one of four ways a Batch reaches etcd. The scan reads through the uncached APIReader, because an informer that has not caught up would report a just-taken ID free and admit the very duplicate the check exists to refuse. A list failure denies rather than admits, matching the webhook's failurePolicy: Fail.

The identifier is scoped to the site, not the cluster. Every reader that resolves one lists within a namespace, so that is the scope the invariant is written at.

A repeat work order is answered with the order that already exists. POST /work-orders?dispatch=immediate looks the work order up before creating anything. A submission whose body matches the existing order returns that order's DTO with 200 OK instead of 201 Created. This is what a retry after a lost response actually wants, and it means the documented retry procedure now converges instead of forking.

A repeat whose body differs is refused with 409, naming the field that differs. Such a submission is not a retry; it is a second order under a reused ID. Choosing a winner between two different bodies is how a plant runs the wrong recipe, and neither choice is defensible: honouring the first silently ignores what the client just asked for, and honouring the second silently mutates an order that may already be running.

The comparison overlays the request onto a copy of the existing batch's own spec and diffs the result, rather than listing fields to compare. A list makes "not compared" the default for every field added after it was written, and a field the create path writes but the comparison forgot is a changed order admitted as a retry. Overlaying inverts that default: exactly what the create path writes is what gets compared.

Reads resolve deterministically, oldest first. Admission stops new forks, but a cluster that ran an earlier gateway can already hold one, and those batches are production records that are not ours to delete. Every MES resolver now picks the oldest batch carrying the ID, breaking a same-second tie on name. The oldest is the batch earlier reads were answered from, and on a retry-caused fork it is the one that actually ran — so the finished record stays reachable rather than being shadowed by the empty twin.

Alternatives Considered

Refuse the duplicate at the MES endpoint and nothing more. Rejected as the whole answer, though it is half of the one adopted. A 409 on a retry after a lost response tells a correct client that its own correct behaviour failed, and leaves it to discover by a follow-up GET that its first submission actually landed. Refusal is right for a different body and wrong for an identical one.

Make the POST idempotent and leave writes otherwise alone. Rejected as the whole answer for the opposite reason: it fixes the door the defect was found at and leaves the invariant unenforced for kubectl, the CLI and the gateway's own batch create — and it is the invariant, not the endpoint, that the audit correlation and the batch.dcs.io/batch-id label depend on.

Leave writes alone and make reads deterministic. Rejected. It picks a winner and still leaves two batches for one order, two production records splitting one run's audit trail, and one of them permanently unreachable. It is retained as a component of the decision, for the forks that already exist, not as a substitute for enforcement.

Enforce with a uniqueness index instead of a scan. Not available. etcd enforces uniqueness on an object's name, not on an arbitrary spec field, and a custom field index is cache-only — which is exactly the staleness the scan avoids. The scan is therefore a read-then-write check and not atomic: two creates admitted concurrently can both see the ID free. That window is the pre-existing behaviour rather than a new failure, batch creation is an operator-paced action, and the check closes the retry-driven duplicate the field was actually forked by.

Make batchID the object's name. Rejected. Batch names are generateName-allocated, a batch ID is customer-chosen text that need not be a valid DNS subdomain, and the change would break every existing reference to a batch by name.

Consequences

  • POST /work-orders?dispatch=immediate answers 200 on an identical repeat and 409 on a differing one. A client that treated any non-201 as a failure will now see a success code it did not see before; the published integrator guide documents both.
  • Creating a Batch with an in-use batchID through any path fails at admission. The gateway maps that denial to 409 with the webhook's own message, which names the batch already holding the ID, instead of the 500 a raw create error would have produced.
  • The Batch webhook now sits on the create path for every Batch in the cluster. Its failurePolicy was already Fail, so a webhook outage already blocked Batch updates; it now blocks Batch creation too.
  • Sites that already hold a forked pair keep both batches. Reads become consistent and point at the older one, and the newer twin remains visible in GET /work-orders and in the batch list, where an engineer can see and dispose of it.
  • The audit cross-contamination is closed going forward. A BatchRecord finalized before this change, in a site that held a fork, may already carry the other twin's operator actions; such a record is immutable by design and is corrected by review rather than by rewriting it.
  • ADR 0050 — the same placement argument for a barrier: enforce at the one point every writer reaches, not at one spelling of the write.
  • ADR 0024 — the access barrier that settled on the same reasoning.
  • MES integration guide — the integrator-facing contract this decision changes.