Skip to content

ADR 0081: A refusal is not a lost connection

Status: Accepted Date: 2026-08-23 Issue: #1780 Related: ADR 0008 (the hold-lifecycle trail this queue carries), ADR 0077 (the other persisted file in the same directory)

Context

The unit runtime's store-and-forward queue exists to survive a control plane it cannot reach. A message that will not publish is written to {DataDir}/queue/queue.jsonl and replayed in order when the broker comes back. ADR 0008's hold-lifecycle events ride it, and pkg/mqtt/topics.go publishes the guarantee in its own doc comment: the control plane materializes the replayed events into AuditRecords for 21 CFR Part 11.

replayQueue read every publish failure as the same fact. It logged, it broke out of the batch, and because nothing had been replayed the offset did not move. The comment beside the return stated the assumption plainly. A partial batch means the connection dropped. The only thing that could call the function again was OnConnectionChange(true).

On the bench, during a #942 partition drill, the broker refused one topic. Its ACL did not grant .../controlmodule/aux-output/health to the dcs-runtime identity. That refusal is a fact about the broker's configuration, and it holds until an operator changes it. The message sat at the head of the queue. Behind it were 8404 others, including the four hold-lifecycle events from the partition the drill existed to record. The runtime published live telemetry at 1481 messages every 15 seconds throughout. dcs_runtime_mqtt_queue_replayed_total did not exist on that runtime, because it had never replayed anything at all. The queue stayed at 8404 for the life of the pod.

The audit trail therefore says the hold was armed and later released, both of which travelled the live path, and says nothing about it having been triggered, having driven two outputs to zero, or having settled. It is missing precisely the interval it exists to cover.

Two separate mistakes produced that.

The first is the classification. A broker that refuses a message has answered it. A PUBACK cannot arrive over a link that is down, so a refusal carries two facts at once: this message was not accepted, and the connection is up. Reading it as a lost connection inverts the second one. paho reports the condition as a formatted sentence, error publishing: The PUBLISH is not authorized., and it discards the reason code along with the response value nothing was reading. The only machine-readable form of the broker's answer never reached the caller.

The second is the wake-up. Even a genuinely transient failure left the queue waiting on a reconnect, and a healthy link does not produce one. Any stall was terminal for the life of the pod, whatever caused it.

Decision

A refusal is a verdict about one message, and the queue advances past it.

Client.PublishRaw returns a *PublishRefusedError when the broker answers with a reason code at or above 0x80. The type carries the code itself, and Permanent() says whether that code is a verdict about the message rather than about the broker's capacity or the session.

replayQueue branches on it. A permanent refusal is written to {DataDir}/queue/deadletter.jsonl with its payload, its reason and the time it was given up on, counted on dcs_runtime_mqtt_queue_dead_lettered_total, and advanced over. A refusal that may be answered differently later is left in place. Anything else stops the pass at the entry that failed, which is the old behaviour and is right for a link that may really have gone.

The replay worker runs on a 30-second timer as well as on every reconnect.

Five things this rests on.

  • The batch boundary is not the unit of failure. The old loop advanced by the length of an unbroken prefix of successes, which is why a refusal at position one cost the other forty-nine. Entries the broker took and entries it refused are both consumed. The two are counted separately, because a dead-lettered message is not a delivered one.
  • The queue is one FIFO with one offset, so there is no skipping in place. A message the broker will never take either comes off the front or holds everything behind it. The dead-letter file is what makes the first of those something other than a silent discard.
  • An unrecognised reason code is permanent. The failure being prevented is a queue held forever behind a message that will never go, so an answer the product does not recognise resolves towards letting the queue drain. The two codes treated as retryable are named individually: 0x91, a packet identifier already in use, which is session state, and 0x97, an administrative limit, which is a come-back-later.
  • A permanent refusal is not queued on the live path either. The broker is reachable and has already answered, so enqueuing would only move the same refusal onto the replay path at the cost of the entries behind it. It is dead-lettered where it happens and counted refused, an outcome distinct from dropped because the two say different things about where to look.
  • The announcement is once per topic. A refused topic is refused on every publish to it, and the bench's was published on a timer. The log names the topic and the reason code the first time. The counter carries the rate.

A held queue and an idle one must not read the same. dcs_runtime_mqtt_queue_blocked is 1 while the head of the queue could not be published on the most recent attempt. Depth and the replay counter cannot separate those two states, because both are flat in both. That is how a runtime with 8404 undelivered messages looked healthy for half an hour.

Alternatives Considered

Fix the ACL and nothing else. The unauthorized topic is the trigger and it is a one-line chart change (#1781, fixed separately). It leaves the class open. Any single un-publishable message voids the guarantee the same way: an oversized payload, an invalid topic name, a retained-flag refusal, a topic granted today and revoked tomorrow.

Match on the error string. strings.Contains(err.Error(), "not authorized") would have worked on the bench and is not a thing to branch on. The wording belongs to a dependency, it differs per packet type, and it says nothing about the six other codes. The reason code was already on the wire and already in the response value. It only had to stop being thrown away.

Drop a refused message without recording it. Rejected. The queue's whole purpose on these topics is a Part 11 audit trail, and advancing past a message without keeping it would mean the runtime silently discarded a record it was holding. The file is capped at the configured queue size, because the first refusals identify the problem and the thousandth repeat of one rejected topic says nothing the first did not.

Retry a permanent refusal a bounded number of times before giving up. Rejected as buying nothing at the cost of the thing being fixed. The refusal is identical on every attempt by construction, so the retries are pure head-of-line delay for the entries behind it. The codes that genuinely can change their answer are the ones that are not dead-lettered at all.

Consequences

One refused message costs one message. The queue behind it drains, which is the difference between an audit trail with a hole in it and an audit trail with a named gap and a counter beside it.

A non-zero dcs_runtime_mqtt_queue_dead_lettered_total is telemetry the plant will not get back, and reason says why. not_authorized is a broker ACL that does not grant a topic to the runtime's identity, and it is fixed in the chart, not on the node.

make lint-runtime-state-files reads os.OpenFile by its flags now. It classified calls by name, and neither readFuncs nor writeFuncs held OpenFile. Both this dead-letter file and the queue file are append-only logs. Each registered as read by the runtime and written by nothing. The queue file passed only because its rotation happens to write through os.Create and os.Rename. A gate about persisted state that cannot see the most common way of appending to a file was one commit away from a false failure, and would have been a false pass for any file without a rotation.

pkg/mqtt/topics.go, architecture.md § Telemetry Store-and-Forward and ha-failure-modes.md each stated the guarantee without its one exception. They state the exception now.

The classification is held against a real broker. TestIntegration_UnauthorizedPublishCarriesTheBrokerReasonCode runs Mosquitto with an ACL that grants one topic and denies another, and asserts that the denial arrives as 0x87, that Permanent() says so, and that the connection is still up and still publishing afterwards. No unit test can make that claim, and every branch above it is worthless if the code does not arrive.

Amendment: an ACL refusal is a verdict the next upgrade can reverse (#1814)

Status: Accepted Date: 2026-08-24 Issue: #1814

The bench moved from chart 0.5.1 to 0.6.0 on 2026-08-24. The unit runtime came up on the new build at 22:29:47Z with 8,781 messages on its disk, and it drained them at once. The broker pod came up at 22:34:07Z, four minutes and twenty seconds later. Only then was the new ACL in force.

283 of those messages were control-module health. The grant that authorises them is topic write dcs/+/equipment/controlmodule/+/health, and it ships first in the very release that was rolling out while they were discarded (#1781). The rendered ACL on the rig carries the grant today.

The decision above says an unrecognised reason code is permanent. The alternatives above reject retrying a permanent refusal, on the grounds that "the refusal is identical on every attempt by construction". That holds for every code the product names but one. 0x90 reads the topic name and 0x99 reads the payload, and a packet does not change between attempts. 0x87 reads the broker's ACL. This ADR's own Consequences say what to do about one: it "is fixed in the chart, not on the node". The product's remedy for a wrongly refused message is therefore an upgrade, and an upgrade is the one moment the verdict is unsafe to trust.

Decision

A not-authorized refusal on the replay path is held for a bounded second chance. It is given up on only if the broker refuses it again after the window has run out.

The entry is written to {DataDir}/queue/deferred.jsonl with the instant of its first refusal, and it is advanced over. The rule the original decision rests on is untouched: the queue is one FIFO with one offset, and nothing behind a refused message waits on it. deferred.jsonl is the second place to put it. The store is offered again on every replay wake-up, paced to one sweep every two minutes, and a reconnect skips the pacing. It is capped at the configured queue size like the other two files. dcs_runtime_mqtt_queue_deferred says how many are waiting, _deferred_total how many have been held, and _deferred_recovered_total how many the broker took when it was asked again.

Four things this rests on.

  • A reconnect cannot be the trigger, because the chart's own ACL delivery produces none. The passwd-reloader sidecar rewrites the hashed passwd file and copies the ACL, then SIGHUPs a running mosquitto. That is what makes the ADR 0061 rotation window a window, and it is why the broker pod deliberately carries no auth checksum (#1811). #1781's own commit message says it: "a helm upgrade is enough". The bench restart that would have supplied a connection edge came from the release's other broker inputs. Time is the only observable left, so the bound is a window measured from the first refusal.
  • The window bounds the retrying. It is no deadline on the message. A record is dead-lettered when it is offered and refused again after the window has passed. A runtime that cannot reach a broker offers nothing, so it discards nothing. A restart does not restart the clock either, because the first-refusal instant is in the file.
  • Only the replay path defers. A live publish that is refused is a message the runtime is producing now and will produce again. Its topics include the 1 Hz value stream, so deferring there would feed the store at the rate the plant scans. The queue is the other case in both respects. What is in it is a backlog nothing will re-derive, and since #1779 that includes the ADR 0008 hold-lifecycle trail. It is also finite, because a drain empties it. That asymmetry is what bounds the retry cost: the store can only be filled by a backlog, and a backlog only exists after an outage.
  • The deferred store is offered before the queue. A deferred entry came off the head, so it is older than everything still queued. Several of these topics are retained. Publishing the backlog first and the entry that preceded it second would leave the broker holding the older payload as the retained value of that topic.

Alternatives Considered

Roll the broker before the runtimes. Named in the issue. It narrows the window and does not close it. A runtime restarts for its own reasons at any point in an upgrade, and an ACL change that arrives through the sidecar rolls nothing at all. It also buys nothing for an operator repairing an ACL on a cluster that is not being upgraded.

Accept the loss and make the dead-letter file legible. Also named. The file is already written and already counted, and nothing in the product reads it. So "legible" means a new read path for records the product has decided to throw away. Recovering them is strictly better and costs the same file. The dead-letter file keeps its role for what the window runs out on.

Re-queue the refused entry at the tail of the queue itself. Rejected. It needs no second file, and it has two shapes. One re-offers the entry immediately, which is the same refusal at the cost of a rewrite. The other stops the pass when the drain reaches it, which is the head-of-line stall this ADR exists to remove.

Retry indefinitely. Rejected. An ACL that is simply wrong then never reaches a verdict. dcs_runtime_mqtt_queue_dead_lettered_total never moves, and the signal that says "your broker is refusing this runtime" is the thing that got #1781 found in the first place.

Consequences

dcs_runtime_mqtt_queue_dead_lettered_total{reason="not_authorized"} now means something narrower and more useful. The broker refused this message, and was still refusing it fifteen minutes later. Alerting on it is unchanged, and it no longer fires on the window an upgrade opens.

The premise the whole amendment rests on is that a widened ACL is enforced on a connection that was already open when it arrived. That is a claim about mosquitto, and no unit test can make it. TestIntegration_AnACLChangeReachesTheRunningBroker holds one session across the Secret change now. It asserts that publishing on that session goes from 0x87 to 0x10 with no reconnect, beside its existing assertion that mosquitto's pid did not move. The fresh-session probe it already had would have passed even if the ACL were consulted only at connect, which is the one case that would make the deferral useless.