ADR 0061: A rotation window is two broker accounts¶

Status: Accepted Date: 2026-08-15 Issue: #1579 Related: #221 (the window that never worked), #1552 (the containment), #1510 (broker identities are separate on purpose)

Context¶

The Helm chart carried a graceful MQTT password rotation from #221. An operator set mqtt.auth.users.<id>.rotationPassword, the broker was supposed to accept the old password and the new one at the same time, clients rolled at their leisure, and nothing was ever locked out.

It never worked on any Mosquitto 2.x. The mechanism was two lines in the passwd file for one username, and the builtin password backend keys on the username. It was drilled on 2026-08-14 against three releases.

Image	Broker starts with the window open	The password every running client holds
`eclipse-mosquitto:2.0.18`	up	rejected
`eclipse-mosquitto:2.0.21`	up	rejected
`eclipse-mosquitto:2` → 2.1.2	exits	—

On 2.0.x the last entry wins, so opening the window swapped the accepted credential and did not widen it. On 2.1.x the broker reports Duplicate user 'dcs-gateway' in password file and terminates. From the reloader sidecar that means SIGHUPing a healthy broker into a crash loop. Opening the window was worse than not rotating at all.

Two things let that ship for four months. The chart test asserted the shape of a rendered manifest, and a manifest cannot be asked what a broker does with it. And mqtt.image.tag was the floating "2", so the failure mode moved from "wrong credential accepted" to "broker terminates" with no commit on our side and no operator opting in.

#1552 contained it. The chart refuses the value, the passwd script refuses to emit the entry, and the runbook stopped offering the graceful path. That left MQTT password rotation with only the non-graceful path, which costs roughly two minutes of client disconnect and puts any Running batch on Hold through the runtime watchdog. This record is about the window that replaces it.

Decision¶

An MQTT client identity is served by up to two broker accounts, its credential slots. A rotation moves the identity from one slot to the other and retires the one it left.

The builtin backend gives one password per username, so a window needs two usernames. Slot a is the identity's base username, slot b is that name with a -b suffix, and both accounts carry the same ACL rules.

identity "gateway"  →  account dcs-gateway     (slot a)
                       account dcs-gateway-b   (slot b)

Four rules follow from that.

Both slots carry the same rules, generated from one list. The ACL blocks are emitted per account from the identity's single rule list in cloud-native-dcs.mqtt.identities. Rules written once and emitted twice cannot drift. Rules written twice can. Parity is then asserted behaviourally as well: pkg/mqtt connects as both slots and compares the SUBACK and PUBACK reason codes the broker returns for the same probe topics.

Slot a is spelled without a suffix. This is the one asymmetry in the design, and it is deliberate. Every install that predates slots already has an account named dcs-gateway. Renaming it would make the upgrade that introduces graceful rotation the least graceful upgrade the chart has ever shipped. Slot a is therefore the account that is already there, and a release with no rotation in flight renders exactly the Secret it always rendered.

Slot b exists only while a rotation runs through it. It is not generated, not carried at rest, and not created on install. A standby credential nobody asked for is an account nobody watches. An empty passwordB with nothing stored means no second account, no second passwd line and no second ACL block.

Retiring the old slot is a step, and it is the step that makes a rotation a rotation. retireSlot drops an account. Leave it unset and the old credential keeps working, so the flip has added an account and rotated nothing. The chart refuses a retireSlot that names the active slot, because that is the account the clients are authenticating as.

Rotation is three upgrades¶

Open. Set passwordB. The second account appears, and the reloader sidecar rehashes and SIGHUPs the broker in place. Nothing restarts, and no client changes credential.
Flip. Move activeSlot. Only that identity's pods roll, onto the new account. Both accounts are served throughout, so the roll can take as long as it takes and a pod that has not moved yet is not locked out.
Retire. Set retireSlot to the slot just left. The old account stops being served.

The per-pod restart annotation covers the identity's active credential alone, and no longer the whole Secret. That is what makes step 1 restart nothing and step 2 restart only what it must. It also means rotating the historian's password no longer restarts the gateway.

Steps 1 and 2 are separate upgrades for a reason that is not stylistic. The raw credentials reach the broker through a mounted Secret, so kubelet's sync period and then the reloader's poll sit between the upgrade and the account existing. A flip that overtakes them rolls the clients onto an account the broker has not loaded yet. The operator's gate is reading the account out of the broker's own passwd file, which the runbook makes step 1's last instruction.

Retiring needs `data`, and `stringData` cannot do it¶

The auth Secret renders data with base64 values. Every other Secret in this chart uses the readable stringData, and the difference is not a style preference. stringData is write-only: the API server merges it into data and clears the field, so the live object never carries it. A key the chart stops rendering is therefore a key Helm's three-way merge has nothing to diff against, and the credential stays in data untouched.

Under stringData, retireSlot produced a flawless manifest and retired nothing. Every render test agreed the account was gone, because in the manifest it was. Only a rotation run against a live release showed the old password still authenticating three upgrades later, which is the same shape as the bug this whole record is about, one layer down.

What the chart cannot enforce¶

A slot left standing keeps the password it was serving. Flipping back onto it later therefore re-adopts a credential that was already in use, which is an addition wearing a rotation's clothes. Render-time values say nothing about which slot was active before, so this cannot be a fail. A guard has to be expressible in what the guard can read. It is a hazard the runbook carries, and it is the reason step 3 is not optional.

The broker image is pinned to a release¶

mqtt.image.tag names 2.1.2-alpine, and no longer the series tag "2". It is the same image "2" resolves to today, digest for digest. The pin therefore changes nothing about what a running deployment gets. What it stops is the next Mosquitto release arriving on its own and changing what the chart's own files mean. The interop suite reads the pin out of values.yaml and starts that image. It does so for the tests that have nothing to do with rotation too, because proving this product's client against a broker the product does not ship proves nothing about the product.

Where the proof lives¶

The proof is in make test-interop, and a render test could never carry it. pkg/mqtt/rotation_passwd_integration_test.go renders the real chart, writes the rendered Secret's keys as the raw auth mount, and runs the chart's own passwd script over them. That script is lifted verbatim out of _helpers.tpl, because a Go copy would agree with the chart by construction. The pinned broker then starts against what the script produced, and everything asserted from there is what the broker does.

Two of those tests exist to keep the others falsifiable. One requires each slot to refuse the other's password, so "both accounts accept" cannot be satisfied by two accounts that accept everything. One requires some probe to come back denied for some identity, so "the slots agree" cannot be satisfied by a broker whose ACL never loaded.

Alternatives Considered¶

Keep the single account and live with the outage. This is what #1552 left in place, and it is a real option: roughly two minutes of client disconnect in a planned quiet period. It was rejected because the cost is not the two minutes. Any batch in Running goes to Hold on the runtime watchdog and an operator has to Resume it, so the credential can only be rotated when the plant is idle. A control system whose passwords can only change when nothing is running has passwords that do not change.

A dynamic security backend. Mosquitto ships mosquitto_dynsec, which holds clients and roles in a JSON store and takes changes over a control topic. It would give multiple passwords per client and an API to rotate them. It was rejected because it replaces a file the chart writes and a sidecar reloads with a stateful store and an authenticated control channel. That is a much larger blast radius for the problem at hand, and the store then becomes a thing to back up, restore and reconcile against the Helm values. Slots reach the same window with a passwd file and one more line in it.

Client certificates in place of passwords. mTLS is already in the chart for the runtime and gateway channels, and cert-manager rotates certificates without anyone opening a window. This is the better long-term shape and is not blocked by this record. The MQTT credential is also the one an external integrator is handed, and passwords have to work for as long as that is true.

A -rotation key that becomes a distinct account. Keeping #221's values and simply hashing dcs-gateway-rotation as its own username would have been the smallest diff. It was rejected because it names the mechanism after the moment and not after the thing. There is no activeSlot to move, so a client can never be told to use the new account. The window then has no way to close. The rotation password would be a permanent second credential under a name that says it is temporary. The chart refuses the key instead. The passwd script refuses to hash it, so a -rotation key in an operator's own Secret creates no account.

Consequences¶

MQTT password rotation has a zero-lockout path again, and this time a broker makes the claim where a manifest used to.
The account count doubles for the duration of a rotation and only then. A release at rest has the five accounts it always had.
Revoking an identity now means revoking both of its slots. The identities themselves stay separate for the reason #1510 recorded: revoking the OMF egress must not take the historian's feed with it. A slot is a credential for one identity, and it is no way to merge two.
mqtt.auth.users.*.rotationPassword is refused at render time. The refusal names the replacement, because an operator who set it was trying to do something the product now supports.
An operator's own mqtt.auth.existingSecret is theirs to key by account name. The chart reads activeSlot to decide which key each client mounts and cannot check that the key or its ACL rules are there.
Unit runtimes are bare Pods written by the physical operator, so their slot travels as MQTT_AUTH_USERNAME on that operator and applies to pods created after it. Running pods are left alone on purpose. Recreating a runtime pod puts the batch it is executing on Hold. Which moment to take that on is an operator's decision, and a reconcile cannot make it.