ADR 0034: Infrastructure actions are operator-executed requests recorded on a NodeMaintenance object, and shutdown is offered only by a power channel that can also power on¶

Status: Accepted Date: 2026-08-04 Issue: #1225

Context¶

The Servers surface is read-only and carries exactly one action, servers:read at the read tier. Taking a chassis out of service and returning it is the management capability the surface exists to deliver, because for a customer this UI is the only management surface the hardware has and the plant engineer does not speak kubectl. Adding those verbs raises four questions that have to be answered together: which component may hold the privilege, what records the intent, what a power action even is, and what ceremony each verb carries.

The privilege already exists, and it is not where the proposal assumed¶

The epic proposed that the gateway perform cordon and uncordon directly on the grounds of low blast radius, with drain and the power verbs routed through an operator. Reading the two roles inverts the argument.

The gateway holds nodes: get, list (config/rbac/gateway_role.yaml). The physical operator already holds nodes: create, delete, get, list, patch, update, watch and policy/evictions: create (config/rbac/physical_operator_role.yaml). Routing an action through the operator therefore costs no new privilege anywhere. Performing cordon in the gateway costs a new write verb against a cluster-scoped object, in the component with the largest attack surface.

The product already cordons, drains and deletes nodes, with no ceremony at all¶

ControllerReconciler removal (internal/controller/physical/controller_controller.go) already does all three. It sets node.Spec.Unschedulable, calls drainNode, and deletes the Node object. Two properties of that path matter here:

drainNode fires one eviction per pod and returns without waiting for any pod to terminate. It swallows IsTooManyRequests, which is exactly the status the API server returns when a PodDisruptionBudget refuses the eviction.
The caller logs a drain failure and continues, with the comment "Continue anyway — best effort drain", and then deletes the Node.

So the capability that cluster 3 is about to wrap in ceremony is already reachable today through a delete confirmation, and it reports success while leaving PDB-protected workloads in place. Any answer here has to converge on that implementation rather than grow a second one beside it.

Kubernetes has no power verb, and in-band channels are asymmetric¶

Cordon, uncordon, drain and delete-node is the entire vocabulary the gateway can reach. Reboot and shutdown need a host-layer channel, which is the same wall #1168 hit when it omitted the VIP holder.

The decisive property is asymmetry, and it belongs to the channel rather than to any implementation. An in-cluster privileged agent can shut a machine down and can never power one back on, because the agent dies with the machine. The Talos API has the same shape, since a powered-off machine serves no API. Only an out-of-band channel such as a BMC over Redfish, a cloud provider API, or a switched PDU can do both. Shipping shutdown over a channel that cannot power on turns a remote action into a drive to the plant.

Quorum is already derived twice¶

internal/gateway/servers.go derives etcd quorum for the read surface, and deriveEtcdQuorum in internal/controller/alarm/server_alarm_controller.go derives it again for the ADR 0032 alarm rules. ADR 0032 recorded the duplication as an accepted cost. A quorum guard (#1238) would be the third copy of one stacked-etcd assumption.

Constraints in force¶

Action names are a product contract under ADR 0024, so a rename is a breaking configuration change and the names must be settled before the first release that carries them (#1211).
ADR 0006 owns re-binding a unit's runtime, with a fencing certification and a typed reason.
ADR 0033 forbids any CR that mirrors an object the cluster already reconciles.
ADR 0004 delegates node join to the deployment layer.
A BMC sits on the management network, which is a separate zone under the IEC 62443 posture claimed in docs/compliance/iec62443-4-2.md CR 5.1.

Decision¶

The gateway holds no write privilege against a Node. Every infrastructure action is a request the gateway authorizes, audits and records on a cluster-scoped NodeMaintenance object, which the physical operator executes. The power channel is a deployment-configured capability provider, and shutdown is offered only by a provider that also advertises power-on.

1. One path, through the operator¶

All four verbs (cordon, uncordon, drain, and the power verbs) route through the maintenance object. The gateway's RBAC is unchanged by this epic, which is a checkable outcome rather than a stated intention.

Cordon joins the same path as the rest for three reasons. The operator already holds nodes: patch, so the split buys no reduction in total privilege while adding a node write to the internet-facing component. A gateway compromise that can cordon every node in the cluster is a plant-wide denial of service, and "instantly reversible" describes the operator who did it deliberately rather than the one who has to discover it first. Cordon is also the first step of every maintenance procedure, so making step one structurally different from the steps after it is how a second implementation gets built.

The cost is honest and accepted: a cordon now takes a reconcile round trip and mints an object where a PATCH would have done. The object is required for the lifecycle regardless, and a maintenance abandoned after its cordon is precisely the state an operator needs to see and undo.

2. `NodeMaintenance` is a cluster-scoped CRD recording intent¶

It records an operator's intent and its lifecycle, and mirrors nothing, so ADR 0033's no-mirror rule permits it. It is cluster-scoped because a node is, and a maintenance on one chassis is one event even where the cluster hosts several sites.

One object per maintenance, and the intent declares how far it goes. spec.intent is one of cordon, drain, reboot, shutdown, and the lifecycle advances only as far as the intent requires. A cordon-only maintenance reaches Cordoned and waits there for return to service. This gives alarm shelving (#1236) one scope to hang on, the quorum guard one chokepoint, and drain (#1235) the lifecycle that has to survive a gateway restart.

States. Requested → Cordoned → Draining → Drained → Rebooting | PoweredOff → Returning → Complete, with Refused and Failed as terminal states and an abandon path from any non-terminal state back through Returning. Rebooting and PoweredOff are the two places a maintenance parks once its power verb has been sent, and a reboot needs its own because Drained is the phase that dispatches the verb. A reboot that stayed there would be sent again on every wake-up. The pre-flight snapshot taken at request time is recorded on the object, so what the operator was shown when they decided is recoverable afterwards.

The physical operator owns it. It already holds every verb involved, it owns Controller, Unit and node discovery, and it hosts the ADR 0032 server alarm reconciler that shelving has to reach.

The existing drain becomes the one implementation. The controller-removal path and the maintenance path converge on it, which means the PDB-blindness and the missing wait get fixed once for both. Filed as #1242.

3. Node deletion is not an infrastructure verb¶

Removing a Node object is decommissioning. ADR 0004 delegates node join to the deployment layer, so node departure belongs there too, and the product's own record of un-adoption is the removal of the Controller. Deletion stays where it is today and gains no button on the Servers surface.

4. The power channel advertises capabilities, and shutdown requires power-on¶

A capability provider is configured at the deployment layer. The UI renders only the verbs the configured provider advertises, and a deployment with no provider configured shows no power verbs at all. That is the correct default and it needs no feature flag.

The capability rule. A provider advertises reboot, shutdown and power-on independently. A provider that does not advertise power-on may not offer shutdown. Reboot over an in-band channel stays available, because the machine returns by itself and the asymmetry never bites.

Amendment (2026-08-05, #1288): the rule was attached to the wrong thing.

What this section protects against is a shutdown nothing can undo: the machine is off, nobody is there, and somebody has to drive to the plant. That is a property of whether the shutdown is recoverable, and a channel that can power on is only one of the two ways to be. The other is a person at the rack who has said they are there.

The rule is therefore: a shutdown is offered only where it is recoverable, by either route. A deployment declares the second through powerProvider.attendedRack, and ADR 0038 § 5 makes the same declaration per site for a whole-site outage.

Two things follow, and the second is why this amendment exists at all.

The Talos backend now advertises shutdown honestly, because it really can perform one: Talos stops its own services, drains the kubelet and unmounts cleanly, which is a graceful power-off rather than a cut. It still never advertises power-on. The structural guarantee that used to live in CapabilitySet — where a set carrying shutdown without power-on could not be constructed — moves to power.OfferFor and power.Invoke, the two places that know the deployment's recovery model. An interface shape could never have expressed attendance anyway, because it is a fact about a place rather than about a channel.

And the original rule made the product recommend the worse act. ADR 0038's attended form left the machines running so a human could remove power at the rack, which is a hard cut of a live operating system, chosen over the sequenced shutdown the product could already have asked Talos to perform. The asymmetry objection does not apply to that case, because the attended form's whole premise is that the person is already there.

This is what makes an in-band first backend safe. The Talos API is the sensible first implementation, because it is what the bench runs, it can be proven on our own metal, and it brings the VIP holder within reach through the same channel. It advertises reboot and withholds shutdown. Redfish over a BMC comes second and unlocks shutdown and power-on where the hardware has a BMC. Cloud provider APIs are the third case, where the repairable versus replaceable classification (#1233) says the useful verb is terminate.

An in-cluster privileged agent is rejected outright, because it is the one channel that offers no path to symmetry at any later date.

The provider runs in the operator, which already occupies the privileged position, so the management-network conduit originates in one component rather than in the gateway. A deployment with no provider opens no conduit at all, which keeps the default posture unchanged. Provider credentials arrive as a Secret reference supplied by the deployment layer. They are machine identities, so engineering states the scope and blast radius and files the issue in cndcs-operations, and that issue is part of #1237's definition of done.

5. Tier and ceremony¶

Infrastructure actions sit at PermAdmin, and configuring the power provider does too.

This follows a founder ruling of 2026-08-04 that states a general default: where an action needs elevated access, PermAdmin is the safe answer unless there is a specific reason to sit lower. The reasoning is that a tier is granted once and lived with for years, an over-granted tier is discovered only after it is abused, and an under-granted one is discovered immediately by an engineer who cannot do their job. The two errors are not symmetric, so the default belongs on the side that fails loudly.

Applied here, a chassis leaving service is exactly the class of act that default reaches for. The plant-floor argument for a lower tier is real and it is answered by delegation rather than by the default.

Delegation runs through ADR 0024 in the allow direction. A plant that wants its maintenance engineers to cordon a node without holding terminal access and bulk export adds servers:cordon and servers:uncordon to the allow list of whichever role those engineers hold. The per-action policy machinery works in both directions, and starting from admin means every such grant is a deliberate, reviewable line in the deployment's roles file. Starting from operate-lead would have made the same access the silent default in every deployment that never opened the file.

Action	Ceremony
`servers:uncordon`, return to service	Reason
`servers:cordon`	Reason
`servers:drain`	Reason, plus acknowledgement of the pre-flight impact
`servers:reboot`, `servers:shutdown`	Reason, pre-flight acknowledgement, typed node name
Any action consuming the last of the quorum margin	Hazard tier: warning title icon, typed reason, typed node name
Any action that would drop ready control-plane nodes below quorum	Refused
Drain, reboot or shutdown of a controller node hosting live control	Routed through the ADR 0006 failover path first, with its fencing certification

Every action carries a reason and every reason meets the deployment's reason policy, matching the #687 treatment the failover re-bind already gets.

A refusal is never upgraded into a ceremony. Below-quorum is refused outright, so clicking through a sufficiently frightening dialog never becomes the path to killing the cluster. A refusal explains itself with the member count, the quorum size and the margin.

Absent derivation refuses. A cluster whose control-plane nodes are not labelled derives no quorum, and ADR 0032 already records that it therefore raises no quorum alarm. A guard that passed because it derived nothing would be the same silence with worse consequences.

The quorum derivation collapses to one implementation. The guard does not become a third copy. The derivation moves to a shared package with three callers: the gateway read surface, the ADR 0032 reconciler, and the guard. That closes the divergence ADR 0032 accepted as a cost.

The guard is authoritative in the operator, which is the component that executes, so no client and no direct API call can bypass it. The gateway runs the same shared derivation at request time so the refusal is synchronous and the operator sees the numbers immediately. A request that clears the gateway and fails the authoritative check lands the object in Refused with the numbers in status.

Virtual nodes refuse every action. Nodes labelled dcs.io/virtual-node are the simulation fixtures the Controller reconciler creates, and ADR 0032 already excludes them from alarms. The refusal is enforced in the operator rather than by hiding buttons, because a hidden button is defeated by a direct API call and the docs capture rig runs on exactly these nodes.

6. The action names, settled¶

Seven names in the servers: domain, which already covers controller-bound chassis because servers:read does. Each is independently deniable under ADR 0024.

Action	Default tier
`servers:read` (exists)	`PermRead`
`servers:cordon`	`PermAdmin`
`servers:uncordon`	`PermAdmin`
`servers:drain`	`PermAdmin`
`servers:reboot`	`PermAdmin`
`servers:shutdown`	`PermAdmin`
`servers:maintenance-abandon`	`PermAdmin`

Cordon and uncordon are separate names so a deployment can let a wider set of roles restore service than can withdraw it.

Amendment (2026-08-05, #1281): the domain grew, and this table is no longer the whole contract. ADR 0038 settles a site-scoped outage, which is a different blast radius in a site: group of its own, and it adds an eighth name here: servers:power-on at PermAdmin. Seven names covered taking a machine out of service and none brought one back, so a shutdown that reached PoweredOff had no route back through the product (#1282). The § 4 capability rule governs it unchanged: a channel that cannot power on never advertises it, which is the same channel that may not offer shutdown.

7. Audit records the human act, status records the machine's¶

The gateway mints the AuditRecord at request time, because it holds the authenticated identity and already audits every tag write. It carries the actor, the reason, the target node, and the name of the NodeMaintenance object, so the trail and the object are joinable. The source is Server/<node>, the identity ADR 0032 already uses.

The operator records execution outcomes on the object's status. These are two records of two different things: what a human asked for, which is what 21 CFR Part 11 and IEC 62443 want in the trail, and what the machine subsequently did.

Amendment (2026-08-07, #1324): PoweredOff says what was requested, and a condition says what was observed.

A shutdown reaches PoweredOff the instant the channel accepts the request. That is the whole of what an in-band channel can ever support, because the machine stops serving the API it was asked through and its silence afterwards is indistinguishable from a network fault. An out-of-band channel has its own processor and its own power, so it keeps answering while the machine is dark and can report that the machine reached off. Those are two different claims and a plant record must not print the stronger one when it only holds the weaker.

The phase therefore never waits on a readback, and the PowerOffConfirmed condition carries the stronger claim where a deployment can make it. True is an observation, False is a readback still running, and Unknown is settled without one — either the channel cannot see, or it never reported the machine off before the window closed. status.poweredOffAt records when the channel accepted the shutdown and is what bounds that window.

Who polls, and how often, is settled here too. The reading is taken for the one machine this product just stopped, at a cadence of its own rather than the drain's, and only until that shutdown lands or the window closes. It is never an ambient property of the Servers list: a five-second UI poll multiplied across a rack is a load pattern the management network did not ask for, and no surface needs a general "off versus absent" distinction that a maintenance sitting at PoweredOff does not already give. Consuming the reading at the moment it changes what the product may claim is the whole of what the mechanism is for.

Two consequences. The power advertisement stopped serving its stateReadback flag, which had been on the wire since #1237 and read by nothing: the fact is consumed where the reading is taken, and an advertisement that promises a capability no surface acts on is read as a promise the product keeps. And a SiteOutage leg still completes on the phase rather than on the confirmation, because a worker finishing its stop does not gate the next leg of a sequence.

Alternatives Considered¶

The gateway performs cordon and uncordon directly, as the epic proposed, keeping the operator for drain and power. Rejected on the RBAC reading. The operator already holds nodes: patch, so the split reduces no privilege while adding a cluster-scoped write to the most exposed component, and it splits one maintenance procedure across two implementations that the quorum guard would then have to be written into twice.

Record the intent as an annotation on the Node, following the ADR 0006 precedent where failover is triggered by annotations on the Unit. Rejected on three counts. It is a write to the object ADR 0033 keeps the product out of, it has no status subresource to carry a lifecycle or a pre-flight snapshot, and ADR 0031 already recorded that node-level state is lost on re-enrolment.

An in-cluster privileged agent for power. Cheapest to build and needs no credentials outside the cluster. Rejected because it can never power a machine on, and unlike the Talos API it has no path to symmetry later.

Redfish as the first backend. It is the symmetric channel and the eventual answer for bare metal. Rejected as the first one, because no bench hardware has a BMC, so the first backend could not be proven on our own metal, and the bench is what the product's hardware claims rest on. The capability rule keeps the asymmetry trap closed without requiring Redfish to come first.

Amendment, 2026-08-07 (#1284). Redfish is now the second backend and it is built. The ordering argument above survived, and the argument that had been made for putting a switched PDU ahead of it did not: that argument was the same "prove it on metal we own" reasoning, applied to a device we do not own either. A PDU client would have had to guess a vendor, a protocol and an authentication scheme for hardware nobody has chosen. Redfish is a published specification, so it is implementable against the spec and exercisable against an emulator, and it needs no purchase to be correct. A PDU remains the right channel for hardware with no BMC, which is most small plants, and it is a separate backend rather than a replacement.

Put infrastructure actions at PermOperateLead, the tier unit failover already uses, on the reasoning that maintenance happens during a planned window and is performed by maintenance engineers who should not need terminal access and bulk export to cordon a chassis. This was the ADR's original position and the founder overruled it on 2026-08-04, setting the general default recorded in the Decision. The practical concern survives the ruling and is answered by an ADR 0024 allow list, which makes the same access a deliberate line in a deployment's roles file rather than the silent default everywhere.

Let a sufficiently severe ceremony authorize a below-quorum action. It respects the operator's judgment and covers the genuine emergency. Rejected because the failure it prevents is unrecoverable from the UI that offered it, and an emergency that truly requires it has kubectl.

One servers:power action covering reboot and shutdown. Fewer names to maintain. Rejected because the two verbs differ in exactly the property this ADR turns on, and a deployment must be able to allow reboot while denying shutdown.

Consequences¶

The gateway gains no write against a Node. Its nodes rule stays at get and list, and any future patch adding a node write to config/rbac/gateway_role.yaml is a signal that something has left this decision.

Correction, 2026-08-05. This bullet originally claimed the gateway's RBAC was unchanged by the whole of cluster 3 and called that a checkable outcome. It was checkable and it was false. The gateway role enumerates physical.dcs.io resources explicitly with no wildcard, so it could not create the NodeMaintenance object this ADR has it author, and every cordon would have been refused in a real deployment. #1234 added nodemaintenances at create, get, list, update. The decision is unaffected, because that is a write against the product's own object rather than against a Node, which is the privilege this ADR withholds. - The existing drainNode becomes load-bearing, and its two defects have to be fixed before the maintenance path uses it. It swallows PDB refusals and never waits for pods to leave, and the removal path deletes the node anyway. Filed as #1242. - The quorum derivation improves rather than degrading. Collapsing the gateway copy and the ADR 0032 copy into one shared package with three callers closes a divergence that ADR 0032 accepted. - #1236 gets its shelving scope, since the maintenance object carries the window, and shelving expires with the object instead of depending on a clean return path. - #1237 gets a first backend it can prove. Talos over the bench, advertising reboot and withholding shutdown, with Redfish adding the symmetric verbs.

Amendment, 2026-08-07. Talos no longer withholds shutdown, because #1288 moved the rule from the channel to the recovery. Redfish landed in #1284 and is the first channel that advertises power-on, so a shutdown over it is recoverable without anybody at the rack. servers:power-on still has no request path; the channel can perform it and nothing in the product asks yet. - A cordon is now a recorded event with a reason. Plants that treat cordoning as a throwaway act will find this heavier than kubectl, which is the intended trade in a regulated environment. - The compliance surface moves once. docs/compliance/iec62443-4-2.md CR 5.1 gains the management-network conduit the power provider opens, stated as originating in the operator and absent by default. The Part 11 trail gains infrastructure actions. - A deployment that wants maintenance engineers to cordon must write a policy line. That is the intended cost of the admin default: the access exists and it is visible in the roles file rather than implied by a tier. - Reversibility. The tier assignments and the ceremony ladder are cheap to change before release. The seven action names are a product contract the moment a release carries them, and the NodeMaintenance schema is a migration once a deployment holds objects.