Skip to content

ADR 0038: A site outage is one sequenced act on its own object, quorum is broken deliberately at the end or not at all, and shutdown without a power-on channel is available only in an attended form that says so

Status: Accepted Date: 2026-08-05 Issue: #1281

Context

A planned outage is an ordinary plant activity. Turnarounds, holiday shutdowns and seasonal idling all end with a site going quiet, and the product cannot perform one. Every outage today is performed by a human at a breaker or by an engineer at a terminal, and the plant loses its record of what it was doing.

ADR 0034 settled the infrastructure action model for one node at a time. A whole-site outage breaks three of its assumptions at once, and each break needs a decision recorded before code freezes it.

The bench surfaced it and the UPS justifies it

The founder's framing is a nightly ritual: "currently every night, to shut down the rack, I prompt you to shut it down." That is a convenience, and it is not the justification.

The justification is mains failure. A UPS gives a customer some minutes, and at the end of those minutes the rack dies ungracefully with outputs frozen, history unflushed and etcd holding whatever it held. Every serious industrial computing stack answers that with an orderly automated shutdown. With a sequence, the answer to "what happens when the power fails" becomes "the plant safed itself, the batch is on Hold with a reason, and the history is intact."

That matters here because the UPS is not a second feature bolted onto a button. It is a second caller of the same sequence, and it is a caller that cannot be argued with. That single fact decides more of this ADR than anything else.

Three assumptions ADR 0034 made that a site breaks

Its domain is a node. All seven names in ADR 0034 § 6 are scoped to one chassis. Under ADR 0024 an action name is a product contract and renaming one is a breaking configuration change for every deployment's roles file, so a site-scoped verb needs its name settled before the first route exists.

Its capability rule forbids what a site outage needs. ADR 0034 § 4 rules that a provider which cannot power on may not offer shutdown, and the reasoning is sound: shipping shutdown over an asymmetric channel turns a remote action into a drive to the plant. A site outage on hardware with no power-on channel is exactly that trap, and it is also exactly what a person standing at the rack wants to do.

Its quorum guard refuses the last step. #1238 refuses any action that would drop ready control-plane nodes below quorum, and ADR 0034 § 5 says a refusal is never upgraded into a ceremony. A site outage intends to break quorum. It cannot route through that guard, and it must not become the loophole that guts it.

Two properties of the sequence that are not details

The surface disappears partway through its own sequence. The gateway reporting progress runs on the machines being stopped, so the audit record of the human act and the terminal status both have to land before the apiserver does.

The last leg is not cluster-mediated. Nothing inside the cluster can tell the final control-plane node to power off, because by then there is no cluster. Whatever performs that act is outside the product's reach.

What is already built

The epic's other three issues have landed or are scoped, and this ADR decides on top of them rather than assuming them:

  • The four stranded node verbs have a request path (#1282), so the sequence can drive per-node work through NodeMaintenance rather than inventing a second path.
  • A terminal stop safes the field (#1283), and records that it did on a store-and-forward queue that survives the plane going down. That record is what lets an outage's trail assert the field was safed rather than imply it.
  • An out-of-band symmetric power backend is scoped (#1284) and gated on hardware that has not been bought.

Decision

1. The action names, settled

Four names in a new site: outage group, and one addition to ADR 0034's servers: domain.

Action Default tier What it does
site:outage PermAdmin Takes a site out of production, in order
site:outage-abandon PermAdmin Stops a sequence in flight and returns the site to service
site:outage-continue PermAdmin Resumes a sequence parked at Quiesced from the step that stopped
site:outage-read PermRead Reads the sequence and where it has got to
servers:power-on PermAdmin Powers a machine that is off back on

site:outage-continue was added in #1294. § 3 below has always made continue one of the two things reachable from Quiesced; the first implementation shipped only site:outage-abandon, which left Quiesced a phase with exactly one exit. It has its own name rather than sharing site:outage because resuming an act and requesting one are different decisions a plant may want to grant to different roles, and it sits at PermAdmin because continuing is what stops the rest of the site.

The verb is outage, not shutdown and not planned-outage.

site:shutdown was rejected on two counts. It collides with servers:shutdown, which is a different blast radius under a name a reader would have to disambiguate by domain alone. Worse, it asserts a power act, and the product's responsibility ends before power is cut (§ 4). A name that claims more than the verb performs is a name that will be quoted back at us.

site:planned-outage was rejected because of the UPS. The same sequence has two callers and one of them is a machine reacting to mains failure, which is not planned by anybody. An action name that is accurate for one caller and false for the other is worse than a shorter one that is true for both.

Starting and returning are separate names, for the reason ADR 0034 gave when it split cordon from uncordon: a plant may want a wider set of roles able to restore service than to withdraw it. abandon is deliberately the same word servers:maintenance-abandon uses, and it means the same thing — stop a procedure in flight and put things back — so a reader who has met one has met both.

Reading is separable from acting. site:read already exists and reads the Site CR; an outage sequence is a different object with its own lifecycle, so it takes its own name. The deployment need is concrete: a plant will want its operators to watch an outage they cannot start.

servers:power-on is ADR 0034's eighth name. ADR 0034 § 6 froze seven and none of them brings a machine back, which #1282 surfaced: a shutdown that reaches PoweredOff today has no route back through the product. The name is settled here because the return half of a site outage needs it and because minting one inside an implementation issue is how a product contract gets added by accident. It is built where a symmetric backend exists (#1284); a deployment with an in-band-only channel never advertises it, exactly as ADR 0034 § 4 requires.

2. The sequence is one object, and the last leg is not a NodeMaintenance

The sequence is carried on its own cluster-scoped object, for the reason ADR 0034 gave NodeMaintenance: the lifecycle has to survive a gateway restart, and the record of what was asked for outlives the asking.

Per-node work inside the sequence goes through NodeMaintenance objects, so the drain semantics, the PDB awareness, the ADR 0006 failover routing and the ADR 0032 alarm shelving are the ones already built rather than a second implementation.

The control-plane leg is the exception, and it has to be. #1238's guard is a property of the NodeMaintenance reconciler and it is absolute. A sequence that expressed its last step as NodeMaintenance objects would be refused by its own machinery, which is the correct outcome for that machinery and the wrong outcome for this act. So the sequence owns the control-plane leg itself and does not route it through the node path.

This is stated as a decision rather than discovered as an implementation detail because the alternative — relaxing the guard so the sequence can use the node path — is the loophole this ADR exists to refuse.

3. Quorum is broken deliberately at the end, or not at all

The node-scoped guard stays absolute. No action in ADR 0034's servers: domain may drop ready control-plane nodes below quorum, under any ceremony, for any reason. This ADR does not weaken it and no future issue may cite this ADR to.

The sequence owns quorum as a property of itself, under two rules:

  • It refuses to start unless it can take the whole site down. A sequence that could reach step 4 and no further would leave the cluster in the state the guard exists to prevent, arrived at deliberately, which is worse than the accident.
  • A partially quorate cluster is never an outcome. The sequence either leaves quorum fully intact or breaks it as its final act. There is no intermediate resting state where some control-plane nodes are down and the cluster is limping.

A sequence that fails partway is a defined state, not an error. It parks as Quiesced: the process is safed, some machines are stopped, the control plane is untouched and quorum is intact. Quiesced is stable and can be sat in indefinitely. From it, exactly two things are reachable: continue, or site:outage-abandon, which returns everything to service. Every step that can stall — a drain a PDB blocks, a controller that will not reach safe state, a node that never goes Ready-false — parks here rather than inventing a state of its own.

4. The product's responsibility ends at safed and quiesced

The product's job is to bring the plant to a state where stopping is safe, and to stop the software in an order that preserves the record. Cutting power is performed by a channel that survives the cluster, and on a deployment with no such channel it is performed by a human.

What the product may assert, and what it may not. This governs both the UI and the audit trail:

May assert Because
The field reached its configured safe values The runtimes record it on the #1283 store-and-forward queue
The history was flushed and the archive landed The historian and archiver report it before they stop
The record of the human act is durable It is written before the apiserver is stopped (§ 6)
The control plane was stopped The sequence performed it
Power was cut, on the unattended form only The power channel reported the result of a verb it performed

The product never asserts that power was cut on the attended form. It asserts that it asked a human to do it. A UI that says "site powered off" when a person is still walking to the rack has told an operator something false about the plant, and an audit trail that says it has manufactured evidence.

The way back is a readout, not a record (#1312, founder ruling 2026-08-07). The return to service is reported by GET /api/v1/sites/{site}/return, which derives six stages from live state on demand: machines and kubelets, etcd quorum, the operators and message bus, controllers and their I/O, unit runtimes and their leases, and units at rest. It has no start event and no end event, it stores nothing, and it adds no action — it shares site:outage-read, because whoever may watch a site leave production should watch it come back. Convergence is deliberately not written to the trail: a computed fact written down becomes a claim to maintain, and this one is re-derivable at any moment from objects the product already publishes.

The rule above governs it unchanged. Every stage names the observable it read and what it cannot see, and two readings in particular are false green rather than absent:

  • A machine that is not answering is a kubelet that is not reporting, never a machine that is powered off. The product has no view of power on the attended form and does not acquire one by inference.
  • A Unit's status survives the outage in etcd. Every unit is driven to Idle before anything stops, so the moment the apiserver answers again every unit reports Idle — from a write made before the machines went off. That is a true record of the past and an unchecked claim about the present, so a unit counts as back only once its runtime is ready, which is when something is alive to have written the state.

The readout separates the five control-system stages from the sixth. A unit running a batch is not at rest, and gating the surface on all six would put a "coming back" readout in front of an operator whose plant is demonstrably working. What summons it is machinery that has not returned; where the process has got to since is reported inside it and decides nothing. Neither reading ever means the plant is ready for work, which is a judgement the product does not make.

5. The attended carve-out

A site outage is offered in two forms.

The unattended form requires the deployment's power channel to advertise power-on for the site's machines. It is what a remote or unstaffed site gets, and it is the only form that can terminate without a person present.

The attended form is available with no power channel at all. It shuts the machines down gracefully through the in-band channel, and its ceremony states in plain words that they do not come back by themselves: somebody has to press the power button at the rack.

Corrected 2026-08-05 (#1288). This originally said the attended form leaves the machines running, so a human could remove power at the rack. That was wrong in the direction that matters: it made the product recommend a hard power cut of a live operating system over the sequenced shutdown it could already have asked Talos to perform. The error came from reading ADR 0034 § 4 as forbidding shutdown over a channel that cannot power on, when what that section protects against is a shutdown nothing can undo, and this form's whole premise is that somebody is there to undo it. ADR 0034 § 4 is amended to match.

Availability is a property of the site; invocation is a property of the moment. Both are required, and the split is the decision:

  • Whether a site may use the attended form is deployment configuration on the Site, not a per-request choice. Whether somebody can be at the rack is a standing fact about a place, and a remote site must not be able to click through to a form that assumes a person who is a thousand kilometres away. It sits on the Site rather than in chart values because a cluster can host several sites and a power channel is cluster-wide while attendance is not.
  • Each invocation additionally carries an explicit acknowledgement that a person will perform the manual act. A site that is normally attended is not attended at 03:00 on a public holiday, and the standing configuration cannot know that.

The audit record distinguishes the forms, and this is the point of the whole carve-out. The record states which form ran, that no power-on channel existed, and who acknowledged the manual act. A person who clicked through an attended shutdown and then went home has done something the record must be able to show, and the only way it can is if the two forms are not interchangeable in the trail.

6. Ordering, and the record that has to outlive the plane

The order is fixed and each step has a different precondition:

  1. Bring the process to a state where stopping is safe. Every unit Idle or Held, no batch mid-phase.
  2. Write the audit record of the human act, and make it durable. This is step 2 rather than a footnote at the end: the apiserver is going away and a record still being written when it goes is a record that never existed. The sequence does not proceed past this point until the write is confirmed.
  3. Stop the controllers, with outputs driven to their safe values through the #1283 terminal-stop path. Controllers first, because they are the machines touching the plant.
  4. Stop the workers, so the historian flushes and the audit archiver lands what etcd holds.
  5. Stop the control plane, breaking quorum deliberately.
  6. Cut power, out of band or by a human, per § 5.

A progress UI that becomes unreachable during step 5 is acceptable, because the record is already durable. A progress UI that becomes unreachable while the record is still being written is the failure this ordering exists to prevent.

7. What refuses the start, and the one caller that cannot be refused

A human-initiated outage refuses on any of these:

Precondition Disposition
Any batch not in a terminal state Refused. Never acknowledgeable
Any unit above Idle, other than Held Refused
A unit in Held Acknowledgeable. Held is a deliberate safe resting state and a turnaround routinely starts from one
A unit in Held with an operator prompt outstanding Refused
The site cannot be taken down in full (§ 3) Refused
No power channel and the site is not configured attended (§ 5) Refused

The outstanding-prompt case is the interesting one and it is refused deliberately. A prompt is a question the plant asked a person, and going down with it unanswered destroys the context in which it was asked: the operator returns to a prompt whose situation no longer exists. Answering it or abandoning the work are both cheap, and neither is recoverable after the fact.

A machine-initiated outage is not refused by any precondition a human could have cleared. When the UPS fires, the mains are already gone. Refusing achieves nothing except a worse ending, because the alternative to an orderly stop is not a tidy plant — it is the ungraceful death this feature exists to replace. So on the machine-initiated path the preconditions are evaluated and recorded rather than enforced, and the record states what was running when the sequence began.

This asymmetry is the sharpest decision in the ADR, and it is deliberate. The same act is refused from a button and permitted from a UPS, because a refusal is only useful when the person refused has a better option available. The record carries the initiator, so a trail can never confuse the two.

Alternatives Considered

A site outage as a fan-out of node shutdowns, with no sequence object. Rejected. It has no ordering (the control plane would race the controllers), no place to hold a partial failure, nothing to abandon, and no object on which to record the human act before the plane goes. It also runs straight into #1238's guard on the control-plane nodes and would have forced the loophole.

Relaxing #1238's quorum guard so the site verb can use the node path. Rejected outright. The guard's value is that it is absolute; a guard with one sanctioned bypass is a guard whose next bypass is an argument rather than a decision. The sequence owning its own last leg costs more code and keeps the guard worth having.

Making the attended form a per-request choice only. Rejected. It would let a remote site's operator select the form that assumes a person at a rack they are nowhere near, and the product would have no basis to object. Standing configuration is what carries a fact about a place.

Making the attended form deployment configuration only. Rejected for the mirror-image reason: a site that is normally attended is not attended at 03:00 on a holiday, and a standing flag cannot know that. Requiring both is not belt-and-braces, it is two different facts being asserted by the two parties who can each know one of them.

Naming the verb site:shutdown for symmetry with servers:shutdown. Rejected. The symmetry is the problem: the two acts have different blast radii and, more importantly, the site verb does not perform the power act its name would claim.

Refusing the machine-initiated path on the same preconditions as the human one. Rejected, and it is worth stating why it was tempting. Uniform rules are easier to reason about and easier to test. But the UPS caller has no better option to be steered toward, so a refusal there is not a safety measure — it is the product declining to help while the plant loses power anyway.

Deferring servers:power-on to #1284. Rejected. #1282 established that a PoweredOff node has no route back, and the implementing issue is exactly where a name gets minted without the contract being considered. Settling it here costs one table row.

Consequences

  • A new action group, and an eighth servers: name. ADR 0034's seven-name table is no longer the whole of the infrastructure contract, and its § 6 gains an amendment note pointing here.
  • The Site spec grows an attendance field. It is deployment configuration expressed on a product object, and a site that sets nothing gets the unattended form only, which is the safe default: a deployment that never opens the file cannot reach the form that assumes a person.
  • The sequence cannot reuse the node path for its last step, so the control-plane leg is code that exists only for this feature. That is the price of keeping #1238's guard absolute, and it is the right price.
  • The audit trail gains a record class that must be written mid-sequence rather than at completion. Every other consequential act in the product records at request time and reports at completion; this one cannot report at completion because the reporter is gone.
  • The UPS trigger is now a design constraint rather than a follow-on. The sequence has to be invocable by something other than a button from the first commit, because retrofitting a second caller onto a handler is how the precondition asymmetry would get lost.
  • Quiesced is a state a plant can sit in. A partial failure leaves the site out of production with the cluster healthy, which is recoverable and boring. That is deliberate: the states worth designing are the ones a plant will actually be found in at 02:00.
  • The compliance surface moves. 21 CFR Part 11 § 11.10(e) gains the outage record, including the attended-form acknowledgement. docs/compliance/iec62443-4-2.md CR 5.1 gains whatever conduit #1284's backend opens, on top of what ADR 0034 already recorded for the in-band provider.
  • Reversibility. The ceremony, the precondition table and the tier assignments are cheap to change before release. The four action names are a product contract the moment a release carries them, and the sequence object's schema is a migration once a deployment holds one.
  • What this ADR does not decide. Bringing a cluster back up beyond what the power channel offers, and any scheduler for recurring outages. Both are follow-ons once the sequence exists, and both would have made this decision larger without making it better.