Skip to content

Edge-Runtime Failover Runbook

This runbook covers re-binding a unit's runtime to a designated standby node when its edge controller is lost: the operational procedure for the hold-then-resume failover mechanism decided in ADR 0006 and implemented across the API, runtime, operator, CLI/gateway, and UI.

Failover moves the software runtime (the FB scan loop and I/O driver connections) to other hardware. The controller↔I/O link is a network connection (remote Ethernet I/O), so the standby needs field-network reach but no physical adjacency. It does not replicate live state: recovery is an ISA-88 Restart, and field outputs hold last value (or device fail-safe) during the gap.

Before you start

  • Know the unit's availability mode. dcs get unit <name> -s <site> shows it, or the System app → Controller detail → Hosted Units table (Availability column). The procedure differs by mode (below).
  • Confirm a standby exists. dcs unit failover <name> -s <site> (no --to-node) lists the eligible targets the operator resolved from spec.availability.failoverTargets. An empty list means no enrolled, Ready, field-reachable standby is configured. Fix that first (see production-deployment.md).
  • A manual failover requires the unit not be actively Running. A dead-node unit is normally already Held, whether by the watchdog/self-hold path or (Failover mode) by the automatic path itself once the lease expiry outlasts the hold bound. If it is still Running, Hold the owning batch first (dcs command Batch <batch> Hold -s <site> --reason "…").

Mode Autonomy (default) — manual fenced failover

In Autonomy mode the edge keeps controlling through a control-plane partition, so the operator must physically fence the old node before re-binding. Otherwise two writers could drive the same field devices.

  1. Physically fence the old node. Power it off, or disconnect it from the field network. This is the load-bearing step: confirm it, do not assume it. On Modbus there is no session exclusivity. A live old node and a new one both writing is undetected, last-write-wins.

    On a bonded controller, the field network is two cables

    The recommended posture bonds the field interface across two ports on two switches (ADR 0026). Pulling one member fails the bond over to the other and the node keeps writing. A half-fenced node is exactly the dual-writer hazard this step exists to prevent. Pull both members, or power the node off. Powering off is the unambiguous fence. Prefer it.

    1. List eligible targets: dcs unit failover <name> -s <site>.
    2. Re-bind, certifying the fence: 
      dcs unit failover <name> -s <site> --to-node <standby> \
  --confirm-fenced --reason "primary IPC PSU failure, node powered off"
      --confirm-fenced is required in Autonomy mode and is recorded on the AuditRecord (21 CFR Part 11). Omitting it is rejected. --reason is required on every failover path (#687) and must meet the deployment reason policy.
    3. If the standby already hosts another unit's runtime, the co-location guard rejects the request. Add --allow-colocation only after accepting the 1:1-controller:unit trade-off.

UI equivalent: System app → Controller detail → Hosted Units → Failover (visible to operate-lead). The dialog's fenced-confirmation checkbox and justification are required and the target picker shows co-location occupancy.

Steps 2 to 4 on a two-node edge pair: the standby listed as occupied before anything is submitted, the unfenced re-bind rejected, the co-located re-bind rejected, and then the same request with the fence certified and the trade-off accepted, landing with a new fencing epoch. The node is fenced by the operator off camera. Nothing here failed on its own.

Mode Failover (opt-in) — lease-based

In Failover mode the runtime holds an operator-brokered control lease and self-fences when it cannot renew (FB output writes stop, reads continue). The decision runs on its own local clock, so it works under a control-plane partition. The operator then re-binds automatically.

Automatic path (normal case): no operator action. Once the operator has observed the lease Expired for the safety margin (½ the lease duration, clamped 10–60s), it re-binds the runtime to the first free eligible standby, stamps a new epoch, and recreates the pod there. If the unit is still Running when the lease expires (a batch mid-run when the node died), the operator first places it on ISA-88 Hold itself once the expiry outlasts the hold bound (½ the lease, clamped 5–30s). The result is the same Hold, Critical alarm, and RuntimeCrashDetected condition the runtime crash detector produces. After the margin it re-binds (hold-then-resume, ADR 0006). A runtime that recovers and renews its lease inside the hold bound rides through with no Hold. Watch it land:

dcs get unit <name> -s <site>   # RuntimeBinding + FailoverRequest conditions
The automatic path with nothing for an operator to do but watch. The control lease expires and the runtime self-fences, the operator re-binds it to the standby, and only afterwards does Kubernetes mark the lost machine Offline. The kill is injected off camera. What is filmed is the lease and the two chassis on either side of it.

When the automatic path is blocked: if every eligible target is occupied, a NoFreeTarget condition is set and no re-bind happens. Either free/provision a standby, or run a manual co-located failover:

dcs unit failover <name> -s <site> --to-node <standby> --allow-colocation \
  --reason "no free standby; co-locating during incident <ref>"

Planned maintenance (lease still Held): to move a healthy Failover-mode unit (e.g. to service its node), the lease has not expired, so use the fenced-confirmation override exactly as in Autonomy mode. Fence the node first, then --confirm-fenced.

After failover (both modes)

  1. Verify the new binding: dcs get unit <name> -s <site> shows RuntimeBinding.node = the standby and an incremented epoch. For Failover mode the lease returns to Held once renewals resume.
  2. The control program redeploys automatically from the control-plane source of truth (CRDs). There is no hostPath migration. Confirm the runtime pod is Ready and drivers reconnected (Diagnostics page, or dcs get unit <name> -s <site>).
  3. Restart the process. Recovery is an ISA-88 Restart. Issue it through the batch (dcs command Batch <batch> Restart -s <site>) and confirm the recipe-defined restarting logic re-establishes process conditions.
  4. Clear the alarms. The lease-expired (critical) and failover (high) Alarms auto-clear when the lease re-establishes. Acknowledge per your ISA-18.2 workflow.

Failing back

There is no automatic fail-back. The binding is status-owned (RuntimeBinding.directedNode) and survives a GitOps sync of the spec. Once the original node is repaired and re-enrolled, fail back with another explicit failover to it (same procedure, naming the original node).