Switch Commissioning Runbook

Procedure for taking a factory-default managed switch to a zone-segmented, management-hardened state, over the network, without a serial console, and with no second trip to the rack.

The ordering and the verification logic are the portable part. Vendor syntax is not: the MikroTik SwOS commands below are a labelled worked example and will not transfer to a Cisco or Juniper platform. Every step states its intent first and its syntax second, so the intent survives a vendor change.

The zones this procedure enforces at the physical layer are the ones described in Network Requirements (External, Control, Field, and Field Bus), plus the management zone that this runbook exists to protect. Read that page first for the zone model itself, the port matrix, and the controller-node bonding posture. This page covers only the act of configuring the switch.

The governing idea

The management plane is a zone in its own right. A switch that enforces a beautiful IT/OT split while accepting configuration from any port, any VLAN, and any address has not segmented anything. It has published an override switch on the same wire it is supposed to be protecting. Segmentation and management restriction are one piece of work.

Everything below follows from that, plus one operational constraint: the network is being reconfigured over the network, so any single mistake can strand the engineer outside the device being changed.

Example values

Intent statements use placeholders. The vendor command blocks use the worked example values, because the encoding traps described below only make sense against concrete numbers. Substitute the site's own values.

Placeholder	Meaning	Worked example
<switch>	Address the switch answers on during commissioning	10.10.10.3
<mgmt-subnet>	Subnet permitted to manage the switch	10.10.10.0/24
<mgmt-vlan>	VLAN carrying the management zone	10
<admin-port>	Switch port the commissioning host is cabled to	1
<identity>	System identity written to the device	<site>-sw1
<snapshot>	Committed, readable config dump	<identity>-config.txt

Commission while the device is still at factory defaults

The cheapest recovery from a lockout is the reset button, and reset costs nothing while there is no configuration worth losing. Once the switch carries a week of accumulated changes, the same reset is a disaster, and risky-but-correct changes start getting avoided.

Commission early, and keep a snapshot so reset stops being frightening.

Pre-flight

1. A snapshot you could rebuild from. A readable dump that could be retyped, generated by reading every config document the device exposes, and committed to the site's deployment repository as <snapshot>. A vendor backup blob that has never been restored does not qualify.
2. Physical access, or someone at the rack. This procedure takes lockout risk deliberately. Take it when reset is a walk. When reset is a drive, wait.
3. A known-good management path that is not about to change. A dedicated host on <admin-port> is sufficient.
4. The device's real schema. See the next section.

Read the device's own schema — do not trust documentation

Vendor docs describe a model. The firmware implements one. Where they differ, the firmware wins, and the docs will not say so.

Worked example — MikroTik SwOS

The entire SwOS UI is a single gzipped index.html with the field definitions inline, so the authoritative semantics are readable directly:

curl -s --compressed --digest -u 'admin:<password>' \
  http://<switch>/index.html -o swos.html
python3 -c "s=open('swos.html').read(); i=s.find('avln'); print(s[i-180:i+180])"

That is how the avln field turns out to mean "Allow From VLAN", an access restriction. Reading it as a management-VLAN assignment would have migrated the switch CPU onto <mgmt-vlan> for no reason, taking real lockout risk to achieve nothing.

Use a string search. These files have 11k-character lines, and a regex such as .{0,120}avln.{0,120} backtracks until it is killed.

Prove the write mechanism on something harmless

Before any change that can strand you, write one cosmetic field and read it back. That proves authentication, request format, encoding, and persistence in a single step that cannot hurt anything.

The system identity is a good candidate, and it is worth setting regardless.

Worked example — MikroTik SwOS

curl -s --digest -u 'admin:<password>' -X POST \
  -d "{id:'$(printf '<identity>' | xxd -p | tr -d '\n')'}" http://<switch>/sys.b

This is also where encoding traps surface while they are still free. On SwOS, IP fields are little-endian: 10.10.10.3 is 0x030a0a0a. The big-endian reading, 0x0a0a0a03, is the trap. Reversed, the write is a syntactically valid address that strands the device.

Never trust the status code — read back and compare

A SwOS POST has been observed returning HTTP 401 while applying the change anyway. Retrying on that 401 applies the change twice. Treating it as a failure and moving on leaves the engineer with a wrong mental model of the device state.

Verify by reading the value back, every time. This is not paranoia about one vendor's bug. It is the only check that is actually about the thing you care about.

Ordering: the part that prevents lockouts

The sequence matters more than any individual setting.

1. Pin management addressing first

While the switch holds a DHCP lease, its address depends on a router whose port is about to be reconfigured. Break that dependency before creating it.

Worked example — MikroTik SwOS

curl ... -X POST -d '{iptp:0x01,ip:0x030a0a0a}' http://<switch>/sys.b

iptp:0x01 selects static addressing, and ip carries 10.10.10.3 in the little-endian encoding described above.

2. Define the zones, but do not apply them

Creating VLANs changes no forwarding on its own. Get the whole table in place and readable before anything starts using it. One VLAN per zone from the zone model, plus <mgmt-vlan>.

3. Configure every port that has no link

Dark ports cannot strand you, and on a partially-built rack that is most of them. Configure them to their final intended state now, so the remaining work is cabling with the switch already done.

4. Move the admin port and the uplink or trunk in a single write

This is the step people get wrong. Doing the two sequentially creates an interval where the admin host is in the new VLAN and the router is still in the old one. On hardware that egresses the new VLAN tagged toward a router expecting untagged frames, the link drops mid-procedure.

If two ports must agree for your session to survive, change them in one transaction.

5. Harden management in tiers, verifying between each

Apply the least risky first, so a failure names its own cause.

Tier	Control	Blocks	Risk
1	Source subnet	Anything off <mgmt-subnet>	Low
2	Management VLAN	Every host in the Control, Field, and Field Bus zones	Medium
3	Admin port	Every other host, including the site's own servers	Higher

Tier 3 is the one with real teeth. After tiers 1 and 2 the remaining exposure is "any host on the management VLAN", which on a DCS deployment means the Kubernetes nodes, the largest attack surface present. A compromised workload that escapes to a node should not be able to reconfigure the segmentation protecting the Field zone.

Worked example — MikroTik SwOS

curl ... -X POST -d '{alla:0x000a0a0a,allm:0x18}' http://<switch>/sys.b  # 10.10.10.0/24
curl ... -X POST -d '{avln:0x000a}'               http://<switch>/sys.b  # VLAN 10 only
curl ... -X POST -d '{allp:0x00000001}'           http://<switch>/sys.b  # port 1 only

Tier 3's cost, stated plainly: moving the admin cable to another port now locks you out. That is an acceptable trade when the port is labelled and the config is snapshotted. It is not acceptable when the recovery story is "rebuild from memory".

Verification: separate what was proved from what was assumed

The discipline that matters here is refusing to record an untested control as verified. Split the commissioning record in two.

Proved. Each of these is a read-back or an observed behaviour:

• Every port's PVID, VLAN mode, and receive filter, by read-back after write.
• Whether the management-VLAN restriction compares the classified VLAN or the frame tag. On SwOS it is the classified VLAN, so untagged ingress on a port whose PVID is <mgmt-vlan> still manages. That was genuinely uncertain beforehand and could have been a lockout. Doing it resolved it where reasoning about it could not.
• That management survives all three hardening tiers.

Not proved, and not claimed: whether the hardening actually blocks anything. Verifying a deny rule needs a host in a denied position, so it is not testable until the zones have live members. An untested deny rule is a hypothesis. It belongs on the acceptance list of whichever commissioning step first puts a host in each zone.

Negative tests are vacuous alone

A ping that fails against an address with nothing answering is indistinguishable from one stopped by segmentation. Always pair a deny test with an allow test against the same live target in the same session. The positive is what gives the negative meaning.

A suite that only ever runs the negative passes just as happily against an unplugged switch.

Rollback — and the trap inside it

1. Hold reset while powering on to return to factory defaults.
2. Reach the device at the vendor's fallback address (SwOS: 192.168.88.1) or by DHCP.
3. Replay every step against the committed <snapshot>.
4. Finish with a field-by-field diff. Not a look-over — a diff.

Budget roughly 15 minutes. This path has been executed for real, so the estimate is a measurement.

A partial restore looks complete

This is the failure mode worth internalising, because nothing about the device will warn you.

A factory reset reverts everything, including the management hardening, and the hardening is invisible in normal operation. Restore the addressing, the VLAN table, and the port assignments, and the switch looks entirely correct: zones enforced, traffic flowing, every visible behaviour right. The management plane is wide open and there is no symptom.

The segmentation is the part you can see. So it is the part you check, and the part that gets restored. The access control protecting it is the part you forget.

This has happened in practice. A password change locked the operator out, reset was the only way back, and the restore stopped after addressing, VLANs, and ports, leaving management open on a switch believed to be fully restored. It was caught by diffing the device against the committed snapshot, and by nothing else.

So: verify by diff

Commit a verification script alongside <snapshot>, so that "check it matches" is a command:

./verify-switch-config.sh          # exit 0 = matches snapshot, 1 = drift

At minimum it reads every config document back from the device, compares each field against the snapshot, and exits non-zero on any difference. That includes the management-hardening fields, which are the ones a restore silently omits. Run it at the end of every restore, and whenever you want to know whether the device still matches what the repository claims.

Rehearse the whole rollback path once in a lab, so the first real execution is not at a client site.

A snapshot is not the design, and the diff needs a third opinion

The snapshot records what the device was last seen doing. The port map records what the device is supposed to do. Those are different claims, and a verifier that only compares the device against the snapshot cannot tell you when they have come apart.

The way they come apart is a design decision that lands as an edit. Someone re-plans a zone, rewrites the port table, and commits. No write reaches the device, the snapshot is not re-captured, and the verifier keeps passing. Device and snapshot agree with each other, and both disagree with the document the engineer is reading. The port table meanwhile still says its rows were read back from the device, because they were, before the edit.

This is not the partial-restore trap wearing a different hat. There the documentation was right and the device was wrong. Here the device is doing exactly what it was last told, and the instruction never arrived.

So have the verifier parse the port map too, and compare the documented PVID for every port against the live device. It costs a few lines of parsing, and it converts a silent gap into a failing command:

DRIFT — live switch does NOT match the port map in switch-ports.md

  port 13 PVID
    documented: 30 (switch-ports.md)
    live:       20 (live)

A port map edit is not a configuration change. If the table was edited
for a design decision, the write to the device is still owed.

Two habits fall out of it. Prove the check detects, by running it against a tampered copy of the port map, because a comparison that silently parses zero rows passes exactly like one that matches. And commit a capture script next to the verifier, since a snapshot nobody can regenerate is a snapshot nobody refreshes. Reach for capture only after deciding the device is the side that is right. Run against a drifting device, capture does not fix the drift. It writes the drift into the repository and silences the alarm.

Credentials are not part of this procedure

Commissioning is done with the factory credential, and the device is handed over still holding it. Setting and vaulting the real password is a separate, owner-different activity.

A commissioning runbook that also mints credentials encourages the engineer doing the work to hold them, which is exactly what the split exists to prevent. Finish commissioning, file the credential request, hand over.

But hand over the device's hazards with it

Ownership splits. Knowledge must not. At handover the commissioning engineer knows the device's API intimately and the credential owner knows none of it. The credential owner is about to write to the least forgiving endpoint on the device.

The handover must carry the write hazards along with the requirements. One handover that omitted them cost a lockout and a factory reset: SwOS silently truncates an over-long password written through its API, the follow-up write to "fix" the mismatch is encrypted with the old password, and the device offers no recovery path but the reset button. Every one of those was knowable at handover time and none of it was written down.

Include, at minimum:

• Length and character limits on the credential.
• Which endpoint writes it.
• Whether a failed write is recoverable in software.
• The rule for what to do on a suspected mismatch. For SwOS that rule is: set it once, at most 15 characters, verify by re-authenticating, and never chase a suspected mismatch with a second blind write.

Tell the credential owner what is already applied

A second-order failure from the same handover: the credential owner read a snapshot predating the hardening and recorded management-narrowing as still-pending work when it had already shipped. Combined with the reset, that turned "already done" into "quietly undone", with a written note claiming it was never started.

When two teams touch one device, the device is the source of truth and the diff is how you consult it. State plainly in the handover what is already applied, and point at the verification command. A snapshot you point at instead would age.

Client-deployment deltas

Everything above is executable in a lab with nothing but a network cable. Four things change at a client site, and each one moves from improvement to requirement.

• Serial console access. A production site should not depend on the reset button. Have console before taking tier-3 risk.
• A management VLAN with no other tenants. A small lab can fold management into the IT VLAN because it has a handful of hosts. A real site gives management its own zone, so tier 2 alone carries the weight tier 3 carries on a bench.
• A second admin port. Tier 3's brittleness is halved by allowing a labelled spare, at negligible security cost.
• Change control. Every step here is a configuration change on a device whose failure stops production. At a client site this runs under the site's change process, and <snapshot> is the before-image that process will ask for.

Vendor translation. The three tiers map to access-class or management ACLs on most platforms. Ordering, atomicity, and read-back verification transfer unchanged. Only the syntax moves.

Commissioning record

Record the commissioned state alongside <snapshot> in the site's deployment repository, and date the verification. The record is what the next engineer diffs against.

Setting	Value
Identity	<identity>
Address	Static <switch>
Allow from subnet	<mgmt-subnet>
Allow from VLAN	<mgmt-vlan>
Allow from ports	<admin-port>
VLAN mode	strict, all ports
Port map	Reference to the site's port assignment document
Password	Set and vaulted; reference to the credential record
Verified	Date, and the command whose exit status proved it

Related Documentation

• Network Requirements — the zone model, port matrix, and controller-node bonding posture this switch configuration implements
• Security Hardening — the cluster-side configuration checklist that turns the same zone model into NetworkPolicy
• Threat Model — why the management plane is treated as a zone of its own
• Device Enrollment — how an edge device joins the Field Zone once the switch enforces it
• Reference Architectures — the hardware patterns these port assignments assume