DR Drill Log — Certificate Expiry (2026-04-22)

Evidence artifact for Scenario: Certificate Expiry. Exercises the cert-manager-failed degenerate case: cert-manager is down while a component's mTLS cert is minutes from expiry, and the only way back is manual re-issue from the CA Secret.

Infrastructure

Item	Value
Cluster	Two-droplet k3s, dcs-dr-ctrl + dcs-dr-edge, DigitalOcean NYC1
Chart revision	sha-257f3dd
cert-manager	v1.16.2 (patched --cluster-resource-namespace=dcs-system)
mTLS	mtls.enabled=true, certManager.clusterIssuer=true, duration=24h, renewBefore=8h
CA Secret	cloud-native-dcs-mtls-ca in dcs-system (auto-created)
Target component	Gateway (cloud-native-dcs-gateway-mtls Certificate / Secret)

Drill procedure (wall clock timestamps are UTC)

Step	Action	Timestamp	Notes
1	Baseline dcs health green	19:20:33	Pre-flight
2	Scaled cert-manager + cert-manager-webhook + cert-manager-cainjector to 0 replicas	19:20:40	T0 — "cert-manager failed" simulated
3	Patched cloud-native-dcs-gateway-mtls Secret tls.crt with garbage (Zm9vYmFyCg==); kubectl rollout restart deploy/cloud-native-dcs-gateway	19:21:05	Forces new pod to fail cert load so failure surfaces; old pod stays up until new one is ready
4	kubectl delete pod on the old gateway pod to end the grace window	19:21:25	Deployment now has no Ready replicas; port-forward returns EOF
5	Confirmed cert load failure in pod logs: failed to build runtime TLS transport: loading certificate: tls: failed to find any PEM data in certificate input	19:21:34	Detection confirmed
6	Pulled tls.crt + tls.key from cloud-native-dcs-mtls-ca Secret	19:21:40	Break-glass material
7	openssl ecparam -name prime256v1 … + CSR with DNS:cloud-native-dcs-gateway{,.dcs-system.svc,.dcs-system.svc.cluster.local} + openssl x509 -req -days 1 … against the CA	19:21:58	Intentional 1-day expiry forces followup once cert-manager is healthy
8	kubectl create secret generic cloud-native-dcs-gateway-mtls --from-file=tls.crt=… --from-file=tls.key=… --from-file=ca.crt=… --type=kubernetes.io/tls --dry-run=client -o yaml \| kubectl apply -n dcs-system -f -	19:22:10	Secret patched in place
9	kubectl rollout restart deploy/cloud-native-dcs-gateway; rollout status successfully rolled out	19:22:43	T1 — service restored
10	dcs health green via port-forward	19:23:13	End-to-end verification
11	Scaled cert-manager back to 1; deleted the manually-issued Secret; cert-manager re-created it within ~10 s; rolled the gateway to pick up the canonical cert	19:24:00	Returned to steady state
12	Ran dcs qualify dr --fail-on-deviation; signed qualification qual-20260422-192458-ca6ef6e0	19:24:58	Seals the drill

Measured RTO

Metric	Value
cert-manager-down to gateway-healthy (wall clock)	123 s
Detection window (cert-manager-down to pod logs showing TLS error)	~54 s
Recovery window (openssl + kubectl apply + rollout)	~69 s

The bulk of the wall clock is openssl invocation + kubectl round-trips. A templated re-issue-component-cert.sh script checked into ops would compress this substantially — file that ticket after the drill.

Evidence trail

Artifact	Value
Qualification AuditRecord	audit-rkm22 (dr-qual-20260422-192458-ca6ef6e0)
Target Certificate	cloud-native-dcs-gateway-mtls in dcs-system
CA fingerprint (SHA-256)	FC:5F:FB:86:10:E2:5D:66:70:39:7A:A7:B6:32:C9:4B:3B:FD:59:95:1F:6C:1D:43:E0:52:77:9C:B6:B0:34:FD

Follow-ups

• The Certificate Expiry section of dr-runbook.md was flipped from "blocked on #168 / tabletop-only" to a drilled procedure with the full openssl + Secret-patch sequence and the measured 123 s wall clock.
• Break-glass certs minted with -days 1 — if the drill clone of this procedure is ever run on a live cluster, verify the followup cert-manager re-issue happened before that window closes.
• Templated re-issue script: open a ticket against the deploy skill to land scripts/break-glass-component-cert.sh so the openssl steps are canned and tested.

Teardown

Drill cluster destroyed via scripts/dr-drill-infra.sh down. The qualification AuditRecord above is the only surviving artifact.