DR Drill Log — Audit Archive Corruption (2026-04-22)¶

Evidence artifact for 21 CFR Part 11 §11.10(c) restore-path qualification. Exercises the detect → quarantine → restore-from-Object-Lock → deviation-report loop documented in Scenario: Audit Archive Corruption.

Infrastructure¶

Item	Value
Cluster	Two-droplet k3s, `dcs-dr-ctrl` + `dcs-dr-edge`, DigitalOcean NYC1, provisioned via `scripts/dr-drill-infra.sh up`
Chart revision	`sha-257f3dd` (all component images)
Archival config	`historian.audit.archival.enabled=true`, `schedule=/5 * * *`, `activeRetentionDays=0`
Immutable mirror	`historian.audit.archival.immutable.enabled=true`, in-cluster minio `minio.minio-drill.svc`
Bucket	`drill-audit-archive` created `mc mb --with-lock`, default retention `COMPLIANCE 30d` (per-object 365d via archiver)
Credentials	Secret `dcs-audit-archive-s3` (AWS_ACCESS_KEY_ID/SECRET_ACCESS_KEY for minio root)
HMAC signing key	`dcs-signing-key` key `v1` (chart-auto-generated)

Seed corpus¶

Site name is historical

This drill ran on 2026-04-22, when the pharmaceutical reference set was the site pharma-plant. #1178 re-homed that set onto riverbend. The names and object keys below are reproduced as they were recorded, because they are literal evidence: the object keys are the paths the archiver actually wrote, and rewriting them would falsify the record. Do not update them in a retired-name sweep.

Three AuditRecord CRs in site-pharma-plant with spec timestamps 2026-04-21T10:00Z / 11:00Z / 12:00Z. A single archiver run produced one audit_archive_manifest row covering all three records.

manifest_id	record_count	key_id	object key
`e8739a19-f7c2-4d45-9d18-7b8f0a002b41`	3	`v1`	`audit-archive/site-pharma-plant/2026/04/22/e8739a19-f7c2-4d45-9d18-7b8f0a002b41.json`

Baseline dcs audit verify --archived green (total=1, passed=1, failed=0).

Drill procedure (wall clock timestamps are UTC)¶

Step	Action	Timestamp	Notes
1	Captured pre-mutation row of `seed-audit-1` (uid `d1bea468-…`) as evidence	19:27:08	`SELECT manifest_id, uid, namespace, ts, message FROM audit_records WHERE uid='…'`
2	Corrupted one row via `UPDATE audit_records SET message='tampered by drill' WHERE manifest_id='…' AND uid='…'`	19:27:08	One-row mutation; digest scope covers message field
3	Ran `dcs audit verify --archived`; exit 1, one result with `reason=digest_mismatch`	19:27:08	Detection RTO <1 s — single-manifest verify
4	Created quarantine AuditRecord `audit-quarantine-zc8pl` in `dcs-system` with `Target.Kind=ArchiveIntegrityCheck` and manifest-id as `Target.Name`	19:27:27	Tampered PG row left intact as forensic evidence
5	`mc stat` confirmed Object Lock on bundle (`X-Amz-Object-Lock-Mode: COMPLIANCE`, `Retain-Until-Date: 2027-04-22T19:18:32.000Z`)	19:27:42	Bucket was created `--with-lock` at T0; per-object retention = 365 days set by archiver (chart floor `MinRetentionDays=365`)
6	`mc cp` pulled the bundle out; manifest fields match the PG row (digest, keyID, recordCount)	19:27:45	Records array inside the bundle still carries the canonical un-tampered messages
7	Built a single `BEGIN; UPDATE×3; COMMIT;` SQL transaction from `.records[].message` and applied via `kubectl exec … psql` stdin	19:29:10	Transactional restore; three rows updated
8	Re-ran `dcs audit verify --archived`; exit 0, all passed	19:29:12	Recovery complete
9	Filed deviation AuditRecord `audit-deviation-lzcxh` with root cause, Object-Lock proof, restore txn summary	19:29:30	21 CFR Part 11 §11.10(e) trail
10	Ran `dcs qualify dr --fail-on-deviation`; signed qualification `qual-20260422-192940-443c0887`	19:29:40	Seals the drill

Measured RTO / RPO¶

Metric	Value
Detection RTO (via on-demand `dcs audit verify --archived`)	<1 s
Detection RTO (via scheduled `gateway.archiveIntegrity`, `interval=15m` floor)	bounded by scheduler interval (worst case = interval; 15 min floor)
Full-recovery RTO (corruption to clean re-verify)	124 s wall clock
RPO	0 — mirror captures every signed batch; no loss

Detection on a single-manifest corpus is noise-floor; a production cluster with thousands of manifests will be limited by the historian query scan, not the HMAC re-compute. Budget 1s/manifest as a conservative upper bound on future drills.

Evidence trail (AuditRecord names)¶

All AuditRecords live in dcs-system on the drill cluster. Before teardown, the critical three were captured to the drill state directory so the archive-attested deletion interlock (the tampered PG rows) has a durable off-cluster copy:

Role	AuditRecord name	Action	Result
Quarantine	`audit-quarantine-zc8pl`	Update	Failure
Deviation report	`audit-deviation-lzcxh`	Update	Success
Qualification (drill seal)	`audit-flml2` (`dr-qual-20260422-192940-443c0887`)	Create	Success

Promoted status¶

docs/compliance/21cfr11.md §11.10(c) row flipped from Implemented (archival + tamper-evidence + scheduled verification) / Untested (restore) to Implemented and drilled (archival + tamper-evidence + scheduled verification + restore).
docs/dr-runbook.md "Audit records (archived, in historian)" row in the RTO/RPO table flipped from n/a — pending drill to the measured numbers above, Last drill stamped 2026-04-22.

Teardown¶

Drill cluster + in-cluster minio destroyed via scripts/dr-drill-infra.sh down. AuditRecord evidence above is the only surviving artifact; the bundle itself is gone with the bucket. For a future audit-integrity scenario, preserve the bundle by copying out of minio before teardown.