Skip to content

Capacity Planning

Status: Per-component resource requests, storage formulas, and scale-up signals are populated from the shipping Helm defaults (deploy/helm/cloud-native-dcs/values.yaml) and the observed metric catalogue. Throughput ceilings (tag updates/sec, historian writes/sec, gateway QPS, etcd size per CR) are not yet profiled — entries marked "Needs profiling" should not be cited as design limits. Production-tier node ceilings remain TODO until first-customer data exists.

Resource budgets, throughput limits, and scaling rules for a Cloud-Native DCS deployment. Use this doc to size a cluster before installation and to decide when to scale an existing deployment.

Deployment Tiers

The reference sizes below are sizing anchors, and the tier boundaries are advisory. "Control-plane requests" sums the Helm default resource requests for everything that runs on the Kubernetes control-plane side (operators + gateway + MQTT + historian + CNPG + docs). Edge-device requests cover the per-unit runtime pod.

Tier Target use case Units Tags Batches/day Control-plane nodes Edge devices
Bench Single-unit R&D 1 ~50 ~10 1 1
Pilot Multi-unit pilot line 5-10 ~500 ~50 1 (single node) 5-10
Production Single site, multiple process cells 20-50 5000+ 100+ 3 (HA) 20-50

Bench

  • Control-plane node (min): 2 vCPU, 4 Gi RAM, 20 Gi disk.
  • Control-plane pod requests (sum): ~910m CPU, ~1.3 Gi memory with the historian enabled (includes CNPG at its operator-default 100m / 256Mi), and ~560m CPU / ~840 Mi without.
  • Control-plane pod limits (sum): ~2.1 vCPU, ~2.4 Gi memory with historian.
  • Edge device (min): 1 vCPU, 1 Gi RAM, 8 Gi disk (compact fanless x86_64 IPC class).
  • Historian storage: 10 Gi default is sufficient for ~1 year at bench tag rates.
  • Tested reference host: single-node Talos Linux on a 4 vCPU / 8 Gi RAM x86_64 VM (the reference deployments). An earlier k3s-on-Ubuntu bench rig with a Raspberry Pi 5 edge device is now a community/experimental configuration, since release images are linux/amd64-only.

Pilot

  • Control-plane node (recommended): 4 vCPU, 8 Gi RAM, 40 Gi disk (single node).
  • Control-plane pod requests (sum): ~1.1 CPU, ~1.3 Gi memory with historian (dominated by operators + gateway + historian + CNPG). Unit-runtime pods live on the edge devices and add nothing here.
  • Control-plane pod limits (sum): ~2.3 vCPU, ~2.5 Gi memory with historian.
  • Edge devices: 5-10× 1 vCPU, 1 Gi RAM. Each unit-runtime pod requests 100m CPU / 128 Mi memory. It rarely exceeds 200m / 256 Mi at default scan rates.
  • Historian storage: 50 Gi recommended for 1-year retention at ~500 tags with mixed 0.1–1 Hz sample rates (see Storage Sizing).
  • Gateway autoscaling: disable for pilot (single replica is sufficient at ~50 batches/day, well under the 100 req/s read rate-limit ceiling per replica).

Production

Honest TODO: production-tier node ceilings depend on mix of batch frequency, tag density, historian retention, and HA posture. Populate this section from the first production deployment's observed steady-state usage plus a 50 % headroom margin. Until then:

  • Control-plane: plan 3× (8 vCPU, 16 Gi, 100 Gi) for HA k8s + operator replicas + historian + CNPG with WAL.
  • Edge devices: 20–50× per-unit runtime pods at 100m / 128 Mi requests (200m / 256 Mi limits). Each runs on a dedicated device node with dcs.io/device=<unit-name> label.
  • Historian storage: apply the formula in Storage Sizing with the customer's expected tag count, sample rates, and retention window, then add 30 % headroom for CNPG WAL and TimescaleDB chunk overhead.

Per-Component Resource Requests

These are the shipping Helm defaults from deploy/helm/cloud-native-dcs/values.yaml. They have been sized against the bench and pilot reference setups. Production deployments should override via a values file once observed usage justifies it.

Component CPU request CPU limit Memory request Memory limit Notes
physical-operator 100m 200m 128Mi 256Mi Scales with unit + IOModule count; reconciler is cache-bound
procedural-operator 100m 500m 256Mi 512Mi Highest limits — runs recipe-binding + SFC evaluation
batch-operator 100m 200m 128Mi 256Mi Scales with concurrent running batches
control-operator 100m 200m 128Mi 256Mi Scales with ControlProgram + ControlModule count
dcs-gateway 100m 200m 128Mi 256Mi Scales with UI sessions + REST QPS; HPA optional (2–5 replicas)
unit-runtime (per device) 100m 200m 128Mi 256Mi One pod per Unit; runs on edge device via nodeSelector
historian 250m 500m 256Mi 512Mi Buffer + flush worker; scales with tag write rate
CNPG (historian DB) (operator default) (operator default) (operator default) (operator default) Not set in our chart; inherits CloudNativePG operator defaults. Override in a user values file once profiling justifies it
MQTT broker 50m 100m 64Mi 128Mi Mosquitto single-instance; does not cluster natively
audit-archiver (CronJob) 100m 200m 128Mi 512Mi Runs daily; short-lived pod. Heap is bounded by listPageSize × record_size since pagination landed (#245), so memory does not scale with namespace cardinality
omf-egress 100m 500m 128Mi 512Mi Off by default. Always one replica — ordering is held inside one process. The memory limit has to cover omfEgress.buffer.queueLimit records held during an endpoint outage, so raising that value raises this one
docs 10m 50m 16Mi 32Mi Static nginx — negligible

HA overrides (values-ha.yaml): operators run 2 replicas with minAvailable: 1 PDB, the gateway autoscales 2–5 replicas at 80 % CPU, and MQTT and historian remain single-replica (manual scale). See HA and Failure Modes for the full HA posture.

Throughput Limits

Ceilings below are measured by the test/benchmark/ suite invoked by make bench. A nightly CI job re-measures on ubuntu-latest runners and uploads bench-results.txt as a workflow artifact. See .github/workflows/bench.yml.

Dimension Measured ceiling (reference host) Source Notes
Tag updates/sec per unit runtime (simulation driver) ~6.3 M reads/sec, ~2.0 M writes/sec, ~2.9 M mixed/sec, ~4.3 M parallel reads/sec test/benchmark/driver/ In-process floor. Real-protocol drivers (Modbus, OPC UA) sit below this and are a nice-to-have follow-up
Historian buffer ingest (no flush) ~145–200 k samples/sec single goroutine test/benchmark/historian/ Topic-parse + JSON-unmarshal + buffer append; end-to-end MQTT → CNPG numbers land on the first nightly run with Docker
Concurrent running batches per batch-operator replica ~30 reconciles/sec baseline against envtest (~32 ms/reconcile) test/benchmark/batch/ Envtest is a lower bound (no client-go throttle, no watch-cache cost). The n=10/50/100/500 sweep numbers arrive with the first nightly run
Audit records/sec sustained (per operator) ~31/sec at default 20/30 client-go throttle, ~150/sec serial at QPS=500, ~780/sec under 8 concurrent writers test/benchmark/audit/ Envtest apiserver ceiling. Real-cluster numbers will sit below the high-QPS line because of etcd replication
Gateway REST QPS (in-process chain) ~1200 ops/sec limiter-off, ~770 ops/sec limiter-on (100 RPS read cap + 200 burst); parallel ~5200 ops/sec internal/gateway/gateway_bench_test.go Measures middleware stack + rate limiter; backend handler is a no-op, so the number is the HTTP ceiling and overstates any real-API ceiling

Numbers above are measured on a single-CPU-socket reference host (Intel i7-10510U, 8 hardware threads, WSL2). Nightly CI re-measures on GitHub Actions ubuntu-latest runners. Individual b.ReportMetric columns (ns/op, p50/p95/p99 latency) live in the bench-results.txt artifact.

Storage Sizing

Historian disk

disk = tags × sample_rate_hz × retention_sec × bytes_per_sample × (1 / compression_ratio)

Reference values for TimescaleDB 2.x with native compression on an append-only hypertable of (time, tag_id, value) rows:

Parameter Reference value Notes
bytes_per_sample (uncompressed) ~32 B time (8) + tag_id (4) + float64 value (8) + row overhead (~12)
compression_ratio 8–12× TimescaleDB columnar compression on sequential timestamps
bytes_per_sample (compressed, effective) ~3–4 B derived

Worked example: pilot tier, 500 tags at 0.1 Hz average, 1-year retention:

500 tags × 0.1 Hz × (365 × 86400 s) × 3 B ≈ 4.7 GB

Plus 30 % headroom for WAL, chunk overhead, and uncompressed recent chunks → ~6 GB. Round up to the 10 Gi default, and keep 50 Gi for pilot to allow retention extension.

Worked example: production tier, 5000 tags at 1 Hz average, 2-year retention:

5000 × 1 × (2 × 365 × 86400) × 3 B ≈ 945 GB → plan 1.5 TiB after headroom.

Worst-case growth between prune cycles (demo / dev only)

When historian.prune.enabled: true, worst-case PVC growth between two successful prune runs is bounded by ingest_rate_per_hour × (prune_interval_hours + missed_cycles × prune_interval_hours). The prune CronJob template defaults to tolerateDiskPressure: true and priorityClassName: system-cluster-critical so the Job continues to schedule even on a node carrying node.kubernetes.io/disk-pressure:NoSchedule. Without those, missed cycles compound silently (issue #256, root-caused during a 2026-04-30 demo-instance incident). Sizing the root disk so that two full-window misses still fit gives operators time to act on the DCSHistorianPruneJobFailing alert.

Audit archive

There are two independent sizing decisions: how long records live in etcd (activeRetentionDays) before the archiver moves them, and how long they live in PostgreSQL (archiveRetentionDays) afterwards. The etcd side is the one that has caused outages (see the kine bloat note below), so size it first.

activeRetentionDays — etcd / kine sizing

etcd_audit_bytes = records_per_day × ~1 KB × activeRetentionDays

The 1 KB figure is the per-CR storage cost from the etcd table below (~830 B wire + 20 % BoltDB index/revision overhead).

  • Chart default historian.audit.archival.activeRetentionDays: 90 is calibrated for low-volume clusters (<~1k AuditRecords/day). On a busy pharma site that generates hundreds of records per minute, 90 days will push the apiserver datastore past practical limits. k3s/kine sqlite becomes unresponsive once state.db clears 1 GB, and embedded etcd's --quota-backend-bytes defaults to 2 GB.
  • Reference-cluster data point (2026-04-27 incident): four sim sites generating ~300 AuditRecords/min produced 4,676 records in one namespace over 21 days. At the default 90-day window that namespace alone would have held ~36k records (~36 MB) and the whole cluster ~140k records, past kine's comfortable working set. The reference cluster now runs activeRetentionDays: 1.
  • Sizing rule for production: keep records_per_day × activeRetentionDays × 1 KB under 20 % of your apiserver datastore budget (so e.g. ≤200 MB on a 1 GB k3s/kine, ≤400 MB on a 2 GB etcd). Lower activeRetentionDays first. The records still live in PG via the archiver, and dcs audit verify --archived queries them transparently.

archiveRetentionDays — PostgreSQL sizing

archive_disk = records_per_day × bytes_per_record × archiveRetentionDays
  • bytes_per_record: ~2 KB (AuditRecord serialized as JSON in PostgreSQL, including object diff + electronic signature block).
  • Pharma retention: 21 CFR Part 211 requires 7 years of batch records. The chart default (historian.audit.archival.archiveRetentionDays) is 1095 days (3 years). Override to 2555 days (7 years) for pharma production.

Worked example: pilot tier, ~5000 audit records/day (CRUD on recipes + batches + phase state changes), 7-year retention:

5000 × 2 KB × (7 × 365) ≈ 26 GB.

Initial drain after enabling archival

When historian.audit.archival.enabled flips from false to true on a long-running cluster, the first CronJob run sees the full backlog of records past the cutoff. The archiver is paginated (issue #245) and heap-bounded, but a single run still has to complete its paginated walk within the etcd compaction window (default 5 minutes) before the Continue token expires. At ~50 ms per record (sign + PG insert + optional S3 upload + annotate + delete), the practical ceiling per run is roughly 6,000 records. Plan on multiple daily runs to clear a larger backlog. Already-archived records are skipped on re-run via the audit.dcs.io/archived-to annotation, so the work is idempotent.

etcd

Measured on-wire size per CR (via List + json.Marshal against envtest, test/benchmark/etcd/, n=100 each):

Kind Bytes/CR (wire) +20 % overhead
Site ~520 ~625
Batch ~550 ~660
AuditRecord ~830 ~995

Rule of thumb: ~1 KB per CR including headroom. The wire size excludes etcd's internal revision history and BoltDB index overhead. The 20 % column folds those in. For N CRs of mixed kinds, plan N × 1 KB × <churn factor> before compaction. AuditRecord accumulation is the dominant growth driver over time. Keep the audit-archiver CronJob enabled so it drains records into PostgreSQL (see Known Scaling Limits).

The measurement is a one-shot go test that only runs when invoked explicitly (gated behind testing.Short()). Nightly CI does not re-run it. Flip the gate or run locally to refresh the numbers after a significant CRD change.

When to Scale

Each row points at a specific metric in Monitoring and Metrics. Thresholds are starting points. Tune against observed baseline for each deployment.

Signal Metric Threshold Action
Reconcile queue depth rising workqueue_depth{name="<controller>"} (controller-runtime) > 10 sustained 5 min Add operator replica + enable leader-elect failover (already on)
Reconcile error rate elevated rate(dcs_reconcile_total{result="error"}[5m]) > 0.1 /s Investigate before scaling — usually a bug, and capacity is the wrong lever
Historian write latency histogram_quantile(0.99, rate(dcs_historian_flush_duration_seconds_bucket[5m])) > 100ms p99 Scale CNPG instances or shard tags across multiple historians
Historian buffer backlog dcs_historian_buffer_size > 5000 sustained 2 min Flush interval too long or DB slow — check CNPG IOPS first
Gateway API latency histogram_quantile(0.99, rate(dcs_gateway_request_duration_seconds_bucket[5m])) > 500ms p99 Enable autoscaling (gateway.autoscaling.enabled=true) or raise maxReplicas
MQTT broker saturation rate(dcs_runtime_mqtt_publishes_total{outcome="published"}[1m]) + broker-side $SYS/broker/messages/inflight Inflight > 10k Shard topics across broker instances (requires bridge config)
Runtime disconnected dcs_runtime_driver_connected == 0 Any, > 1 min Not a capacity signal — investigate device / network

Known Scaling Limits

These are architectural ceilings, and exceeding one takes a code change. More replicas move none of them.

  • MQTT does not cluster natively. Mosquitto is single-instance. Multi-replica deployments require manual bridge configuration via a ConfigMap overlay (the chart has no values key for bridge config). No built-in sharding.
  • Gateway anti-replay nonce store is in-memory and per-replica. With multiple gateway replicas + rate-limited clients hitting different pods via a load balancer, nonce reuse across replicas is not detected. Noted in values.yaml (gateway.antiReplay). Fix requires a shared store (Redis / etcd), tracked separately.
  • Historian is single-replica. historian.replicas=1 is the default and HA values do not override it. Scaling writes requires either sharding tags across multiple historian instances (manual) or a queue-based ingest redesign.
  • Gateway client-go QPS throttle. Default rest.Config is 20 QPS / 30 burst. Audit-heavy endpoints (mass AuditRecord creation, bulk batch queries) saturate this before any component CPU limit. Override via gateway flag when profiling identifies the bottleneck.
  • etcd as audit-record store. AuditRecord CRs accumulate in etcd until the audit-archiver CronJob moves them to PostgreSQL. The chart default activeRetentionDays: 90 assumes low-volume traffic. On a busy pharma site (hundreds of records/min) it will saturate kine sqlite or hit etcd's quota inside a week. Size activeRetentionDays against your record rate per the activeRetentionDays formula above, and always enable historian.audit.archival for long-running deployments. The archiver is paginated and memory-bounded (issue #245). The limit is how fast records can be drained per CronJob run before the etcd Continue token expires.
  • Scheduler audit records. The gateway archive-integrity scheduler (gateway.archiveIntegrity.interval, default 6h) writes one AuditRecord per run. At the default cadence this is 4 records/day, which is negligible against batch-generated audit traffic. The scheduler load itself scales with the total number of manifest rows (one query per interval). For archives with >100k manifests, raise the interval to 12–24h.
  • S3 Object-Lock mirror egress. When historian.audit.archival.immutable.enabled is true, each archival run writes one JSON object per batch to the bucket. Typical object size is tens of KB (manifest metadata plus canonical records), so even a daily archiver run with 100 batches produces a few MB of egress. Bucket size grows linearly with retention. At 7-year retention and ~10 batches/day, budget a few GB of locked storage with Compliance-mode lifecycle → Glacier-Deep-Archive transitions to keep cost bounded (objects cannot be deleted before expiry, so transition tiers are the only cost lever).