Historian Disk-Pressure Runbook¶

What to do when the historian PostgreSQL PVC fills the node disk and kubelet starts evicting pods. This is a cluster-wide outage: once the node hits DiskPressure, ingress-nginx, cert-manager, and every other workload on the affected node will be killed too.

Storage-provisioner framing

Parts of this runbook were written against a k3s cluster using the local-path provisioner, where the historian PVC shares the node root disk. The reference platform is now Talos. On any cluster with a quota-enforcing CSI driver the failure contains to the PVC with no node-wide cascade, but the diagnosis and remediation steps still apply.

Symptoms¶

https://<hostname>/ returns TCP RST (port 80/443 Connection refused), not a timeout. DNS resolves, the droplet pings, but nothing is listening.
kubectl get pods -A shows dozens of pods in Evicted, ContainerStatusUnknown, or Error (especially ingress-nginx-controller-*).
kubectl get nodes shows the node as Ready but with DiskPressure=True in .status.conditions.
df -h / on the node reports >90% used, usually climbing.
du -h -d 1 /var/lib/rancher/k3s/storage/ shows the …_historian-db-* directory consuming most of the disk (the local-path provisioner does not enforce the PVC spec.resources.requests.storage size, which is advisory only).

Why this happens¶

TimescaleDB partitions hypertables into chunks of chunk_time_interval (default 7 days). The retention policy registered by pkg/historian/pg.go:SetRetention drops chunks whose entire time range is older than the retention window. Concretely:

retention	chunk_time_interval	worst-case retained data
7 days	7 days	up to 14 days
7 days	1 hour	up to 7 d 1 h
30 days	7 days	up to 37 days

At reference-cluster ingest rates (~15 GB/day of tag_values) a 14-day window is ~210 GB, far more than a 77 GB droplet root disk can hold. Production sizing must account for the full retention + chunk_time_interval sum. Sizing from retention alone is how the table above overruns.

The first-chunk gotcha (fresh installs and recovered PVCs)¶

drop_chunks(older_than => INTERVAL '<window>') drops a chunk only when the chunk's range_end is older than the cutoff. A brand-new chunk's range_end is in the future (specifically, range_start + chunk_time_interval). On a freshly initialized hypertable nothing is therefore droppable for the first chunk_time_interval of ingest, no matter how short the retention window is set.

The 2026-04-30 incident (#253) was this exact failure mode: a recovered PVC had its hypertables created with TimescaleDB's 7-day default chunk_time_interval, the prune CronJob ran with INTERVAL '6 hours' retention every 6 h, and the SQL completed cleanly each cycle while dropping zero chunks every time. The single first chunk grew unbounded for 3 days (58 GB on a 77 GB disk) until kubelet started evicting.

set_chunk_time_interval() only affects chunks created after it runs. It cannot retroactively shrink the first chunk. The durable fix is to pass chunk_time_interval to create_hypertable() in the schema migration that provisions the table. That is what pkg/historian/migrations/*.sql do since #253. If you operate a historian outside this codebase, the equivalent is:

SELECT create_hypertable('tag_values', 'ts',
    chunk_time_interval => INTERVAL '1 hour',
    if_not_exists       => TRUE);

The prune CronJob's set_chunk_time_interval is a backstop for old DBs that predate the fix. It does not protect a new DB during its first chunk_time_interval.

Immediate diagnosis¶

# From the node
df -h /
du -h -d 2 /var/lib/rancher/k3s/storage/ | sort -rh | head
du -h -d 1 /var/lib/kubelet/pods | sort -rh | head

# From kubectl
kubectl get nodes -o json | jq '.items[].status.conditions[]|select(.type=="DiskPressure")'
kubectl get pods -A --field-selector=status.phase=Failed | wc -l
kubectl -n dcs-system get cluster,pvc,pods
kubectl -n dcs-system exec -c postgres <cluster>-1 -- \
  psql -U postgres -d historian -c \
  "SELECT hypertable_name, time_interval FROM timescaledb_information.dimensions;"
kubectl -n dcs-system exec -c postgres <cluster>-1 -- \
  psql -U postgres -d historian -c \
  "SELECT application_name, config FROM timescaledb_information.jobs
   WHERE proc_name='policy_retention';"

# Per-chunk inventory — surfaces the first-chunk gotcha. If the oldest
# chunk's range_end is in the future, drop_chunks can't drop it yet.
# chunks_detailed_size is a FUNCTION taking one hypertable, not a view:
# joining it bare fails with `relation "chunks_detailed_size" does not
# exist`. Run it per hypertable.
kubectl -n dcs-system exec -c postgres <cluster>-1 -- \
  psql -U postgres -d historian -c \
  "SELECT hypertable_name, range_start, range_end,
          pg_size_pretty(total_bytes) AS size
   FROM timescaledb_information.chunks
   JOIN chunks_detailed_size('tag_values') USING (chunk_schema, chunk_name)
   ORDER BY range_start;"

# Chunk counts and time span per hypertable — the quick read on whether a
# prune has anything to drop.
kubectl -n dcs-system exec -c postgres <cluster>-1 -- \
  psql -U postgres -d historian -c \
  "SELECT hypertable_name, count(*) AS chunks,
          min(range_start) AS oldest, max(range_end) AS newest
   FROM timescaledb_information.chunks GROUP BY 1 ORDER BY 1;"

# The same per-chunk inventory across EVERY hypertable, biggest chunk
# first. The historian has five (tag_values, alarm_events,
# state_transitions, audit_records, cm_health_events), so the single-table
# form above answers for tag_values alone. Reach for this one when the
# aggregate says the space went somewhere else. The lateral call is what
# supplies chunks_detailed_size its one-hypertable argument per row; a
# hypertable with no chunks yet contributes no rows rather than failing.
kubectl -n dcs-system exec -c postgres <cluster>-1 -- \
  psql -U postgres -d historian -c \
  "SELECT c.hypertable_name, c.range_start, c.range_end,
          pg_size_pretty(d.total_bytes) AS size
   FROM timescaledb_information.hypertables h
   CROSS JOIN LATERAL chunks_detailed_size(
          format('%I.%I', h.hypertable_schema, h.hypertable_name)::regclass) d
   JOIN timescaledb_information.chunks c
     ON c.chunk_schema = d.chunk_schema AND c.chunk_name = d.chunk_name
   ORDER BY d.total_bytes DESC LIMIT 20;"

kubectl get cronjob shows the historian prune CronJob on its schedule; the per-chunk inventory query lists the oldest tag_values chunks with their time ranges and sizes, a week back; an aggregate query gives chunks per hypertable with the oldest and newest boundary; kubectl create job --from=cronjob runs the prune on demand and the Job log shows set_chunk_time_interval on each hypertable followed by drop_chunks naming every chunk it dropped; the aggregate query re-run shows fewer chunks and the oldest boundary moved forward into the retention window — The inventory query above, the prune CronJob kicked on demand, and the same inventory afterwards: fewer chunks and an oldest boundary inside the retention window. Recorded on a healthy plant. The node is not under `DiskPressure` and no eviction cascade is running.

Remediation — production (data-preserving)¶

Use this path when the historian holds real BatchRecord-linked time-series that cannot be lost. All steps are non-destructive to committed data.

Self-sealing cascade warning. On a disk-pressured node, kubelet will evict BestEffort pods including the helper-pod-delete-pvc-* that local-path-provisioner uses to rm -rf a released PV directory. That means you cannot rely on the Delete reclaim policy alone to free disk — the helper pod will itself be evicted before it can run. You must free a small chunk of disk by other means first (below), then the Kubernetes-native cleanup paths will work.

Free a small amount of disk so kubelet exits DiskPressure.
docker image prune -a / crictl rmi --prune on the node — containerd image cache is typically recoverable and 2–8 GB.
Truncate the largest journals: journalctl --vacuum-size=200M.
Delete orphan Evicted pods: kubectl delete pods -A --field-selector=status.phase=Failed.
After freeing disk, remove the stale taint. Kubelet does not always auto-remove node.kubernetes.io/disk-pressure:NoSchedule after manual cleanup on the host. Run kubectl taint node <node> node.kubernetes.io/disk-pressure:NoSchedule-. Kubelet re-adds it if pressure actually returns.
Restore the write path. Once the node is out of disk pressure, ingress-nginx and cert-manager will self-heal. Verify with curl -sk https://localhost/healthz on the node. If you use Flux, also verify the CNPG operator webhook has endpoints before Flux retries:
```
kubectl -n cnpg-system get endpoints cnpg-webhook-service
# Must show at least one IP; empty endpoints → any Cluster/Backup/Pooler
# admission call fails → Flux HelmRelease loops in RollbackFailed.
```
Restore from backup into a new, larger PVC (the correct fix when the disk is fundamentally undersized):
Provision a larger volume on the underlying IaaS (or a new StorageClass).
Use CNPG's recovery: bootstrap pointing at the Barman S3 archive: kubectl apply -f a new Cluster with spec.bootstrap.recovery and spec.externalClusters referring to the previous cluster's WAL archive. See backup-recovery.md.
Flip the historian Deployment's DATABASE_URL to the new cluster. Rotate the old Cluster out.

Shrink in place (last resort, only if backup restore isn't viable). Drop the oldest chunks until the filesystem fits. This is a data loss operation, narrower than a full wipe:

SELECT drop_chunks('tag_values',        older_than => NOW() - INTERVAL '30 days');
SELECT drop_chunks('alarm_events',      older_than => NOW() - INTERVAL '30 days');
SELECT drop_chunks('state_transitions', older_than => NOW() - INTERVAL '90 days');
SELECT drop_chunks('cm_health_events',  older_than => NOW() - INTERVAL '30 days');
-- audit_records: NEVER drop; 21 CFR Part 11 requires retention
VACUUM (VERBOSE, FULL) tag_values;  -- FULL rewrites + returns space to OS

VACUUM FULL takes an exclusive lock and rewrites the table. Expect writes to block for its duration.

Adjust chunk_time_interval going forward so the next retention run actually drops chunks:

SELECT set_chunk_time_interval('tag_values',        INTERVAL '6 hours');
SELECT set_chunk_time_interval('alarm_events',      INTERVAL '6 hours');
SELECT set_chunk_time_interval('state_transitions', INTERVAL '6 hours');
SELECT set_chunk_time_interval('cm_health_events',  INTERVAL '6 hours');

Existing chunks keep their old size. Only new chunks use the new interval.

Never drop audit_records chunks. 21 CFR Part 211 mandates 3-year retention. The audit-archiver CronJob handles compaction (see compliance/21cfr11.md).

Remediation — disposable-data clusters¶

A simulation cluster runs disposable data, so wiping is acceptable (these are the steps used in anger on our demo instance on 2026-04-17):

kubectl -n dcs-system scale deploy cloud-native-dcs-historian --replicas=0
kubectl -n dcs-system delete cluster cloud-native-dcs-historian-db
kubectl -n dcs-system delete pod --force --grace-period=0 cloud-native-dcs-historian-db-1
rm -rf /var/lib/rancher/k3s/storage/*_historian-db-*   # on the node
# Re-apply the Cluster CR (Flux's Helm controller won't notice drift on deletion):
helm get manifest cloud-native-dcs -n dcs-system \
  | awk '/^---/{...}/kind: Cluster$/{...}' \
  | kubectl apply -f -
kubectl -n dcs-system rollout restart deploy cloud-native-dcs-historian

Prevention¶

1. Sizing (production)¶

Size the historian PVC for retention_days + chunk_time_interval_days worth of peak ingest, plus 50% headroom for WAL and index bloat. Example for a plant emitting 100 Hz aggregate across 200 tags (= 20k rows/sec, ~3 B/row effective with TimescaleDB compression, derivation in capacity-planning):

20000 rows/s × 86400 s × 3 B ≈ 5.2 GB/day
30 d retention × 5.2 GB + 1 chunk (7 d × 5.2 GB) + 50% headroom ≈ 290 GB

See capacity-planning.md for the full formula.

2. Set `chunk_time_interval` at hypertable creation¶

The migrations in pkg/historian/migrations/*.sql already pass chunk_time_interval => INTERVAL '1 hour' to every create_hypertable() call, so any historian DB initialized by this codebase from #253 onward gets correctly sized first chunks. Do not weaken this. See "the first-chunk gotcha" above for why a smaller value at creation matters more than the prune CronJob's set_chunk_time_interval, which only affects future chunks and leaves the first one at its creation size.

For environments running a forked historian, custom schema migrations, or a DB that predates #253: run set_chunk_time_interval once on each hypertable AND truncate or drop the first oversized chunk if it has not yet aged past its range_end. Otherwise the disk will fill within chunk_time_interval of go-live.

For long-retention production deployments (months or years), 1 h chunks generate too much chunk metadata. Pick a value that satisfies chunk_time_interval << retention while keeping chunk count under a few thousand per hypertable.

3. Enable the historian prune CronJob (demo / dev only)¶

# values.yaml
historian:
  prune:
    enabled: true
    schedule: "17 */6 * * *"    # every 6 h, offset from the top of hour
    retentionHours: 24
    chunkIntervalHours: 1
    vacuum: true
    # Schedule + survive on a node carrying
    # node.kubernetes.io/disk-pressure:NoSchedule. Without these the
    # very Job that frees disk gets blocked from running exactly when
    # it's needed (issue #256, third recurrence on the reference cluster).
    tolerateDiskPressure: true
    priorityClassName: "system-cluster-critical"

The CronJob (defined in deploy/helm/cloud-native-dcs/templates/historian-prune-cronjob.yaml) resizes the chunk interval, calls drop_chunks() with the configured cutoff, and runs non-blocking VACUUM to return pages to the OS. It prunes tag_values, alarm_events, state_transitions, and cm_health_events. It never prunes audit_records. In production, leave prune.enabled: false and use native retention with adequate PVC sizing plus CNPG backups.

The toleration + priority defaults above are safe in any cluster: the Job uses tiny resources, and drop_chunks() only frees disk on the historian PV. It never grows it. If your cluster's PriorityClass admission policy restricts system-cluster-critical to kube-system, set historian.prune.priorityClassName: "" and rely on the toleration alone.

The CronJob wraps psql in a pg_isready retry loop. k3s + kube-router can take a few seconds to install NetworkPolicy rules on a just-started pod's CNI interface. Without the wait, fast-starting psql hits the window where egress packets are RST'd and fails with Connection refused on the first attempt. If you see that in Job logs, the retry loop absorbs it. If it persists past ~30 s, the database is actually unreachable.

4. Monitoring alerts¶

Add these Prometheus alerts (via the monitoring stack):

- alert: NodeDiskUsageHigh
  # Node filesystem, NOT PVC usage. local-path PVC metrics are useless here:
  # the claim is 5Gi but the directory will silently grow past the filesystem
  # root, so kubelet_volume_stats_used_bytes never reflects reality.
  expr: (node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_avail_bytes{mountpoint="/"})
        / node_filesystem_size_bytes{mountpoint="/"} > 0.75
  for: 15m
  severity: warning

- alert: HistorianPruneJobFailing
  # Alert on two consecutive failures; one-off transient DB unavailability is
  # expected during rolling operator updates.
  expr: max_over_time(kube_job_status_failed{job_name=~"cloud-native-dcs-historian-prune-.*"}[12h]) > 1
  for: 5m
  severity: critical
  labels: {component: historian}

- alert: CNPGRetentionJobStale
  expr: time() - timescaledb_job_last_run_seconds{proc_name="policy_retention"} > 86400 * 2
  severity: warning

- alert: CNPGWebhookDown
  # If the webhook has no endpoints, every Cluster/Backup/Pooler admission
  # call fails and Flux can't reconcile. Catches the self-sealing cascade.
  expr: kube_endpoint_address_available{namespace="cnpg-system",endpoint="cnpg-webhook-service"} == 0
  for: 2m
  severity: critical

Disk-pressure eviction is a cluster-level outage. Disk-usage warnings give you the 15-minute head start needed to cut retention, expand the PVC, or force a backup restore onto a larger volume before kubelet begins evicting.

Do not trust kubelet_volume_stats_used_bytes / capacity with the local-path StorageClass. The capacity in that metric is the PVC claim (e.g. 5Gi), but the directory can grow to fill the underlying filesystem. The ratio therefore stays well under 1.0 right up to the moment the node hits DiskPressure. The node-level node_filesystem_* metric above is the authoritative signal on single-node clusters.

Historian — architecture and configuration
Backup and Recovery — CNPG Barman S3 archival
Capacity Planning — sizing math

This runbook codifies the lessons of three April 2026 demo-instance incidents: the prune CronJob (2026-04-17/22), plus the chunk_time_interval schema fix and the prune-job DiskPressure toleration that came out of the 2026-04-30 recurrence (#253, #256). The instance-specific incident records live in the internal cndcs-operations repo.