Troubleshooting

Common issues and their solutions for Cloud-Native DCS.

Administrator sections

Sections marked (Administrator) require cluster access and are intended for system administrators. End users can skip them.

Operator Issues (Administrator)

CrashLoopBackOff

Symptoms: Operator pod restarts repeatedly.

Common causes: - Missing CRDs: Upgrade the Helm chart to install CRDs - Invalid kubeconfig: Check RBAC permissions for the operator service account - Database connection failure (historian): Verify DATABASE_URL is correct and PostgreSQL is reachable

OOMKilled

Symptoms: Operator pod killed with OOMKilled reason, restarts.

Common causes: - Metrics cardinality leak: Too many unique label combinations in Prometheus metrics - Large number of resources: Increase memory limits in Helm values

Fix: Ensure you are running the latest version.

Reconciliation Errors

Symptoms: Resources stuck in unexpected state, reconcile error logs.

Check the Diagnostics page in the gateway UI for reconcile error rates:

Diagnosing a degraded service from the UI

When a Core Service card shows a degraded or offline state, click Diagnose on the card. The card expands into a triage panel with three tabs. The rest of the page stays visible and keeps polling behind it, so the other services are still readable while you work. Escape or Close collapses the panel.

Optional components (Historian, Audit Archiver) that were never installed show a neutral not-installed state instead. That is a supported minimal-install configuration: it raises no active issue and does not affect the overall health verdict. A previously installed component whose workload still exists but has no pods reports offline as before.

The Failure detail tab summarises what we know about the most recent crash: pod failure reason, last exit code, summary message, and (for CronJobs) the last successful run timestamp.

The Events tab surfaces recent Kubernetes events for the component's pods, jobs, and CronJob: readiness probe failures, image pull errors, scheduler decisions. It is the same data that dcs health --events <component> returns, useful when the failure summary alone does not tell you why.

The Logs tab tails the most relevant pod's stdout. For a chronically- restarting Deployment, toggle Previous container to read the prior crash. Output is capped at 1 MiB and 2000 lines. For the full log stream fall back to dcs health --logs <component> --tail 2000 or kubectl logs.

Every component logs in the same format at the same verbosity, which is the console encoder at info (ADR 0063). When a fault needs more detail than the default carries, raise the whole release with logging.level: debug. The product's V(1) lines are per-request and per-scan detail, so expect the 2000-line window to cover a much shorter span while it is on.

logging.encoder and logging.stacktraceLevel sit beside it. See Security Operations § Log format for what each accepts and when a collector wants JSON.

For Deployment-backed services (everything except the audit-archiver CronJob) the Restart service button on the Failure tab triggers a rollout that honours maxSurge / maxUnavailable. Restarting the gateway itself is blocked at the API. Use kubectl rollout restart for that case.

A restart is only the fix when the fault lives in the pod. When the evidence points below the service (the Historian's Events tab showing readiness probe failures and its Logs tab full of failed to flush tags … database=historian errors), the component is a casualty of the real cause, and restarting it would change nothing. Fix the dependency instead and let the card recover on its own. The walkthrough below triages exactly that case: a Historian degraded by a database outage, diagnosed through the three tabs, then recovered with kubectl against the database while the Restart button stays untouched.

Database-outage triage end to end: the degraded card, the three-tab evidence walk pointing below the service, and a kubectl recovery of the database, with Restart service in frame the whole time and truthfully never the fix.

Runtime Issues

Runtime Pod Not Starting

Symptoms: Unit shows RuntimeReady: false, no runtime pod exists.

Common causes: - Missing node with matching nodeSelector label - Node not ready - Image pull failure

Diagnosis:

dcs get units -s lab

Check the Unit detail in the gateway UI for RuntimeReady condition status.

Runtime Cannot Connect to I/O Module

Symptoms: Runtime pod runs but I/O reads return errors.

Common causes: - Network unreachable: Check that the runtime pod can reach the I/O module IP - Wrong protocol/port: Verify IOModule CR has correct protocol and address - Firewall: Ensure EtherNet/IP (TCP 44818) or Modbus (TCP 502) is allowed

Diagnosis:

Check the Diagnostics page in the gateway UI for runtime connectivity status. The I/O module health is also shown on the Equipment page.

Runtime OPC UA Connection Failure

Symptoms: OPC UA driver fails to connect.

Common causes: - Incorrect endpoint URL in IOModule CR - Security policy mismatch (the client supports None, Basic256Sha256) - Certificate not trusted by the OPC UA server

Diagnosis:

Check the Diagnostics page for OPC UA connection errors. The IOModule status in the gateway UI shows connectivity health.

State Machine Issues

Resource Stuck in Transitional State

Symptoms: A Unit, Phase, or Batch is stuck in Starting, Stopping, Holding, Restarting, or Aborting.

Explanation: Transitional states (ending in "-ing") require a StateComplete signal to advance. The controller that owns the resource must emit this signal when the physical action completes.

Diagnosis:

dcs get phases -s lab

Fix: Check the controller logs for the owning operator. If the phase controller is waiting for a runtime response, verify the runtime is running and reachable.

Command Annotation Not Processing

Symptoms: Setting dcs.io/command=Start has no effect.

Common causes: - Resource is not in a valid state for the command (e.g., Start is only valid from Idle) - The owning operator is not running - RBAC: The operator lacks permission to update the resource

Diagnosis:

dcs get unit reactor-1 -s lab

See the Architecture section for the valid state/command matrix.

Batch Issues

Batch Stuck in Allocating

Symptoms: Batch stays in Allocating phase.

Common causes: - No available Units matching the recipe's unit requirements - Units are already allocated to other batches - Units are not in Idle state

Diagnosis:

dcs get batch <name> -s lab
dcs get units -s lab

Batch Failed

Symptoms: Batch transitions to Failed state.

Diagnosis:

dcs get batch <name> -s lab
dcs get phases -s lab

Check the Batch detail page in the gateway UI for events and the procedural tree for the failing phase.

MQTT Issues (Administrator)

MQTT Connection Failure

Symptoms: Components log MQTT connection errors.

Common causes: - Broker unreachable: Check broker status in the Diagnostics page - Authentication failure: Verify MQTT credentials in Helm values - TLS mismatch: Verify CA/cert/key configuration

Historian Not Receiving Data

Symptoms: Historian REST API returns empty results.

Common causes: - MQTT topics mismatch: Verify runtimes are publishing to expected topics - Database connection issue - Buffer not flushing: Check dcs_historian_buffer_size metric in Monitoring

Historian Database Volume Filling, No Recent Backup in the Store

Symptoms: the CNPG data volume grows steadily with no matching growth in the tables, pg_wal holds far more segments than the retention settings suggest, and the object store has no base backup since the day backups were turned on. The database itself keeps serving reads and writes until the volume is full, at which point the node goes into disk pressure and evicts whatever else is scheduled on it.

Cause: WAL archiving is failing. PostgreSQL does not drop a segment it has not archived, so a store it cannot reach converts into disk usage. No watched error fires. On a cluster whose CNI enforces NetworkPolicy, the usual reason is that the database pods' own policy carries no rule for the store. Their policyTypes is [Ingress, Egress], so they reach exactly what the policy names. Wrong credentials, a wrong bucket and a firewall in the way all present the same way.

Fix: name the destination in the chart. See historian.backup.s3.egress for both forms: CIDRs for a store outside the cluster, pod labels for one served inside it. The port is taken from historian.backup.s3.endpointURL, so a store on :9000 is allowed on 9000. Before #1516 the rule hard-coded 443. On a chart rendered with networkPolicies.enabled this value is required, so a deployment that reaches this state from a policy denial is one whose values predate that requirement.

Confirm the repair against the store, by listing the bucket for a base backup newer than the change. The chart rendering is not evidence the packets arrive.

Audit Archival Issues (Administrator)

Audit Archiver Run Failed on a Fresh Cluster

Symptoms: the Audit Archiver card reads degraded shortly after a first install, its Logs tab ends in failed to connect to database … connection refused, and the overall system-health verdict stays degraded until the next scheduled run.

Cause: nothing orders the archival CronJob after the historian database. On a fresh bring-up the first scheduled run can fire while the CNPG cluster is still initialising, so the archiver has nothing to connect to through no fault of its own. Which installs hit this is decided by where the wall clock falls relative to the database's boot, because the schedule is a fixed cron expression.

Fix: each run now waits historian.audit.archival.databaseWait (15 minutes by default) for the database to accept connections before it fails. Raise it if your storage class makes a first boot slower than that, keeping four times the value inside one schedule period. The wait is spent per pod, so a database that is down for good is waited on once per backoffLimit attempt.

# values.yaml
historian:
  audit:
    archival:
      databaseWait: 30m

A cluster already carrying a failed run does not have to wait for the next scheduled one. Trigger a run by hand once the database is ready, and the verdict returns to healthy as soon as it completes:

dcs health --run audit-archiver

A run that still fails with the database up is a real failure. Read its Logs tab for the reason. A missing dcs-signing-key Secret and a misconfigured S3 mirror both fail the run deliberately. Archiving unsigned or unmirrored records is what the failure prevents.

Audit Archiver Reports the Mirror Endpoint Unreachable

Symptoms: the Audit Archiver card reads degraded on a deployment with the Object Lock mirror enabled, and every run's Logs tab ends in audit mirror endpoint is unreachable — refusing to archive. The historian and the rest of the archive chain look fine, because they are.

Cause: the pod cannot open a connection to the S3 endpoint. On a cluster whose CNI enforces NetworkPolicy, the usual reason is that the archiver's own policy carries no rule for the endpoint. Its policyTypes is [Egress], so it reaches exactly what the policy names. Firewall rules and endpoint outages produce the same message, and so does an endpoint spelled wrong.

Fix: name the destination in the chart. See historian.audit.archival.immutable.egress for both forms: CIDRs for a host outside the cluster, pod labels for a bucket served inside it. On a chart rendered with networkPolicies.enabled this value is required, so a deployment that reaches this message from a policy denial is one whose values predate that requirement.

The check runs at startup and fails the run, which is deliberate: the run would otherwise report a clean no-op on every hour that has nothing past the retention cutoff, and a fresh deployment has 90 days of those. A refusal from the endpoint is not a failure here. The documented bucket policy grants the archiver s3:PutObject alone, so a 403 counts as the endpoint answering.

Diagnostics Gateway Page

The gateway web UI's System app includes a Diagnostics page (◉ icon) that shows: - Operator health status - Runtime connectivity - Recent alarm summary - Reconciliation error rates

Access it at https://<gateway-host>/system#diagnostics.

Related Documentation

• Architecture -- System overview and state machine
• Monitoring and Metrics -- Prometheus metrics for diagnosing issues
• Alarm Management -- Auto-generated alarms for equipment faults
• MQTT Telemetry -- MQTT broker and topic configuration