Skip to content

ADR 0023: Live-feed freshness and cluster health are separate indications

Status: Accepted Date: 2026-07-18 Issue: #963

Context

The HMI top bar carried two indicators side by side, each an 8px dot with a muted label: a connection indicator (Connected / Disconnected / Reconnecting… / No MQTT, client-side WebSocket state, HMI-only, from #88 and #522) and a health indicator (Healthy / Degraded / Critical, a server-side rollup over control-plane workloads and unit runtimes, shared with /system and /data).

They answer genuinely different questions:

  • Feed freshness — are the numbers on this screen current? Scoped to one browser session. A laptop on failing Wi-Fi has a stale screen while the plant is perfectly healthy.
  • Cluster health — is the plant's control software running? Scoped to the cluster. Operators keep seeing live values from a degraded cluster right up until the degraded part is the part they need.

Both directions of disagreement are real and both matter. But the presentation gave no way to tell them apart:

  1. Identical visual grammar — same dot size, same green, same muted label, adjacent. They read as two facets of one indicator.
  2. Neither label named its subject. "Connected" and "Healthy" are predicates with the noun omitted; "connected to what" had no answer on screen.
  3. The split was already leaky. No MQTT is a gateway↔broker fault — a server-side condition — displayed on the ostensibly client-side dot, and it did not appear in the server-side health rollup at all.
  4. Both consumed permanent top-bar space to say "normal" ~all of the time.

A reviewer who knows the product could not explain the difference. No ADR or doc framed the two-indicator presentation as a decision; the rationale existed only implicitly in #88 and #522.

Decision

The two indications stay distinct, but only one of them is a status light.

Cluster health keeps the dot. The .health-indicator pill keeps the dot-and-colour vocabulary, unchanged, across all three apps. It is the single coloured status light in the top bar.

Feed freshness becomes a freshness readout, not a status light. The HMI top bar shows Live · 2s ago — muted text, no dot, no colour — driven by HMIState.lastMessageAt, a local receive clock stamped in HMIConnection.ws.onmessage.

The clock choice is load-bearing. tagValues[].ts carries the broker/source timestamp and drives per-tag staleness (HMIBinding._checkStale, 10s threshold, .stale dimming). That is the right clock for "has this device stopped publishing" but the wrong one for "is my feed alive": it is subject to broker/browser clock skew, and a single chatty tag would mask a feed that had otherwise gone silent. Feed liveness must be measured on the receiving end.

Showing an age rather than a binary is a deliberately stronger claim than the old dot made. A WebSocket can sit open and silent — Connected would stay green over a frozen screen, which is the exact failure an operator most needs to catch. A ticking age cannot lie that way, so the readout keeps ticking on a 1s timer while connected.

Feed faults escalate to a banner, not a chip. Disconnected, reconnecting, and no-broker states render a full-width .hmi-feed-banner fixed under the top bar, on every view, stating explicitly that on-screen values are last-known rather than current. A dead feed invalidates every number on the screen; that is not an 8px dot's worth of news.

Normal state costs no chrome. No banner while healthy, no coloured dot on the readout. This follows the ISA-101 / Hollifield convention: indicate by exception, and indicate data quality on the data.

Broker-link faults register in cluster health too. annotateBrokerLink (internal/gateway/diagnostics.go) downgrades the Message Bus service to degraded when the broker pods are running but Hub.MQTTConnected() is false. The pod-level check cannot see a severed link — network policy, credential rotation, expired TLS — so before this the rollup called the cluster healthy while every process display sat frozen. Degraded rather than offline: the control plane keeps executing batches and the gateway keeps serving REST; it is live data that is lost, not control.

Alternatives Considered

Relabel both pills in place (Live data: Connected / System: Healthy) with explanatory tooltips. Cheapest, and it fixes the missing-noun problem. Rejected because it keeps two status lights competing in the corner, keeps spending permanent chrome on the normal case, and preserves the binary Connected that cannot detect a silent socket.

Merge into one indicator showing the worst of the two, with a popover breaking out feed vs cluster. Minimal chrome, but it collapses precisely the distinction this issue exists to make legible: "my laptop's Wi-Fi is flaky" and "the batch operator is crashlooping" would light the same lamp, and an operator would have to click to find out which.

Move the readout into the process view, next to the data it qualifies — conceptually the strongest placement. Rejected on structure: the HMI has no per-view content header (the breadcrumb lives in the shared top bar), and the only content-area strip, #hmiSummaryBar, is process-view-only and hidden at unit and control-module level. Building one that existed purely to host this readout would have been a larger change than the problem warranted. The banner covers the case where placement actually matters — when the feed is broken.

Consequences

  • The top bar has exactly one coloured status dot, whose subject is unambiguous, and one plain-text freshness readout. They no longer look like the same kind of thing.
  • Steady-state chrome shrinks; the alarm and prompt badges get the attention they should have had.
  • A frozen-but-connected feed is now visible, which the previous design could not express.
  • A severed broker link now shows up in /system#/diagnostics and in GET /api/v1/system/health, where an engineer actually looks for it.
  • #hmiConnIndicator and the .hmi-conn-* classes are gone. Consumers were updated: test/ui/smoke.spec.ts, hack/marketing-hero/capture.mjs, and the Beat 5 selector in docs/videos/hmi-operator-tour.clip.yaml. That clip's narration still frames the beat as a connection heartbeat and is queued for a marketing re-cut.
  • Per-tag staleness is unchanged by this ADR, and its own gap — the sweep only runs on the process view, so alarm and batch views never dim — is tracked separately as #964.