Skip to content

ADR 0084: A live stream holds the newest observation

Status: Accepted Date: 2026-08-24 Issue: #1817 Related: ADR 0083 (the historian half, and where the event-time rules are written down), ADR 0025 (what per-tag staleness means and which of its two failures is worse), ADR 0027 (what a tag timestamp means)

Context

ADR 0083 removed mqtt.Message.Timestamp, a field no publisher ever set, and made the historian read the event time off each payload instead. Two other consumers read that same field, found it zero, and fell back to their own arrival time. Both went on doing so, and #1812 left them doing it in the open: the dead IsZero() branch is gone, and the call to time.Now() is what is written.

The gateway TagBus

internal/gateway/tagbus.go broadcasts WSMessage.Timestamp on four topics, and the HMI subtracts it from now to decide per-tag staleness (HMIBinding.staleAgeFor). Substituting the gateway's receive time means what the HMI judges is the age of the frame it just got. It is not the age of the observation.

That is the failure ADR 0025 names as the more dangerous one. A false "stale" is annoying, and a false "fresh" hides dead data. A tag whose runtime stopped observing it half an hour ago rendered fresh for as long as anything kept republishing it, and ADR 0025 calls dimming a current value a cry-wolf failure precisely because an operator who learns to distrust the cue distrusts it when it is right.

The event time was on the wire the whole time, on the same key the historian now reads. driver.TagValue.Timestamp carries it for a tag value, timestamp carries it on the alarm and state payloads, and since carries it on CM health.

The question the historian did not have to answer

A store-and-forward replay reaches the gateway too. The bench measured one on 2026-08-24 while verifying #1797: a promoted runtime announced a queue of 8,781 messages and drained it over about a hundred seconds. Every one of those is a genuine observation, and the historian wants all of them.

Handing them to a live HMI in arrival order is a different matter. The mimic would animate the last two hours of the plant at replay speed, on a surface whose whole job is answering what the plant is doing now. A replayed value presented as current is a worse outcome than a stale timestamp, because the HMI is a control surface and the historian is a record.

msg.Retained distinguishes a retained snapshot delivered on subscribe from a live delivery. Nothing distinguishes a replayed live message from a fresh one except its own timestamp, which is exactly the field this change is putting back.

The OMF egress

pkg/omfegress/subscriber.go already had the right shape and applied it to one topic. publishedAt(declared) preferred the publisher's own RFC 3339 timestamp and was called for batch state alone, while stamp() returned time.Now() and was what tags, alarms and states got. A PI or AF trend fed by this egress therefore carried the same defect ADR 0083 measured in the historian.

Decision

The event time is read off the payload by every consumer, through one shared reader. pkg/eventtime holds the rules ADR 0083 established: RFC3339Nano as the layout, a marshalled zero instant read as absent, and a 30-second bound in the future direction with none in the past. pkg/historian, the gateway TagBus and the OMF egress all call eventtime.Resolve.

The reader itself is four lines. What is not four lines is the reasoning around it, which is the argument pkg/caplog already makes for being a package: the reasoning is the load-bearing part, and a copy of it drifts. Each consumer keeps its own counter and its own log line, because what a fallback costs differs by consumer. A historian row dated at ingest, a PI sample landing in the wrong minute and an HMI element that reads fresh while nothing is observing it are three different findings with the same cause.

The OMF egress is treated as the record it is. All five handlers go through publishedAt. It applies the skew bound the old two-line version had no way to express, and every fallback is counted on dcs_omf_egress_event_time_fallback_total. Nothing is filtered. A trend wants every sample, at the instant it was observed.

A live stream holds the newest observation of each subject. This is the founder ruling of 2026-08-24, taken against the three candidate behaviours. A frame carrying an observation strictly older than the newest already broadcast for that subject is dropped at the gateway and counted on dcs_gateway_stream_superseded_total. The display therefore holds what the newest observation put there, and ADR 0025's existing sweep dims it and badges its age while nothing newer arrives. During the drain measured above, an operator sees the value freeze, dim, and count up, and then go live again. The mimic does not rewind.

Three rules bound that filter.

  • It applies to the two subjects a runtime can replay. Tag values and control-module health both travel through the runtime's store-and-forward queue. Alarms and equipment states are published by the operators, so neither rides that queue. The alarm topic's retained re-delivery hands over the standing message, which cannot be older than the state it stands for.
  • A frame that fell back to arrival time is neither withheld nor recorded. Withholding it would freeze a display on no evidence, which is the failure the rule exists to prevent one direction over. Recording it would seat the gateway's own clock as the subject's newest observation. The next properly dated frame is then a live reading published 200 ms before it arrived, and the filter would refuse it as older.
  • An observation at the same instant as the newest is admitted. Two publishes can share a timestamp at the wire's precision, and dropping the second would discard a value on evidence of nothing. Only a strictly older observation is refused.

The HMI stops treating an arriving frame as evidence of freshness. Both live-update paths cleared the stale cue unconditionally, which lit an element as current until the next five-second sweep put the badge back. Both now ask HMIBinding.staleAgeFor, the one resolver ADR 0025 established. A bound value, a faceplate row and the sweep therefore cannot disagree.

Alternatives Considered

Render every replayed frame and badge its age. Nothing to build beyond reading the payload, and each frame is literally honest about itself. Rejected because the mimic animates the past at replay speed. Each arriving frame also clears the stale class before the sweep puts it back, so the badge flickers across the whole drain. The operator watches two hours of plant history play out on a surface that answers the present.

Suppress any frame older than the staleness threshold. Simplest to reason about, and the held value ages out and dims so the operator reads "this unit stopped being observed 40 minutes ago". Rejected because it discards the newest queued value when a drain finishes just over the threshold, and that value is the current state of the plant.

Filter in the HMI, leaving the gateway a pure bridge. Needs no server-side state. Rejected on three counts. The rule then has to be implemented once per client, the HMI is meant to be a dumb renderer, and every other consumer of the stream would have to rediscover the same rule for itself.

Bound the ordering map by evicting old entries. Rejected as unnecessary detail. One entry per subject the plant publishes is the same cardinality the historian's own tag table carries, and nothing outside the plant can enlarge it, because the chart's MQTT ACL grants dcs-runtime alone write on these topics (#1781).

Consequences

  • A dead tag dims. The HMI reads a real observation age. A runtime that stopped observing a tag is therefore visible on the surface an operator is watching. It used to hide behind whatever was still republishing.
  • A drain does not rewind a mimic, and it is visible while it happens. dcs_gateway_stream_superseded_total moves for the length of the drain and is flat outside one.
  • A sustained superseded rate outside a drain is a finding. It means two publishers are disagreeing about one address, or one of their clocks is wrong.
  • A PI or AF trend carries the observation. A runtime draining its queue no longer writes an old operating state into the current minute. That is the defect ADR 0083 measured, one consumer over.
  • A skewed node is visible on both consumers, on dcs_gateway_event_time_fallback_total and dcs_omf_egress_event_time_fallback_total with reason future. The remedy is the node's clock, and the counter names which stream is affected.
  • A gateway restart empties the ordering map, so the first frame per subject after one is admitted whatever its age. That is the same no-evidence rule the undated frame gets, and the display then dims on the observation's real age.

References