Skip to content

ADR 0090: A historian holds an outage in time, and says what it lost

Status: Accepted Date: 2026-09-09 Issue: #2116 Related: #1728 (the sitting that found it), #1950 (the historian starts without its database), #1342 (the requeue and the old cap), #1607 (the rate-limited drop line)

Context

On 2026-09-09 the bench's historian lost its database for thirty-seven minutes. The first twelve were a node rebuild. The remaining twenty-five were a credential: the recreated CloudNativePG Cluster minted a new application password, and the historian had read the old one into an environment variable at container start. Nothing restarted it, so nothing reconnected it.

Through that window the ingester did what #1342 built it to do. It requeued every failed batch and held the buffer at ten flush buffers, which is ten thousand records. At the bench's arrival rate that is about two minutes. The rest was dropped, oldest first, at seventy-two records a second, and the log said so once every thirty seconds:

historian buffer still at retention cap; dropping oldest records
  {"type": "tags", "dropped": 2160, "droppedTotal": 45371, "retained": 10000}

The gateway's system card said Historian degraded and database unavailable. That is the same sentence it says while a slow flush catches up, when nothing is lost. An operator reading the card could not tell the two apart, and the account of the loss lived in a log line that the next restart would forget.

Three questions came out of it, and this record answers all three.

Decision

The buffer is sized in time, with a memory ceiling behind it

A record waits for the database for as long as the configured retention, one hour by default, before the historian gives it up. The retention's measure is an outage, and the outage on record is the one above.

Behind the retention is a ceiling on records per type, three hundred thousand by default. A tag record costs about 250 bytes held, and twice that at a garbage-collection peak. The default is therefore roughly 75 MB live inside the chart's 512 Mi limit. The ceiling is the memory bound. At the bench's rate it is over an hour away. At a plant publishing a thousand records a second it is five minutes away, and it then binds before the retention does. That is disclosed. The log line names which bound took the records, the ingest status counts the two apart, and the remedy is to raise the ceiling and the memory limit together.

A record is judged on how long the historian has held it. Its own timestamp plays no part. A runtime replaying its store-and-forward backlog publishes records observed an hour ago, and judged by observation time they would be thrown away the moment they arrived. That is the opposite of what a buffer that carries outages is for. Every buffered record therefore carries its arrival, a failed batch is requeued under the arrival it already had, and TestBuffer_RetentionIsJudgedOnArrivalNotOnEventTime holds the rule.

A loss is a fault with a cause and a count, on the surface and in the record

Every buffer keeps a loss account. An episode opens on the first drop and closes on the next successful flush. While it is open, GET /api/v1/historian/ingest carries the count so far, since when, and the split by bound. It reads memory and never the database, so it answers during the outage it describes.

The gateway's system card reads that endpoint beside /readyz. During an outage the card now says whether data is being lost or merely held, and by how much:

database unavailable; dropping records: 45,371 tag records dropped since 19:32Z
  (buffer ceiling of 300,000 records reached before the 1h0m0s retention)

database unavailable; buffering: 12,400 tag records held for 3m of 1h 0m

A historian that is dropping is degraded whatever /readyz says, because the two are read on different clocks. After the outage the card is Running and says what was lost, for a day, and then stops.

When the episode closes, the ingester writes it to the ingest_gaps table: the type, the window, and the count by bound. A write that fails is kept and retried after the next successful flush. The row is the record of the loss, and unlike the records it is never given up. GET /api/v1/historian/gaps reads the table. A trend with a hole in it can then be read against the reason for the hole. This is the durable signal ADR 0063 asks a log line to have.

The credential is a file the kubelet rewrites, read before every dial

The chart mounts the CloudNativePG app Secret as a whole-volume Secret file and hands the historian its path in DATABASE_URL_FILE. The historian re-reads the file before every new database connection and applies the user, password, host, port and database it finds there. Connections already open keep working until the server closes them, which is the rotation window every other client in the product gets under ADR 0061. A recreated Cluster is followed within the kubelet's sync period, with no hand on the pod.

The mount is never a subPath mount, because a subPath mount is frozen for the life of the pod and would undo the whole of this. The chart test historian-db-credential.sh refuses one. --database-url and --database-url-file are exclusive, and the historian refuses both set. Resolving them by precedence would silence whichever one somebody was counting on. The external-URL mode keeps its environment variable: that string is the deployer's own, and nothing rotates it.

Alternatives Considered

A spill to local disk. The historian runs on a read-only root filesystem with no volume. A disk spill is a file format, a replay order and a rotation policy, all of which the unit runtime already carries for its own store-and-forward queue, and a second copy of that machinery is a second thing to keep right. The in-memory bounds carry the outage on record with a margin, and a longer one is a values change. If a deployment needs hours of carry at plant rates, this is the road, and it is open.

An Alarm resource for the annunciation. The historian is a platform service with no Kubernetes client, by design. The surface a platform fault belongs on is the system card, and dcs health prints the same card. The Prometheus alert rules in the monitoring guide carry the same two facts for a pager.

A reason label on dcs_historian_dropped_total. The split by bound would be useful on a dashboard, but a label change moves every existing series and dashboard at once. The split is on the ingest status and in the gap record instead, and the counter keeps its shape.

A bigger retainFactor. The old bound was ten flush buffers, and the first draft of the fix raised the ten. A record-count bound answers a question nobody asks. An operator asks how long an outage the historian carries, and a count only answers that at one arrival rate.

Consequences

  • A database outage of up to an hour at the bench's rate loses nothing. Longer outages, or higher rates, lose the oldest records first and say so in three places.
  • The historian's memory during an outage is bounded by the ceiling and not by the arrival rate, as before. The default ceiling costs about 75 MB live per type at the limit. A deployer raising it raises the memory limit with it.
  • The system card's degraded on the historian now carries the cost of the outage, and the same message reaches dcs health.
  • ingest_gaps is schema version 7. A historian upgraded onto a database below it applies the migration on its first successful connection, the way every migration is applied.
  • The historian's Deployment carries one more volume. A chart consumer that restates the historian's pod spec by hand will need the mount, and the chart test says what shape it must have.
  • The rotation runbook's historian step no longer restarts the pod.

Amendment: the account counts what a flush holds, and the Service carries the pod through the outage (#2124, #2125)

The first retake of the three Diagnose-panel outage frames against a v0.7.11 stack (#2119) found the card reading database unavailable alone for the first two minutes of the outage, and Readiness probe failed: ... connect: connection refused after that. Neither was the line this ADR added. Two mechanisms, and this amendment records both rulings.

The account counts what a flush holds. A flush swaps every entry out of the buffer for the duration of the insert, and the ingest status read the slice's length. So for the whole of every attempt the account said zero. The attempt that mattered ran for two minutes, because the pool's dial had no bound: pgx gives an unspecified connect_timeout two minutes, the CNPG app secret specifies nothing, and the readiness ping had been bounded against exactly that (#1950) while the flush had not. The ruling is that a record a flush holds is still waiting for the database, so the account counts it and reads its age off the in-flight batch's head. And the pool dials under the same ten-second bound the readiness ping uses, unless the connection string spells its own. One clock for a database that is not there.

The Service carries the pod through the outage. #1950 made the historian's readiness turn on its database, and a Service does not route to a pod whose readiness has failed. From that commit on the Service withdrew the historian at the moment it had something to say. The gateway reaches the historian through that Service, and on the capture stack both of its probes kept working over a connection opened before the outage, then met connection refused on the first redial. Asked directly, the pod answered 503 database unavailable and an account of 238,889 records over 314 s. The ruling is publishNotReadyAddresses: true on the historian Service. Every read the gateway makes wants the historian's own sentence, and a refused dial is not one. There is no second replica a withdrawn endpoint would protect. A chart test reads the chain from HISTORIAN_URL to the pod's labels, so the field cannot be asserted on a Service nothing dials.

The same run measured the ceiling's cost at about 1.5 KB a record resident, against the 250 bytes the Consequences above assumed. That is #2126 and is not settled here.

Amendment: the ceiling is argued in bytes, and measured (#2126)

The #2119 retake read the historian container's cgroup at the 300,000-record ceiling: 536,870,912 bytes, which is the 512 Mi limit itself, with the process one allocation from the kernel's OOM killer and still climbing. The Decision above had sized the ceiling at 250 bytes a record. It had counted the entry and nothing that stands beside it at the peak of a failed flush. The requeue built a merged copy of the batch and the buffer, so the peak was two buffers wide. The COPY to the database boxed every field of every row into an interface for the whole of the insert. And the collector runs the heap out to twice its live size before it collects, which no estimate of live bytes accounts for.

Three rulings.

The ceiling is measured, and the measurement is a test. TestBuffer_CeilingFitsInsideTheChartMemoryLimit reads the historian's memory limit and its buffer ceiling out of the chart's own values.yaml, fills a buffer to that ceiling through the real message handler, runs the failed-flush cycle an outage runs, and reads the runtime's account of the live heap at the peak of it. The modelled resident size is the idle process plus twice that peak, and it has to fit inside the limit. The chart's ceiling has to equal the binary's default, so the two are raised together or the test is red. On the same run the runtime's high-water mark of memory taken from the operating system agreed with the model to within a tenth.

The peak is one buffer wide. The buffer is a ring. A drop at the ceiling moves an index. A swap hands the batch out in the ring's own array. A requeue drops the oldest off the batch's front, moves the survivors down, and appends the newer arrivals into the room that made. At the ceiling it allocates nothing. The COPY source yields one row at a time. The test that measured the old buffer found a fault beside the memory: a drop from the front of a slice was a copy of the whole buffer, once per arriving record, and thirty thousand arrivals past the ceiling cost a hundred and six seconds on the development host. A plant publishing a thousand records a second into a container with half a core could not have kept up, and the ring answers that as well.

The default is 100,000. At the ceiling a tag record measures about 260 bytes of live heap at the peak, so the default models to about 120 MiB resident inside the 512 Mi limit, with most of the limit left to the process. The measured cost after the changes above would let the old default fit, and the smaller one is chosen anyway. The model counts what the test can see. The container on the capture stack held more than that, so the margin is deliberate. At the bench's rate the ceiling is over twenty minutes away, and at a thousand records a second it is under two. The log line, the ingest status and the system card say which bound took the records, as before.

The loss log closes on the flush and nowhere else. At the ceiling the log alternated stopped dropping and dropping within the same second, every thirty seconds, while the account underneath climbed. The rate-limited reporter read a zero-drop arrival as the buffer having drained. Between two failed flushes the swap empties the buffer, so every arrival fits, and each one closed an episode the requeue reopened a moment later. The episode is the buffer's loss account, which closes on a successful flush. The reporter is now closed by the flush that closes the account, with the episode's whole total, and an arrival that fits says nothing. pkg/omfegress keeps the arrival-driven close, because its queue's cap is its depth and an arrival that fits there does mean the queue is below it.

Not taken: a memory limit told to the runtime. GOMEMLIMIT at the container's limit would make the collector work harder as the heap approached it, and the headroom factor in the model would shrink. It is a soft limit. A live heap above it is still killed, so the ceiling has to fit on its own. A collector pinned near its limit on half a core is a different outage. It stays open as a later change, argued on its own measurement.

Not taken: a flush in chunks. A flush at the ceiling swaps the whole buffer out and writes it in one COPY. Writing it in flush-sized pieces would shrink the in-flight batch, at the cost of a round trip per piece and a requeue that has to put a partly written batch back. The ring makes the whole-buffer flush cost one array, and that array is the buffer's own. The measurement says that fits.

Amendment: a healthy historian holds records too, so the card renders what is overdue (#2127)

The #2119 retake read the Historian card before injecting its fault. On a healthy stack at v0.7.12 every poll carried buffering: N tag records held for 0s of 1h 0m under Status: running. The database was up, the readiness probe was passing and the historian was flushing on cadence. The holding clause claimed an outage that was not happening.

Two facts made the line. The card rendered the clause whenever a type's buffered was above zero, and buffered is never zero for long on a historian taking a thousand records a second: records arrive between every two ticks, and since the amendment above the batch a flush holds counts until the insert returns. Before that amendment the same line could appear on the narrower window between a push and the next flush. The amendment made it permanent.

The account says what is overdue. A record held for less than one flush interval is waiting for the tick. One held for less than two may be in the flush's hands, because the tick swaps it out at up to one interval old and the insert then holds it until the database answers. A record older than two intervals was present at a tick and is still here at the next, which a healthy cycle never does. That record is waiting on the database. The ingest status carries overdue, the count of those, beside buffered, and names the flushInterval the line is drawn from. The flushes run the four types in series, so a slow insert on one type ages the others past the line as well. That is the database being slow, which is what the count is for and which a readiness verdict alone cannot see.

The card renders the overdue count and nothing on a healthy historian. The holding clause reads overdue. It does not read buffered. A healthy card carries no line. Through an outage the number on the card is a few seconds of arrivals smaller than what the endpoint calls buffered, and it is the number that is true. The same fix is what dcs health prints, since it reads the same card.

The healthy frame asserts the silence. The healthy Diagnostics screenshot had assertions for a title, a badge and a card count, and none of them could see a line that should not be there. text_absent is the assertion primitive for a surface that is meant to say nothing. A scope that matches no element is refused, because an absence inside nothing holds for free. The frame now asserts that the Historian card carries neither buffering: nor dropping records:.

Not taken: the gateway judging readiness. Rendering the clause only under a failed readiness verdict was the other shape. It loses the case the Decision above named. A ready historian can sit on a database that takes every batch late. The historian knows its flush interval and the gateway does not, so the historian draws the line.

Not taken: one flush interval as the line. A record swapped out at the tick is up to one interval old on a healthy cycle, and the insert holds it past that. At one interval the clause would flicker on for the length of every insert. At two it cannot appear without a tick having been missed.