Skip to content

Filesystem-backed authoring mode

Status: Implemented. dcs edit and dcs diff shipped (issues #450, #451). Owner: Engineering / Product.

Summary

Add a second backend behind the gateway's existing client.Client-shaped seam so the visual ST/SFC/FBD editors can author CRD YAML files in a local directory, with no write to the Kubernetes API server. The engineer drives git with their own tooling. The product never stages, commits, or pushes.

This is a new deployment topology (§F, alongside the existing §A–E in docs/explanation/ui-deployment-topologies.md). The connected flow stays.

A companion, smaller deliverable (#451) ships dcs diff <old.yaml> <new.yaml>. Engineers reviewing in their own git tooling can still see the semantic diff the gateway renders. It is explicitly a convenience, and the 21 CFR 11 review-of-record stays at Promote.

Motivation

The intended authoring story is: open the visual editors, edit control logic, and have the result land as YAML in a working tree that the engineer commits and pushes with their own git tooling. The editor meets them where they are (visual). Git is the substrate. The two are decoupled.

What exists today is different. Both pod-mode and laptop-mode (dcs ui, §D) are interfaces to the Kubernetes API: the gateway holds a single client.Client (internal/gateway/server.go) and every editor Save is a POST/PUT into etcd. Git is reached only at Promote, when the gateway diffs dev-cluster state against the prod repo (internal/gateway/changerequest.go). So laptop mode is "your own cluster on your laptop," not "your own files in git": it needs a running cluster to exist at all, the working copy during authoring is cluster state (not a git tree), and there is nothing to commit until Promote: no local branch, diff, revert, or offline work.

Why this is feasible

The gateway reaches Kubernetes through exactly one seam: the Server.client field (internal/gateway/server.go), a controller-runtime client.Client. Across the whole internal/gateway/ package it uses only plain CRUD: Get, List, Create, Update, Delete, and Status().Update(). There are no watches, no field selectors, no server-side apply, no typed clients, and no discovery calls. ~95% of handlers (the entire physical / recipe / procedural / batch / HMI editing surface) depend on nothing more than that interface.

The ~5% that genuinely need a live cluster (runtime tag read/write, runtime proxy/diagnostics, pod-log streaming, historian queries) already degrade in laptop mode (mqttCfg = nil, no historian URL). In file mode they return a clear "connect to a cluster" response.

Design

Component 1 — filesystem-backed client.Client (internal/filestore)

A new package providing a client.Client implementation over a directory of YAML files. Because the gateway only needs CRUD + status, we do not need a full apiserver. A correct in-memory object store with file persistence is enough.

Construction (decorator over controller-runtime's fake client). At startup the backend recursively walks <dir> for *.yaml / *.yml, decodes every document with the project scheme, and loads them into a controller-runtime fake client (sigs.k8s.io/controller-runtime/pkg/client/fake). We reuse the fake client's well-tested Get/List/Create/Update/Delete/Status and label/namespace filtering. Reimplementing object-tracker semantics is exactly the work this avoids. The filestore type wraps the fake client and, on every mutating call (Create/Update/ Delete/Status().Update()), flushes the affected object back to disk after the delegate succeeds. Reads go straight to the in-memory delegate.

This keeps the surface tiny. The decorator only overrides the four mutating methods (plus Status()), and everything else is delegated unchanged.

File layout: content-addressed. The backend indexes resources by GVK + namespace + name from each document's content. The path carries no meaning. The engineer organizes their tree however they like (plant/units/, plant/recipes/mash.yaml, anything). We maintain an in-memory map objectKey -> sourceFile:

  • Update / Status rewrites the file the object was loaded from, preserving surrounding documents if the file was multi-document (----separated). We preserve document order and only re-serialize the changed document's object.
  • Create of a brand-new object writes to a default path derived from the resource: <dir>/<group>/<kind>/<name>.yaml (namespaced resources nested under <dir>/<namespace>/...). This is a fallback only. Existing files keep their hand-chosen location.
  • Delete removes the document. A file left empty by that removal is deleted, and a multi-document file is rewritten without it.

Serialization fidelity. Saved files stay clean and git-diffable: we strip server-only metadata (resourceVersion, uid, creationTimestamp, generation, managedFields, status timestamps where empty). resourceVersion is synthesized in-memory (monotonic counter) so the fake client's optimistic concurrency keeps working within a session, but is not persisted.

Write-back is comment- and layout-preserving (#454). On Save the store does not rebuild a document from the object graph. It patches the changed fields into a comment-preserving node tree (gopkg.in/yaml.v3) parsed from the source document. That is why # ... comments, key order, and the quoting/indentation of untouched nodes survive. Only the edited fields move in the engineer's git diff. The blast radius is "documents you edited":

  • A document whose data is unchanged is written back byte-for-byte (this covers untouched siblings in a multi-document file).
  • A changed document keeps every comment and the layout of every node it did not touch. The edited scalar is free to re-pick a quoting style for its new value, and added/removed keys append/drop in place. (Blank lines within a changed document are not reconstructed, because yaml.v3 has no representation for them.)
  • A brand-new object (a Create with no source document) is rendered from a clean marshal, as before.

If the source document cannot be parsed or patched, the store falls back to a clean (normalized) marshal, and the Save succeeds.

Concurrency / external edits (v1 scope). Load-at-startup. If the engineer edits files on disk while the editor is open, the in-memory store is the source of truth until restart. A later iteration can add fs-watch reconciliation. It is out of scope for v1 (documented as a limitation).

Open question (flagged for plan review): multi-document file write-back preservation adds complexity. Alternative v1 simplification: support reading multi-doc files, but on write-back split a touched multi-doc file into one-object-per-file (a one-time, visible reorganization). Recommendation: preserve multi-doc files (no surprise reorganization of the engineer's tree) and accept the extra serialization care. Decision belongs in the plan.

Component 2 — dcs edit ./plant subcommand

A new cmd/dcs/internal/cmd/edit.go, structured like ui.go but building the filestore client in place of a Kubernetes client:

  • Positional arg: the directory (dcs edit ./plant). Defaults to . if omitted.
  • No kubeconfig, no cluster, no network required.
  • Reuses pickPort, browserHost, browser auto-open, --port, --bind-address, --no-browser from the existing UI plumbing (factor the shared bits out of ui.go into a small helper, with no copy-paste).
  • Boots gateway.NewServer(...) with ModeDevelopment, auth.AuthModeNone, mqttCfg = nil, no historian (same as laptop mode), plus a new backend-kind signal (below).

Component 3 — file-mode capability signal + graceful degradation

The gateway must know it is file-backed so it can (a) tell the UI to hide/disable live-only affordances (runtime tag values, sim test, diagnostics, logs) and (b) return a clean error from the ~5% of handlers that need a cluster.

  • Add Server.backendKind ("cluster" | "file"), set via a WithBackendKind option from dcs edit. Default "cluster" preserves all existing behavior.
  • The existing capabilities/system endpoint advertises the backend kind. The UI consults it to gray out live-only controls (consistent with the product/deployment split: the product exposes the capability, the UI reflects it).
  • Live-only handlers return 409 Conflict (or a dedicated code) with a machine-readable reason: "requires-cluster" in place of a 500, so the UI can prompt "connect to a cluster to use this."

Realized UI scope-down (issues #456, #457, #458)

In file mode the UI narrows to the authoring surface only:

  • Sub-apps (#456): only System is reachable. The cross-app HMI, Data, and Docs top-bar links are hidden, and GET /hmi / GET /data 302-redirect to /system so a bookmarked URL can't land on a broken operations view.
  • Views (#457): operations/runtime views are removed from the System sidebar and from navigation: Diagnostics (and Archive Integrity), Setup, batch execution, schedule, production information (historian / audit / genealogy), live alarms, change requests, control-module / unit live data, and device discovery. The authoring set remains: physical model, recipes, equipment templates, and the SFC/FB/procedure editors. The startup view is the physical model (not diagnostics, which needs a cluster). The single source of truth is CLUSTER_ONLY_VIEWS in nav.js. Cross-app chrome is hidden by marking elements data-cluster-only (hideClusterOnlyChrome).
  • Live values (#458): components that would show a live equipment value (unit/CM live-tag panels, runtime status, I/O channel reads) render a calm "Live values require a connected cluster" note. An error or disconnected state would be wrong there, and the green/red pair stays reserved for health. The UI detects file mode up front (isFileMode()) and does not attempt the fetch. The requires-cluster 409 is the fallback for any stray live call.
  • Controller-set status (#459, #460): the same principle covers status fields only a reconciler populates. ControlModule.Status.TemplateResolved is set by the control-operator, so it is always false offline. The "Template Not Resolved" banner is suppressed in file mode, and the Template row renders the template name alone, with no resolved/unresolved annotation (#459, #1270). The site-view list tabs (Units, Control Modules, Controllers, IO Modules) likewise omit their runtime/status columns offline (#460).

Component 4 — dcs diff (issue #451, option B)

dcs diff <old.yaml> <new.yaml> renders the semantic diff (the same "Mash Hold: hold time 30 min → 45 min" output the Promote panel shows) in the terminal, reusing the existing renderers verbatim. There is no second implementation to drift.

The renderers (internal/gateway/static/js/diff-renderer.js + 21 per-CRD renderers) are pure, DOM-free, CommonJS-exportable ES6 already exercised under node:test. To keep dcs a single static Go binary, the recommended approach is to embed the renderer JS (go:embed) and execute it in an embedded JS engine (goja, pure-Go) behind a tiny require shim that maps the 21 relative imports. dcs diff decodes both YAML files to JSON, calls computeSemanticDiff, and pretty-prints the returned entries (grouped by impact), with a --format=yaml|json|semantic switch and a non-zero exit when differences exist (so it composes in scripts).

  • Fallback if any renderer uses a construct goja cannot evaluate: shell out to node against an embedded entrypoint when a node binary is present, else emit the raw structural diff with a note. (To be confirmed during implementation by running all 21 renderers under goja. The ES6 features in use are arrow funcs, template literals, const/let, and spread, all within goja's supported set.)
  • Explicitly out of scope for #451-B: the git external-diff driver and the CI PR-rendering check (those were option C). dcs diff is the clean prerequisite for both if we choose to add them later. They would shell out to it.

dcs diff is informational only. It creates no AuditRecord and no e-signature. The banner/--help text says so.

Compliance posture (issue #451)

The 21 CFR 11 review-of-record (the regulated review with the e-signature and the immutable AuditRecord) stays in the connected Promote / ChangeRequest flow in every mode. This is non-negotiable per docs/compliance/21cfr11.md (§11.10(k), §11.50, §11.70) and docs/change-control.md:

  • The e-signature is a server-side HMAC computed in internal/gateway/changerequest.go:stampSignature, cryptographically bound to the target object's spec, stored inside a write-once AuditRecord.
  • Moving approval off-product (a signature sitting in a git commit) would break §11.10(k)/§11.50(b)/§11.70: no immutable trail, signature not bound to the cluster-state change.

Therefore:

  • File mode is dev/offline authoring. Day-to-day review there happens in the engineer's own git tooling. dcs diff gives that review a semantic view. It is a developer convenience, and the review-of-record stays in the Promote flow.
  • Nothing changes in the compliance carrier. When the engineer's files reach a cluster and are Promoted, the semantic diff + e-signature + AuditRecord apply exactly as today.

This posture is recorded in docs/compliance/21cfr11.md and docs/change-control.md (a short subsection each), and in the new topology §F.

Documentation changes

  • docs/explanation/ui-deployment-topologies.md — new §F. Filesystem authoring (cluster-free), plus a row in the decision matrix.
  • docs/explanation/gitops-for-automation-engineers.md — note that the editor can author directly against files (completing the "GitOps is the substrate, not the interface" argument) and that semantic diff is available off-product via dcs diff, with the review-of-record caveat.
  • docs/compliance/21cfr11.md, docs/change-control.md — the #451 posture subsection above.
  • docs/doc-map.json — map internal/filestore/, cmd/dcs/internal/cmd/edit.go, and the dcs diff command to these docs.

Non-goals

  • Embedding a git client in the product (commit/push buttons). Rejected in #450: drags the product into credential/remote/merge-conflict territory.
  • The git external-diff driver and CI PR semantic-diff check (#451 option C). Deferred. dcs diff is the prerequisite.
  • fs-watch live reconciliation of externally-edited files (v1 limitation).
  • Making the live-only handlers (runtime/tag/historian/logs) work offline.

Test strategy

  • internal/filestore: Go unit tests — load fixtures, exercise Get/List/Create/Update/Delete/Status, assert file round-trips, multi-doc preservation, content-addressed write-back to the original path, namespaced vs cluster-scoped layout, metadata stripping.
  • dcs edit: a CLI smoke test that boots the server against a temp dir and hits a couple of CRUD endpoints, plus a Playwright spec that loads an editor against a fixture directory and saves (optional, behind the existing UI smoke harness).
  • dcs diff: golden-file tests over representative before/after YAML pairs for a few CRD kinds, asserting exit-code semantics. If goja is used, a test that loads all 21 renderers under goja and asserts no evaluation error (guards the fallback).
  • Reuse existing test/js/diff-renderer-*.test.js unchanged as the renderer source-of-truth.

Rollout / sequencing

  1. internal/filestore package + tests (the backend).
  2. dcs edit subcommand + capability signal + graceful degradation + tests.
  3. UI: gray out live-only controls when backend kind is file.
  4. dcs diff + tests.
  5. Docs (§F, gitops, compliance posture, doc-map).

Steps 1–3 satisfy #450, and step 4 plus the compliance subsections satisfy #451.