Control-plane maintenance

A code upgrade must preserve running providers, drafts, credentials, user data and unrelated services. Keep current, candidate and compatible rollback images pinned independently. A launcher receipt is evidence of dispatch, not deployment or completion of user work.

Prove prerequisites before draining

Record the exact service and descriptor preimages, effective user, command, immutable image IDs and declared code delta. Compare environment maps and complete configuration, including ports, capabilities, bind entries and mounts. Normalize ordering only; a changed entry is a real difference.

Validate the actual rollback path before maintenance. Check the database file’s measured type, path, device/inode, owner, mode, link count and size, plus the read-only authority query used by recovery. The service user does not establish the file owner. Preserve existing ownership; do not use chmod or chown to satisfy an assumption. A file-stat failure occurs before SQL and must be diagnosed separately.

Docker Compose can record replacement ancestry as a container name rather than a container ID. Accept only the pinned previous ID or the exact preflight-measured name for that service. Compare the remaining labels against both the immutable image and declared service labels; do not widen this into arbitrary generated-label acceptance.

Parse rendered Compose configuration in memory. Save selected nonsensitive fields, equality results and hashes, never the complete output or environment values.

Resolve every external executable in the actor's actual restricted environment before changing files or stopping a service. Use verified absolute paths for privileged helpers such as runuser; an interactive shell can include directories that the bounded execution helper removes from PATH. If a missing executable is discovered after an effect, retain the failure and inspect the installed phase before resuming only the unfinished steps.

For systemd resource changes, inspect the unit's actual drop-in paths and persistent property files. A changed live cgroup value does not prove that the next reload or reboot will preserve it. Keep measured ownership and other limits intact when updating the existing files, then verify both loaded and persistent values.

Verify image and program boundaries

Keep the engine image ID, archive manifest digest, configuration digest and root filesystem layer digests separate. Inspect the actual export format before parsing it; do not assume an engine ID is a configuration digest. Verify the complete manifest-to-configuration-to-layer chain and the selected platform. Check exported tag references before loading an archive so an owned fixture cannot replace an existing service tag.

Report compressed content, unpacked layer data and filesystem occupancy separately. Engine-reported image size does not establish every one of these quantities. Stream large layers when checking their hashes and uncompressed sizes; a gzip footer alone wraps at four GiB. Declare retained input archives, content storage, snapshots and output budgets separately before importing or building.

Do not place generated programs or sensitive request metadata in process arguments. A program can exceed a kernel's single-argument limit even when the aggregate argument budget appears sufficient. Use private immutable descriptors with explicit ownership, sealing where supported, close-on-exec discipline and per-request cleanup. Preserve the existing stdin protocol and authorization checks, and test concurrent requests with independent descriptors.

Keep recovery independent and bounded

Verify an authorized independent status/result path and a finite one-shot recovery actor before changing the controller. Inspection must remain possible while the controller is draining or its normal transport is disconnected.

For a same-installation code upgrade, preserve the controller identity and epoch. Settle actual in-flight work and require coordinating host-command receipts to be terminal. Valid pending commands and known fail-closed uncertainty may remain; never clear records, forge completion or replay uncertain effects to obtain an empty queue.

Hold the same shared barrier across the watchdog decision, rollback and commit. A commit that waited for recovery must revalidate the current candidate. Disable automatic rollback durably before opening the active producer window. Any compatible-image rollback must quiesce the known writer and preserve current data; a stale database restore is forbidden.

Finish with actual readiness

If an observer check fails after replacement, retain the original failure and inspect the exact field. Ordering, generated ancestry and file ownership must be proved from actual preimages rather than guessed. Do not replay image replacement to fix an observer assumption.

When the known compatible candidate has passed all required image, code, configuration, service and preservation proofs, write a durable recovery proof and disarm record under the barrier, then restore the same authority to active. Confirm the normal TUI snapshot route succeeds. If those proofs are unavailable, report the exact blocker immediately through the independent recovery path.

Keep the initial failed actor, later recovery attempts and final acceptance separate. Deployment does not replay an earlier failed operator request or establish that its requested work was fulfilled.

Separate observations from accepted effects

A collector, formatter or test-reporter failure does not establish that an earlier installation failed. Record the accepted effect and its actual outcome first. If a final observer fails, inspect the exact installed artifact, ownership, runtime view and result through an independent read-only path; continue only unfinished verification. Never repeat a completed mutation to obtain a successful wrapper exit.

Qualify supervisors under their actual environment, resource limits and report format. Include driver, test-runner and child-process threads in fixture budgets. Preserve setup failures and measured kernel limit events separately from test failures. When a kernel metric is unavailable, report a sampled measurement as sampled rather than inventing a peak. Monitor deadlines remain bounded, but a slow accounting command is a distinct failure from exceeding the resource limit it measures.

For interactive verification, wait for an actual visible transition and enabled control before sending the next action. A heading may already be present while data is loading, or remain visible behind another view. Use the control's rendered state and the real destination, not a fixed sleep or repeated confirmation keys. Keep negative controls for actions that must remain disabled while busy.

A terminal parser callback does not prove that a synchronized repaint has finished. Wait for the terminal's synchronized-output mode to close, then capture text and cell attributes from the same immutable frame. Preserve one absolute observation deadline across nested readiness checks; partial frames must not trigger another key or a claim that user input disappeared.