Server resource pressure warnings
The host agent checks RAM once a minute in a separate monitor. Checks do not overlap, and long image builds or lifecycle commands do not block them. Monitoring continues when the TUI is closed or the laptop is disconnected, as long as the host agent and active controller remain reachable. It uses the existing controller and database, and reports warnings without changing quotas, killing processes or restarting sessions.
RAM warnings use fresh measured consumption against actual finite ceilings:
- Docker's raw charged
memory_stats.usageand the inspected container memory ceiling confirmed by fresh kernel-reported accounting, falling back to a freshmemory.currentread when Docker statistics are stale or unavailable. Charged cache is included. Docker's host-RAM value for an unlimited container is not treated as a container limit. A mismatch between the requested Docker policy and reported ceiling remains unknown rather than claiming an ignored or unverified limit is enforced. - On configured Linux cgroup-v2 hosts, the user's aggregate
memory.currentandmemory.max, plus finite ancestor ceilings and their own aggregate consumption. Fresh physical container membership determines which ceilings apply. Parents can include infrastructure or sibling workloads. - On Linux, physical host RAM consumption as
MemTotal - MemAvailable, compared withMemTotal. Reclaimable memory is accounted for byMemAvailable.
The controller evaluates these scopes separately and selects the highest ratio for each running managed Claude/Codex session. It does not sum parent and child usage, process RSS and container RAM, or multiply a shared container's consumption by its session count. A scope warning describes shared consumption; it does not attribute all that RAM to the notified agent. Missing telemetry, unrecognized generations, unlimited resources, and ambiguous process identity do not qualify. Physical host telemetry on Docker Desktop describes its Linux VM.
Host swap visibility
The Hosts screen and host inspection in TUI and web show swap used / total
and its occupancy percentage alongside CPU, RAM and disk. Linux host agents read
SwapTotal and SwapFree from /proc/meminfo; the existing heartbeat and
liveness channels carry these byte counters. No extra Docker exec or inventory
scan is needed. Occupancy is yellow at 75% and red at 90%.
A measured zero capacity is Disabled. Missing, stale, offline or unsupported telemetry is Not reported, including older host agents. Swap occupancy alone does not establish active memory pressure: cold pages may stay swapped out while RAM is available. This display does not disable swap, restart containers, change resource limits or generate OOM warnings.
New Docker-in-Docker sidecars run with Docker's init process to reap orphaned
exec probes and nested runtime helpers. Existing sidecars remain running during
reconciliation; adding init is a creation-time setting. A sidecar created without
init needs an explicit replacement to remove accumulated zombies, which also
interrupts its nested workloads. Filesystem cleanup and docker prune do not
reap zombie processes.
During a host incident the operator can create the host-owned marker
<data-root>/capabilities/project-dind-auto-repair-paused before stopping DinD.
Periodic legacy-sidecar repair then leaves nested Docker stopped while provider
sessions remain available. Remove the marker to resume automatic repair. This
does not block a new explicit workspace launch, alter quotas, or remove data. A session launch may verify and use an already healthy, correctly owned sidecar while repair is paused. That path checks the existing network, checkout mounts, isolation boundary and Docker API without starting, reconnecting or repairing a sidecar; unavailable or incompatible sidecars remain blocked.
The default policy enters warning after two samples at or above 85%, enters critical at 95%, and recovers at or below 75%. These are pressure signals, not a prediction of an imminent OOM: charged cache may be reclaimed and workload demand can change between samples. Kernel OOM incidents are reported separately through the session memory-failure detector. CPU saturation/throttling and disk quota pressure do not produce RAM/OOM warnings in this version.
Warning state and cooldowns live in the controller's existing SQLite database. At most one warning is pending for a session. A new warning may be queued only five minutes after both the last queued warning and the last confirmed delivery; escalation, recovery, a new incident, generation replacement, and monitor/controller restart do not bypass this cooldown. A busy composer keeps the same durable command pending, rather than accumulating messages. The existing serialized notification lane orders permissions, file-sync notifications, and resource warnings together.
Delivery authenticates the current host and controller epoch, then verifies the session's owner, container, account UID, generation, provider PID and start ticks. The native lifecycle hook must report a normal waiting-input state. Working agents, interactive user questions, and composers containing user text defer the warning. The provider's native prompt hook acknowledges the exact notification digest; a tmux paste alone does not count as delivery. The message names the measured scope, consumption and actual ceiling, and asks the agent to release memory it owns or reduce its temporary worker concurrency. It explicitly forbids killing unrelated sessions or processes. Replaced sessions and recovered/stale pressure cancel queued delivery. Resource warnings require current managed process-generation metadata; older or unmanaged provider processes may expose RAM telemetry without receiving an automated message.
Session API metadata exposes resourcePressure with status, checkedAt,
generation, reading (scope, consumption, limit and measurement source),
lastQueuedAt, lastDeliveredAt, notificationId, and deliveryStatus.
The Events view records session.resource-pressure.changed, .queued,
.delivered, .cancelled, and .failed. The monitor stores one current state
row per session and removes orphan rows after seven days. Native resource receipt
files retain at most 128 retired entries plus the current notification. Events and
command results remain in the controller's existing audit journal.
Operator configuration
Set DEVCTL_RESOURCE_PRESSURE_CONFIG on the controller to an absolute path to an
operator-owned JSON file. The file must be a regular, non-symlink file of at most
16 KiB, owned by root or the controller UID and not writable by group or others.
Application users cannot change this policy through session or workspace APIs.
{
"enabled": true,
"intervalMs": 60000,
"warningRatio": 0.85,
"criticalRatio": 0.95,
"recoveryRatio": 0.75,
"sustainedSamples": 2,
"cooldownMs": 300000
}
The interval cannot be less than one minute, sustained warnings require at least two samples, and the cooldown cannot be less than five minutes. Recovery, warning and critical ratios must be ordered. Restart the controller after changing this file. User/ancestor scope measurements additionally require the existing operator-owned Linux resource adapter configuration.
Verification
Normal tests use dummy sessions, SQLite fixtures, mocked metrics/cgroup files, and isolated fake tmux/provider hooks. They submit no prompts to real agents. An optional integration fixture reads real Docker memory accounting from a bounded inert Node allocation. Prepare an isolated Docker daemon and preinstall a Node image; the fixture refuses a daemon containing existing managed workspaces:
DEVCTL_RESOURCE_PRESSURE_INTEGRATION_SOCKET=/private/fixture/docker.sock \
DEVCTL_RESOURCE_PRESSURE_INTEGRATION_IMAGE=node:24 \
pnpm --filter @devctl/runtime exec vitest run src/resource-pressure.integration.test.ts
The fixture creates and removes only its own read-only, offline 256 MiB container, with no provider or controller. It is skipped in ordinary CI. Fixture success is not a claim that a real-host integration or production deployment was performed.
Command result storage
The controller reads completed command results by command ID instead of loading their entire history at startup. A cache holds at most 256 recent results and 32 MiB (an oversized current result is kept until the next result arrives). These records report the outcome of operations such as file access, session launch, permission delivery, and tool installation; they are separate from session conversation history and audit events.
Persistent controllers store result payloads in private JSONL journals beside the controller database:
command-results/<database-filename>/
2026-10-15/
folder.operation.jsonl
session.permission.apply.jsonl
session.launch.jsonl
session.shutdown.jsonl
legacy/
folder.operation.jsonl
session.shutdown.jsonl
legacy.jsonl
Each line contains id, type, completedAt, and result. SQLite holds a compact index of the journal file, byte offset, and record length, preserving command lookup and user authorization. New records use the actual command type. Old records are migrated in bounded background batches; known fixed messages identify a few historical types, and unknown types remain legacy. Historical completion timestamps are left null rather than invented. Migration does not run during controller startup.
Use jq to inspect a selected journal, for example:
jq -c 'select(.result.ok == false) | {id,type,completedAt,message:.result.message}' command-results/devctl.db/2026-10-15/session.launch.jsonl
These are controller-private files, not workspace mounts or public downloads. File-read payloads are removed before persistence. Appends are synchronized to disk before publishing the index, and interrupted partial tails are discarded before the next append. Failed migration leaves the original SQLite record available. Keep the journals together with the database for raw filesystem backups; the explicit controller database backup embeds indexed results so that its single SQLite file remains portable. Retention can remove closed, unprotected journals according to the administrator policy below.
Result history retention policy
Open Settings → Result history policy (p), or Storage → Result history policy in either TUI or web. Server administrators can configure automatic cleanup, retention days, maximum journal bytes, and the interval. Defaults are enabled, 30 days, 512 MiB, and five minutes. Saving takes effect on the controller; closing the client does not stop cleanup. Save and clean now runs the saved policy immediately, including when scheduled cleanup is disabled. It deletes command-result history, not agent conversations, project files, or audit events.
Cleanup removes closed JSONL files, oldest first, when they expire or exceed the storage budget. Current-day journals, results newer than five minutes, legacy journals still being migrated, and files containing results referenced by sessions, permission deliveries, compute tasks or durable operations are protected. This is a soft storage limit: protected files can exceed it. The settings show protected bytes and the last cleanup result. A legacy result with no known completion timestamp ages from its migration date.
Deletion uses durable cleanup intents and a temporary trash name. Restart recovery restores files if their index deletion did not commit, or finishes removing unreferenced trash after a committed deletion. Explicit database backups still embed retained records.
Operator endpoints are GET/PUT /v1/command-results/policy and POST /v1/command-results/cleanup. They appear in the existing MCP operator-operation catalog; global policy changes require server administrator authorization. Ordinary server users cannot edit it or read controller journal paths.