Architecture, security and performance remediation

The source review used baseline 1fff689e50e6f048e5adb2994e4be8da590a7d37. The remediation starts from 8cfb1a51. The original source-only security candidates were not confirmed runtime findings; regression tests for the fixes use disposable local data and identities.

A repeat review of 9449aaa2 identified additional import-filter and controller read-path work. The changes below include that follow-up. They are source and local-fixture changes; deploying a new controller or rebuilding a workspace is a separate operation.

Host authority and generated files

Host discovery is validated as a complete batch before heartbeat writes. Docker resources, declared host services, generated service resources and forwards must belong to the authenticated reporting host. Stale heartbeat sequences do not apply discovery changes. Existing controller-owned or other-host records cannot be replaced through discovery.

Rotating or revoking an enrolled host credential immediately unregisters its terminal and relay channels. Host deletion, compute retirement and orphan recovery use the same revocation path. Active relays close; terminal attachments can reconnect through the newly authenticated host channel. This does not stop workspace containers or provider processes. Changing credential hashes outside the controller API is not a supported revocation mechanism.

Devcontainer import reserves generated configuration, lock and report names instead of copying those repository entries. Generated JSON uses exclusive creation with private permissions. Ordinary imported assets retain their existing behavior, including asset symlinks.

Host writes of bundles, handoffs, patches and result capsules now use one bounded Python helper with descriptor-relative directory traversal and atomic file replacement. It pins the root chain as well as relative children. Mutable parent symlinks cannot redirect a privileged write. Protected system aliases such as Darwin's /var have a separate ownership/permission check and a bounded alias count; this is not a general permission to follow project links.

New directories receive shared access and their parent's group at creation, independently of the installer's private umask. Existing directories never have their permissions widened. Real Linux UID tests cover all four command paths, an existing controller-owned private directory substituted during creation, and authorized group access afterward. The subprocess has bounded input/output, cancellation and a deadline. Python3 and POSIX descriptor-relative filesystem support are required; unsupported hosts fail explicitly.

Repository-selected mounts, host namespaces, device access and privilege overrides are also removed independently of the Internet policy. The same option matcher covers attached and clustered Docker short flags in import, restricted egress and native-folder checks. Choosing unrestricted Internet access does not authorize extra host filesystem mounts. Managed mounts, user settings and runtime arguments are still assembled by devctrl. Tests inspect generated configuration; they do not attempt a mount against a real host daemon.

Expensive host operations

The host dispatcher bounds builds, workspace preparation/start, tool installs, restores and activity inspection. Defaults are two concurrent expensive jobs per host, one per workspace and 64 waiting jobs. Jobs exceeding the waiting capacity remain in the controller's durable queue for retry; they are not marked complete. Existing command deduplication also covers admitted and queued jobs.

Permission delivery, resource warnings, cancellation, session termination and network-policy delivery do not consume these slots. The limits apply to command execution, not to the autonomous agent processes already running in containers.

Advanced host-agent environment settings are:

Variable Default Accepted range
DEVCTL_HOST_JOB_CONCURRENCY 2 1–32
DEVCTL_WORKSPACE_JOB_CONCURRENCY 1 1–8
DEVCTL_HOST_JOB_QUEUE_CAPACITY 64 1–1024

Invalid values use the default. The per-workspace limit is still subject to the aggregate host limit. These settings take effect when the host agent starts.

Result journal recovery

Interrupted appends are scanned backward in 64 KiB blocks. A maximum-sized incomplete tail takes at most 130 read calls, including the final-byte probe, instead of millions of one-byte reads. Oversized invalid tails are rejected without changing the journal or its index. Payload-before-index durability, record limits and bounded caching remain enforced.

Terminal command acknowledgements also have controller-owned host receipts. If the host retries after losing the HTTP response, the same authenticated host is acknowledged without applying side effects again or replacing the first result. Another host remains rejected. Deferred execution is not terminal and retains its durable command. Permission withdrawal retires a pending delivery with the same trusted binding, so a late success cannot restore withdrawn rules.

Host result submission stages serialization and journal fsync on a bounded I/O worker. Each worker owns unique journal shards, so the controller does not hold a SQLite writer lock across fsync. Failed staging leaves the command available for retry. Final-result normalization and authority checks precede lifecycle changes; withdrawal and command replacement are rechecked after asynchronous writes. Cold result reads, legacy migration, retention planning/deletion, passive WAL checkpoints and standalone backups use a separate bounded maintenance worker. Both workers are lazy and share the controller's admission budget. Backups use their own copied index to include a consistent journal snapshot. Obsolete unindexed shards are reclaimed in bounded scans after a grace period.

Flushed results keep an explicit file pin until they are published or discarded. A worker's identity is removed from the orphan protection set only after its exit is confirmed. This lets the same running controller reclaim abandoned shards after a worker failure without deleting a result whose caller paused between flushing and publication. Superseded and rejected completions release their staged pins in a finally block. Orphan eligibility uses the enqueue-time cutoff, so a writer created after the protection snapshot cannot become eligible merely because maintenance waited behind slow I/O.

Operational timers read small durable result summaries instead of opening cold journals. A missing legacy summary is rebuilt asynchronously; its caller retries with an explicit loading response rather than treating the operation as pending or complete. Summary fields are allowlisted and capped at 32 KiB. Full manifests and other payloads are read asynchronously. Revision checks discard stale legacy reads, including updates that retain the same SQLite row ID.

Retention discovers historical references on its worker and checks a reference epoch before retiring journal indexes. Unrelated heartbeats do not invalidate a cleanup plan. Live pending references and active writer shards remain protected; file deletion follows the short metadata transaction.

The result index, receipt and command retirement commit in one SQLite savepoint; the in-memory result cache changes afterward. Receipts contain no output, remain on disk and retain a thirty-day retry window with bounded cleanup. Old completed commands without a trustworthy receipt remain fail-closed. This is response-loss idempotency, not an exactly-once guarantee for every external host side effect.

Controller read models and memory

Console state is projected before historical bodies are deserialized. Dashboard requests skip handoffs and result capsules. Closed approvals, archived runs and their events, checkpoints, inputs, triggers and results remain in SQLite and are read on demand; operational records remain resident. Inactive records do not warm the entire heap after restart.

Named-user snapshots select owned inventory before projecting entity bodies. Ownership metadata excludes command/result history; archived approvals, handoffs and capsules are filtered in SQLite before JSON decoding. The final authorization projection still clears unknown collections by default. Public/shared metadata continues to follow the existing grants. Secondary indexes select account usage, grants, bindings, service dependencies and shared-host container inventory before materialization. Catalog pages project metadata in SQLite and read a document body only after authorization. This is bounded controller admission, rather than a per-tenant CPU or memory quota for every application operation.

Latest-session run and Decisions lookups are batched instead of executing one JavaScript-to-SQLite call per session. Automatic Decisions work considers at most 32 candidates per tick across pending and budget-blocked lanes. A rejected budget check starts a durable 60-second cooldown. Budget/settings edits activate bounded rechecks; accounting writes do not wake the blocked backlog. Claims and completions compare the current durable state to prevent duplicate evaluations and stale responses from overwriting a recovered job. Provider failures remain visible and are not automatically billed again.

GET /v1/decisions/automatic/jobs provides bounded history pages with owner/project filters and scoped totals. State responses contain one recent handoff per visible source session plus explicitly referenced handoffs, recent run summaries and historyAccess metadata. Older choices remain available through keyset pages, including the shared TUI/web/desktop continuation picker. History is not silently discarded or loaded in full to fill a selector.

Large documents, skills, MCP definitions and configuration snapshots remain on disk after restart and load individually. A shared LRU accounts for durable cached bodies, while pending replacements and persistence jobs reserve protected bytes. Explicit set followed by the Store save boundary persists a catalog change; eviction never discards a pending replacement. Live session identity and other authoritative operational maps remain resident.

DEVCTL_CONTROLLER_RESIDENT_BUDGET_BYTES sets the shared estimated allocation budget, with a minimum of 65,536 bytes. Its default is the smaller of 32 MiB and one quarter of the effective heap ceiling. Admission also stops expensive work at 80% of that ceiling, respecting an explicit Node old-space limit as well as V8's reported total. Estimates are conservative JSON/object accounting, not a measurement or hard cap on process RSS. Worker heaps and live operational state also consume memory.

HTTP reserves buffered input before parsing, and snapshot size checks run before catalog bodies are decoded. Large read responses reserve their serialized output until the socket finishes; streamed files use bounded buffers and backpressure. Pressure returns a retryable 503 instead of accepting unbounded work. Health, operator diagnostics and small stop/revoke/delete operations remain reachable. SIGTERM/SIGINT stop admission, drain requests and durable result jobs, then close the database; they do not terminate workspace provider processes. Controller shutdown also removes terminal watch and reconnect timers before closing viewer channels, preventing late callbacks from reading a closed database.

Named-user durable session creation uses nested ledger savepoints when called inside the launch transaction. Standalone resource admission retains its own transaction. A failed reservation rolls back its own work without committing or discarding the surrounding launch request.

Cheap preferences and attention-sound polling do not prefetch operational command results or scan every session. The technology-profile monitor observes top-level asynchronous failures, logs them and retries on a later tick; shutdown waits for a tick already in flight.

The shared TUI/web/desktop console compares JSON values directly and shares unchanged branches. It no longer allocates two full serialized snapshots just to decide whether polled data changed. Approval update ordering still prevents a slow inventory response from replacing a newer delivery confirmation.

Controller diagnostics

GET /v1/controller/diagnostics requires the canonical infrastructure operator. It reports RSS, heap use and the actual V8 heap limit; ten-second event-loop delay/utilization samples; and bounded operation timings for HTTP requests, snapshot construction, entity writes, result persistence and archive maintenance. Result serialization, journal flushes and cold reads have separate timing labels (result.serialize, result.fsync, result.read) so a slow flush can be distinguished from JSON processing before changing persistence architecture. Duration counters are lifetime totals; percentiles use the last 256 samples per fixed operation label. No request URLs, credentials, project IDs or payloads are stored in these measurements.

The endpoint is automatically discoverable through the operator MCP catalog. It is not exposed to tenant administrators. Sampling starts with the controller and stops when it closes. These measurements distinguish a slow state query or write from a busy host; they do not infer the cause of earlier incidents.

Transport and interface recovery

The three MCP stdio bridges validate JSON-RPC envelopes and cap each input frame before dispatch. Invalid JSON, scalar values and malformed requests return a protocol error without crashing a live bridge. Native workflow responses must match the request ID and the supported result envelope; malformed socket output becomes a reported failure rather than an unhandled asynchronous exception.

Detached CLI resume has a bounded startup deadline, extends it only for valid attach progress, and checks that a newly ready provider survives its initial startup. A process that exits immediately is reported as a failure. Timers and socket handlers are released after completion. This check detaches the viewer; it does not terminate the provider process.

Refreshing an MCP broker revision first closes that broker's accepted clients, then awaits its listener's shutdown. An attached old client can no longer hold policy or credential refresh open indefinitely. Other sessions' brokers remain connected. The regression uses actual private Unix sockets and child processes.

The account-network shortcut is routed from the mounted console so that input on the first rendered Accounts frame is not lost while its editor mounts. Context-tool image selection clears errors when changing hosts and ignores late results from the abandoned host. A stopped container cannot be updated through a keyboard shortcut whose browser button is disabled.

Follow-up browser tests reproduced a repository-picker race: an earlier Git account's response or error could replace the newly selected account's results. Picker loads now have an explicit generation, and reopening clears the previous actionable repository before fetching account metadata. A queued dialog-close event from the previous opening cannot cancel the new load.

Offline Windows-builder tests exercise release checks and command boundaries; real local subprocess tests cover streamed output, timeouts and cleanup without starting a VM. Build, VM-control and smoke-fixture pipes are explicitly closed. Non-object JSON diagnostics no longer crash build-record or VM-response parsing. These tests do not establish native Windows rendering or installer coverage.

The Python PTY bridge handles a child that has just exited on macOS without turning normal cleanup into a permission error. Where supported, it observes exit without reaping before signaling the owned process group, retaining the leader's identity until its background descendants are terminated. Genuine permission failures for a live child remain errors.

Real Linux private-socket tests also found a buffered-input failure: when the initial request and first terminal input arrived in one read, that input waited for another write. The host runner now consumes the pending frames through the same validated parser used by later reads. Invalid identity types and non-object frames are rejected as protocol errors instead of escaping as attribute errors.

Cloud host discovery rejects malformed or non-public IPv4 addresses using Node's IP parser. Provisioning polls retain the requested SSH user and private key name, and an unknown provider image does not replace known image metadata. These checks use provider HTTP fixtures, not paid provisioning calls.

Landing localization performance

Localized HTML applies validated, non-overlapping edits in one fragment join, instead of copying the complete page after every translation or link change. Existing localization tests and a 2,000-element fixture verify the output. On one local comparison-page sample, old and new implementations produced byte-identical 3.04 MB output; processing took 5,962 ms and 379 ms respectively. That observation is not a production latency guarantee or a CI timing threshold.

Repository checks

Pull requests run the reusable Repository checks workflow: locked dependency installation, lint, typecheck, build, workspace/wrapper tests and landing tests. Standalone client and host image publication depend on that workflow succeeding. Controller and browser-interface changes trigger the image workflow as well.

GitHub branch protection is a separate repository setting. To require the check before merging, select Lint, types, build and tests in the protected-branch rules. The checked-in publication dependency is enforced even without that rule.

Access revocation and asynchronous integrations

History exports and collector streams recheck authenticated role and ownership between batches. Global operator scope also requires the administrator role. Disabling a user, logging out, or changing scope stops later history delivery. Invitation redemption remains single-use and transactional; malformed input returns a client error without consuming the invitation.

Telegram, Slack and Teams recheck that a pairing/binding still exists after awaited authorization and transport calls. Disconnecting or replacing a binding prevents its pending operation from installing a stale binding, emitting output or submitting input/approval actions.

Exchange publication and account-route revocation received additional defensive fixes. See the focused boundary follow-up (source access required) for the existing egress and Windows SSH safeguards, changed publication boundary and platform validation limits.

Validation and remaining boundaries

Additional process and protocol regressions

The client groups Exchange preparation processes, synchronization workers and their retries into one authorization generation. Revocation, withdrawal and shutdown cancel pending preparation and stop owned children with bounded escalation. A real Unison fixture reproduced an orphan after a controller 401 at the previous implementation; the corrected lifecycle terminates it and prevents delayed preparation from restarting it. Upload fixtures stream actual bytes, preserve their digest and publish a private atomic status file.

Control Room's Unix-socket transport validates frame shapes, canonical base64 and output limits, owns stream errors and preserves split UTF-8 characters. Malformed frames and oversized decoded output previously caused uncaught exceptions. Invalid JSON now returns a generic diagnostic rather than echoing a potentially private output prefix. Completion verification encodes bounded stdout/stderr before transporting them as JSON; accepted control characters cannot expand beyond the internal response bound and be sliced into invalid JSON. Real subprocess/socket/UID fixtures cover these cases.

Context-tool cache installation requires regular, single-link executable and license files. A checksummed archive containing a binary symlink previously made chmod widen an unrelated private fixture from 0600 to 0755. The new file type check rejects it before permission changes or cache publication.

Session credential restoration requires a confirmed stopped provider process. An unsuccessful quiescence result cannot proceed to swapping the account home. MCP broker revision changes close only that broker's accepted connections before waiting for its listener to close. Another session's broker remains available.

Shared console regressions cover cancelled credential-copy acknowledgements, duplicate grant submission and keyboard ownership. A run's park/retry/wake/result shortcut no longer opens another Projects action. Voice settings accept Escape while loading or saving, and replacing a credential clears its removal flag. Exchange's DOM tests transfer real chunk bytes over loopback HTTP, verify authorization headers and cancel active XHRs. Closing during an initial folder read cannot recreate a background polling timer; an older queued close cannot invalidate the reopened dialog.

Both Chrome gateway distributions validate HTTP request targets and raw URL encoding before processing them. Background discovery has the same 2 MiB response budget as ordinary forwarding, and malformed CDP frames cannot terminate the process. New targets require a confirmed nonempty ID. Disconnecting during an upstream WebSocket handshake terminates it, drops pending messages and fences late open callbacks; an already-cancelled command cannot run after detachment. HTTP forwarding owns upstream/downstream errors, aborts oversized bodies immediately and enforces a three-second response deadline. Actual raw TCP, HTTP and delayed WebSocket tests reproduced crashes, late command delivery, connection leaks and a truncated-response hang before the corrections. The combined gateway/idle suite passes 53 tests. These source changes do not replace a running gateway service automatically.

The Windows build helper's offline boundary tests also run actual local subprocesses and JSON-RPC fixtures. They exercise timeout reaping, stream closure, malformed records, publication validation and cleanup. They are not proof of a native Windows installer or application UI run.

Regression coverage includes 10,000 archived records with only a few active records, restart and insertion-order behavior, tenant-scoped history, concurrent job admission, credential rotation and deletion, and generated-file symlinks. The TUI idle-memory check retains its 128 MiB heap limit and 6,000 polling cycles. Test fixtures register provider-specific accounts explicitly; production account validation is not relaxed to accommodate old fixtures.

Follow-up scaling fixtures hold one user's inventory constant while adding 2,000 other-user sessions, 6,000 other-user history bodies and 4,000 unrelated owned command/result references. The selected user's full snapshot uses 13 prepared statements in both cases and decodes zero foreign historical bodies. Latest-run selection uses one prepared statement for both 10 and 1,000 sessions; the query plan uses the existing session/time index. These measurements count fixture query work, not production latency or a capacity guarantee. A separate 10,000-blocked-job fixture verifies bounded scheduler work and eligible progress.

The broad validation pass also exposed baseline fixture problems. Synthetic Git review fixtures now provide a repository remote; read-only folder tests assert an authorization denial without queued writes; navigation persistence checks ignore unrelated maintenance writes; empty-tab tests match the configured hide-empty behavior. Asynchronous terminal fixtures wait for rendered state instead of fixed sleeps or raw ANSI fragments. Memory stress tests keep their 128 MiB heaps, iteration counts and growth bounds; only their bounded wall-time allowance accommodates competing CI workers. Public-link metadata tests retain each update's deadline while allowing time for all three polling updates.

Standard product checks are pnpm lint, pnpm typecheck, pnpm build and pnpm test, including the promoted shared fixture package. Website checks run in the independent authorized landing checkout. CI limits Vitest to two workers and runs at most two workspace test scripts concurrently. Tests requiring live credentials remain opt-in; these changes were checked with disposable local identities and data, without restarting active workspace sessions.

The earlier 0.1.11 remediation passed repository lint, full workspace typecheck/build, the release metadata check and these local package checks:

Package Broad run and follow-up result
Runtime 957 passed; 9 integration tests skipped.
Controller 1,682 passed initially; all 17 failures passed after baseline fixture corrections and isolated timing rechecks. Final changed/regression selection: 137 passed.
Shared console 1,362 passed initially; all 15 failures passed after fixture corrections and isolated rechecks. Final interface selection: 96 passed; memory/Hidden selection: 10 passed.

These are combined broad and focused results, not a claim that a single root pnpm test invocation ran green. The first broad runs overlapped compilation and hit real-time deadlines; affected cases were rerun without compilation. Baseline assertion failures were reproduced at the pre-remediation commit before their fixtures changed. No paid-provider, real Docker or production endpoint checks were run.

The 0.1.12 persistence, memory-admission and tenant-read follow-up passed a complete 20-workspace collection: 6,263 tests passed and 42 were explicitly skipped. That includes 1,862 controller tests, 1,418 shared-console tests, 728 host-agent tests and 957 runtime tests. The standalone collection passed 186 Node suites; the updated Python collection passed 196 tests with 13 Linux-only skips, and a disposable Linux transport run passed 17 tests. Whole-repository lint, typecheck and build passed. The regenerated landing passed 145 tests.

Subsequent cleanup/socket/Graphify race corrections have targeted regressions. Their changed-source counters are not borrowed from the earlier collection. See repository coverage (source access required) for the incomplete measured baseline, the whole-repository 95% target and remaining instrumentation boundaries. Test counts and successful builds are not coverage percentages or a passing gate.

The 0.1.12 controller and web checkpoint was deployed with a consistent private copy of its database, external result journals and installation keys. Both services passed health checks, both execution hosts reconnected, and all 176 other running containers retained their identities. The controller reported 18 projects and 66 sessions afterward. This deployment did not replace the host agent or restart workspace containers; subsequent host fixes require their own rollout.

The 0.1.13 follow-up passed a complete 20-workspace matrix with one CLI integration retry: 6,659 passed and 73 skipped. Its standalone Node matrix passed 1,070 tests with 29 skips after repeating the unchanged full-locale suite with the coordinator's 300-second collection budget. Linux Python passed 235 tests. Production source fingerprints match across the corresponding retries; the original failures remain in the artifacts. Additional actual Linux host/runtime checks and shared-console interaction regressions are described above. Typecheck, build and lint passed. These are source/local validation results; the live rollout described above remains the earlier 0.1.12 checkpoint. The whole-repository 95% coverage target is still open.

The 0.1.15 follow-up replaces the Accounts pipe-delimited prompt with a shared agent picker and name field. Generated profile IDs survive retries. Its real DOM/Ink tests also reproduced an owned 750 ms refresh firing after unmount; cleanup now cancels that timer and ignores late responses. Historical startup cancellation is rechecked after the host accepts a start and after the final ready-state read, so a late response cannot attach a cancelled launch.

Bootstrap account import treats JWT display claims as untrusted metadata. JSON arrays, scalars and null no longer cause an attribute error; the original authorization is preserved. The complete refreshed Linux Python collection passes 251 tests with no skips and matching source fingerprints.

Desktop SSH setup keeps its Python helper under a scoped process owner. An invalid UTF-8 response previously returned an error while leaving the helper running. A real child regression reproduced that leak; early I/O and protocol returns now terminate and reap the child. Nine native Rust cases pass on macOS ARM64. This does not establish native Windows application behavior.

This work does not establish a capacity guarantee or diagnose a prior production incident. Small metadata transactions remain synchronous on the controller's single SQLite writer. Production result payload I/O and expensive maintenance run asynchronously on the worker; synchronous internal compatibility methods remain available for explicit local callers and in-memory fixtures. Resource pressure can reject expensive operations, and callers must retain their saved request/result for retry. This is preferable to losing authoritative state on an out-of-memory crash.