Linux server resource adapters

Resource accounting is independent from account sharing: a shared Claude/Codex subscription pools provider quota, while compute reservations remain attached to each user's own physical workspaces on each host. Sharing an external service does not merge tenant containers or resource envelopes. Shared subscriptions (source access required), feature boundaries.

Resource admission and measured use are separate. The controller SQLite ledger reserves RAM, CPU, PIDs, concurrency and retained disk for each user/host. A host must additionally verify the actual enforcement before a hard allocation can run. These adapters are optional and perform no host mutations without an explicit administrator-owned DEVCTL_SERVER_CGROUP_CONFIG file.

Compute prerequisites

All profiles require Linux cgroup v2, delegated memory, cpu and pids controllers, an empty administrator-prepared parent, and a Docker driver that actually places every owned workspace beneath its user parent. An inert probe uses an already installed, administrator-selected image, --pull=never, no network, a read-only root filesystem and dropped capabilities. The host checks its physical PID/cgroup through the real host proc mount.

For rootful Docker with its cgroupfs driver:

{
  "mount": "/sys/fs/cgroup",
  "delegatedRoot": "devctrl",
  "rootlessDelegation": false,
  "placementDriver": "cgroupfs",
  "procRoot": "/host-proc",
  "probeImage": "devctl:remote"
}

The devctrl parent must exist on the real host hierarchy, have the required controllers available, and contain no direct processes. The adapter creates a hashed per-application-user child, writes memory.max, cpu.max, and pids.max, then reads them back and checks every owned container and descendant process. Privileged DinD, host PID/user/cgroup namespace access, dangerous added capabilities, host runtime sockets and writable host cgroup mounts are rejected.

For rootless Docker, a cgroupfs path alone is insufficient. Use its systemd driver and an actual delegated systemd user manager:

{
  "mount": "/sys/fs/cgroup",
  "delegatedRoot": "user.slice/user-1000.slice/user@1000.service/devctrl.slice",
  "rootlessDelegation": true,
  "placementDriver": "systemd",
  "systemd": {
    "user": true,
    "parentSlice": "devctrl.slice",
    "delegationUnit": "user@1000.service"
  },
  "procRoot": "/host-proc",
  "dataRoot": "/home/devctl/.local/share/devctl",
  "probeImage": "devctl:remote"
}

Those paths and UID are examples: use the actual ControlGroup values. The administrator prepares devctrl.slice under the intended delegation. The adapter checks the parent slice and system service's Delegate and ControlGroup properties, creates a per-user slice using a transient empty anchor service, and verifies its real ControlGroup plus Docker's measured placement. Anchors remain active with no process until the administrator retires that allocation; they do not survive a manager restart and are recreated when needed.

The managed host-agent must run as UID 0 inside the matching Docker user namespace, not as an ordinary native SSH UID. Its mapping must cover all container account IDs through 49999, including interchange GID 19999, and provide CAP_CHOWN. A real temporary-file chown probe verifies the data binding. The probe container's UID/GID maps must match the host-agent's maps. This also applies to rootful Docker configured with userns-remap.

Mount the host data directory at the identical absolute path and set DEVCTL_DATA_DIR and the config's dataRoot to that path. Bind the actual host /proc at procRoot, expose the real delegated cgroup hierarchy and appropriate systemd user bus, and set an explicit DOCKER_HOST=unix:///... to the same mounted rootless socket used by Dockerode. Unset DOCKER_CONTEXT; CLI-only context selection does not configure Dockerode. The trusted host-agent may use the host cgroup namespace; managed user workspaces may not. Required mounts and delegation are installation prerequisites, not permissions granted to agent workloads.

A namespace-root process may be unable to authenticate directly to the native host user's bus: the client-side UID and the bus's external authentication UID can differ across user namespaces. The adapter must successfully execute real systemctl --user queries and create/read back its slice; namespace arithmetic is not a bus-access proof. When direct access is unavailable, an administrator can set systemd.commandWrapper to a trusted absolute executable that forwards only the fixed systemctl/systemd-run argument vector to an operator-provisioned native-host user-bus bridge. The bridge and wrapper must be outside workload mounts and inaccessible to application users. They are not automatically installed, and no sudo escalation is attempted. Missing bridge/bus access fails closed. The physical integration test exercises this combination; mocked tests do not demonstrate that a particular host's bus accepts namespace-root.

A missing user bus, incomplete delegation, native ordinary-UID host-agent, unmapped account IDs, missing CAP_CHOWN, mismatched physical PID views, different Docker namespaces, or an unwritable data binding prevents hard certification.

The effective admission pool is the minimum of physical host capacity and every ancestor's memory.max, CPU bandwidth and pids.max. A 2 GiB delegated parent is not a 32 GiB execution pool merely because the physical server has 32 GiB RAM. The adapter reports effective limits and usage at the limiting memory/PID ancestor, including infrastructure processes outside the per-user subtree; allocations exceeding an ancestor ceiling fail closed. CPU/RAM maxima are upper limits, not dedicated performance guarantees.

User workspace storage and CPU affinity

The server administrator configures each user's allocation for each execution host in Server Users → user → Hosts and resource budgets. The first screen lists all users, execution hosts, effective workspace directories, CPU sets and CPU/disk budgets. An allocation may set workspaceStorageBase and cpuSet in addition to the existing aggregate RAM, CPU, PID, disk and container budgets.

Storage bases belong to the execution server. Add optional workspaceStorageBases: ["/srv/devctrl-workspaces", "/mnt/workspaces"] to the trusted host capability configuration; the canonical DEVCTL_DATA_DIR remains available. Provision each base first, with administrator ownership and no group/world write access. When host-agent runs in Docker, mount each base read-write at the identical absolute path on the host and inside host-agent. The host validates physical mount visibility, real paths, ownership and every tenant directory before attesting placement. Symlink aliases and unsafe shared parents fail closed. Repository-controlled configuration cannot choose these roots.

New workspaces use <base>/users/<hashed-user>/workspaces/<workspace-id>/, with separate repository and runtime directories. A changed default applies to unplaced/new workspaces. Every placed workspace keeps its recorded directory, including stopped workspaces with retained data; the overview lists those roots. No files are moved and no session/container is restarted. Accounts, credentials and file exchange keep their existing private controller-managed locations. Keep retained bases mounted and in the host allowlist until their workspaces are retired; removing an old base prevents a retained container start rather than moving its data. There is no implicit storage migration operation.

A positive hard disk budget additionally requires all custom and retained roots to be included in the same user's administrator-provisioned filesystem project quota. Changing the workspace default does not create a filesystem quota or exclude old data from accounting. Missing coverage blocks hard starts and writes.

cpuSet, for example 2-3,7, selects actual CPU IDs from the delegated parent's cpuset.cpus.effective, not indices into a CPU count. Affinity requires hard mode and the cpuset controller in addition to the compute prerequisites above. The host enables delegation, writes cpuset.cpus and cpuset.mems, reads back effective placement, and still applies the user's aggregate cpu.max bandwidth cap. The CPU quota cannot exceed the number of selected CPUs. Ancestor CPU sets also narrow effective host capacity.

Affinity is a user/host boundary: all that user's containers and their sessions, including multiple sessions sharing one container, inherit it. An explicit administrator change updates the existing cgroup without killing/restarting sessions. Leaving the affinity field blank restores the delegated CPU set. CPU sets may overlap between users; this is CPU affinity, not exclusive-core reservation or guaranteed performance. Unsupported delegation, missing physical placement, stale proofs and generation mismatches block new work.

The API exposes administrator overview at GET /v1/server/users/overview and allocation updates at PUT /v1/server/users/:id/hosts/:hostId/allocation. Only the assigned authenticated execution host may attest placement/enforcement. Ordinary users can read their own budgets, but cannot change quotas or select arbitrary host paths. The recovery operator retains its legacy workspace behavior; configure a regular server user for controlled tenant placement.

Disk prerequisites

Cgroups do not limit filesystem space. Configure a real XFS/ext4 filesystem with project quotas enabled and administrator-reserved project IDs. One project ID belongs to one application user; its roots cannot overlap another user's roots. All owned writable storage must be covered, including Docker upper/work layers, container logs, writable volumes/binds, retained stopped-container data, and private Git credential storage even when mounted read-only in the container. Shared immutable image layers are infrastructure storage.

Add this optional object to the compute configuration:

{
  "diskQuota": {
    "mount": "/srv",
    "filesystem": "xfs",
    "manage": true,
    "dockerStorageRoots": ["/srv/docker"],
    "users": {
      "alice": {
        "projectId": 240001,
        "roots": ["/srv/devctl/users/<user-id>"]
      }
    }
  }
}

Replace these roots with actual immutable user storage paths. A quota only on the checkout directory is insufficient when writable Docker layers live under an uncovered shared daemon directory. dockerStorageRoots permits dynamic coverage of the actual exclusively owned container upper/work trees, log directories and label-verified private volumes. Only those private directories receive the user's project ID; the whole shared Docker pool does not. Cross-user private-path ownership collisions, overlapping roots and unverified volume ownership fail closed. Stopped-container private storage remains included. A dedicated per-user Docker tree also works.

Roots must already exist. manage: false verifies administrator-prepared quotas without changing them. manage: true requires actual initial-namespace host permissions to assign inode project IDs and set quota limits; namespace-root alone does not grant those host administrative permissions. Where needed, configure diskQuota.commandWrapper as a trusted absolute executable forwarding the fixed quota tool argument vectors and the fixed project-assignment helper to an operator-provisioned initial-namespace helper. Python 3 is required for assignment. That helper must validate allowed paths/project IDs and must be outside workload mounts. Missing helper permissions fail closed. No automatic sudo is attempted.

Assignment on both filesystems uses a fixed Python helper with directory-relative O_NOFOLLOW file descriptors and Linux FS_IOC_FSGETXATTR/FS_IOC_FSSETXATTR. It sets project IDs and directory inheritance, rejects mount crossings, verifies the inode identity and reads back every change. It refuses foreign project IDs and unassigned hardlinks whose links are not all inside that user's owned roots. It never performs broad recursive chattr or follows symlink targets. A privileged bridge must whitelist this exact helper rather than permit arbitrary Python scripts. Existing unassigned storage must be quiescent during administrative migration: an inode's link-count check and its attribute ioctl are separate operations. Normal background reconciliation refuses unassigned existing regular files. Only the approved pending workspace creation path may assign the newly created container's exclusive private Docker roots before any agent starts, while the controller enforcement proof remains invalid. Existing user bind trees receive no exception; populate them only after administrator-prepared inheritance.

After the first complete verification, a private provenance journal records the root device/inode identities, verified project/inheritance policy and conservative legacy symlink debit. It defaults to control/user-quota-provenance.json below the host data directory; diskQuota.provenanceJournal may select other private host storage outside all workspace/Docker roots. Never share that directory with agents. Subsequent reconciliation checks each root's device/inode, project ID and inheritance plus real kernel quota accounting; it does not walk every dependency again. This optimization requires the actual non-initial user namespace boundary. Kernel inheritance and cross-project link/rename restrictions then preserve attribution. Replaced roots or changed project/inheritance flags fail closed. The conservative initial symlink debit remains until a quiescent administrator rescan. Receipts survive restart; there is no TTL-based substitution for actual enforcement checks.

XFS verification uses xfs_quota project -c, limit -p, state -p and numeric block reports. It requires both accounting and enforcement, verifies every tree, and reads back the actual hard limit. ext4 uses setquota -P, quotaon -p -P, numeric CSV repquota, and inode checks with lsattr -p -d. Symlinks are walked physically without following their targets. Their allocated blocks are deduplicated by device/inode and conservatively debited from the effective hard limit, even when already attributed to the project. Link identity/size changes during verification fail closed and require a retry. Regular inode IDs and directory inheritance remain mandatory. In a non-initial user namespace agents cannot change project IDs/inheritance or move/link foreign project inodes into the tree. A preexisting symlink cannot grow in place; replacing it creates a new inode beneath an inherited project quota. Thus kernel quota plus the conservative link debit bounds the owned tree; the debit can reduce usable headroom. The UI distinguishes filesystem-quota+conservative-symlinks from pure quota measurement. No unsupported tree receives a hard proof.

Limits are filesystem-accounted blocks, rounded down to whole KiB. Quota-reported use and the conservative link bound are reported separately; their sum is an upper bound, not exact logical file sizes or Docker image totals.

Hard project quotas require workloads in a non-initial user namespace, such as rootless Docker or userns-remap. On current Linux, initial-user-namespace file owners can change project quota IDs/inheritance. Dropping SYS_ADMIN alone does not establish the required boundary. The adapter checks physical owned workload UID maps and rejects the initial namespace for a hard disk proof.

Before cloning or storing a private Git credential, hard disk workspace creation invalidates the previous proof and verifies prepared storage. After creation, all new Docker storage must be reverified before any provider starts. Failed checks invalidate the controller proof immediately. Existing files remain; reducing a quota below currently occupied data requires an explicit drain. Disk measurements go to the aggregate user/host ledger and are not added again to workspace bind totals.

Configured hosts refresh enforcement through a separate nonoverlapping maintenance pump every ten seconds, independently of long lifecycle/image-build commands. Controller requests have a twenty-second maintenance deadline. Each allocation must freshly authenticate the same active controller authority before changing kernel policy and before publishing its proof; draining, fenced, standby and changed-epoch sources pause maintenance. Host inventory carries the timestamp of its required container/process observation, before optional nested accounting. Delayed delivery or fresh transport/host-metric samples cannot freshen that inventory evidence. Optional nested accounting has an eight-second deadline and reports unavailable usage on timeout; admission and termination keep their existing inventory freshness requirements. The process probes overlap primary container reads after storage collection. Completed readings older than 30 seconds are discarded before merging, leaving nested workload usage unknown. Nested HTTP accounting has a total eight-second deadline and does not publish partial results. Slow required storage can still delay an inventory cycle.

Verification

Fixture tests mock kernel files and tools. They do not modify real hosts. An opt-in integration test uses an explicitly prepared isolated host and config:

DEVCTL_RESOURCE_INTEGRATION_CONFIG=/private/isolated-cgroups.json \
DEVCTL_RESOURCE_INTEGRATION_ALLOCATION=/private/isolated-allocation.json \
pnpm --filter @devctl/host-agent exec vitest run src/server-user-resources.integration.test.ts

The allocation file contains userId, hostId, generation, mode: "hard", and limits with ramBytes, cpuCores, pids, and diskBytes. This test performs actual configured local operations; it is skipped in normal CI. No real-host integration run or production migration is implied by fixture tests.

Sources