Linux server resource adapters
Resource accounting is independent from account sharing: a shared Claude/Codex subscription pools provider quota, while compute reservations remain attached to each user's own physical workspaces on each host. Sharing an external service does not merge tenant containers or resource envelopes. Shared subscriptions (source access required), feature boundaries.
Resource admission and measured use are separate. The controller SQLite ledger
reserves RAM, CPU, PIDs, concurrency and retained disk for each user/host. A host
must additionally verify the actual enforcement before a hard allocation can run.
These adapters are optional and perform no host mutations without an explicit
administrator-owned DEVCTL_SERVER_CGROUP_CONFIG file.
Compute prerequisites
All profiles require Linux cgroup v2, delegated memory, cpu and pids
controllers, an empty administrator-prepared parent, and a Docker driver that
actually places every owned workspace beneath its user parent. An inert probe
uses an already installed, administrator-selected image, --pull=never, no
network, a read-only root filesystem and dropped capabilities. The host checks
its physical PID/cgroup through the real host proc mount.
For rootful Docker with its cgroupfs driver:
{
"mount": "/sys/fs/cgroup",
"delegatedRoot": "devctrl",
"rootlessDelegation": false,
"placementDriver": "cgroupfs",
"procRoot": "/host-proc",
"probeImage": "devctl:remote"
}
The devctrl parent must exist on the real host hierarchy, have the required
controllers available, and contain no direct processes. The adapter creates a
hashed per-application-user child, writes memory.max, cpu.max, and pids.max,
then reads them back and checks every owned container and descendant process.
Privileged DinD, host PID/user/cgroup namespace access, dangerous added
capabilities, host runtime sockets and writable host cgroup mounts are rejected.
For rootless Docker, a cgroupfs path alone is insufficient. Use its systemd driver and an actual delegated systemd user manager:
{
"mount": "/sys/fs/cgroup",
"delegatedRoot": "user.slice/user-1000.slice/user@1000.service/devctrl.slice",
"rootlessDelegation": true,
"placementDriver": "systemd",
"systemd": {
"user": true,
"parentSlice": "devctrl.slice",
"delegationUnit": "user@1000.service"
},
"procRoot": "/host-proc",
"dataRoot": "/home/devctl/.local/share/devctl",
"probeImage": "devctl:remote"
}
Those paths and UID are examples: use the actual ControlGroup values. The
administrator prepares devctrl.slice under the intended delegation. The
adapter checks the parent slice and system service's Delegate and ControlGroup
properties, creates a per-user slice using a transient empty anchor service,
and verifies its real ControlGroup plus Docker's measured placement. Anchors
remain active with no process until the administrator retires that allocation;
they do not survive a manager restart and are recreated when needed.
The managed host-agent must run as UID 0 inside the matching Docker user namespace, not as an ordinary native SSH UID. Its mapping must cover all container account IDs through 49999, including interchange GID 19999, and provide CAP_CHOWN. A real temporary-file chown probe verifies the data binding. The probe container's UID/GID maps must match the host-agent's maps. This also applies to rootful Docker configured with userns-remap.
Mount the host data directory at the identical absolute path and set
DEVCTL_DATA_DIR and the config's dataRoot to that path. Bind the actual host
/proc at procRoot, expose the real delegated cgroup hierarchy and appropriate
systemd user bus, and set an explicit DOCKER_HOST=unix:///... to the same mounted
rootless socket used by Dockerode. Unset DOCKER_CONTEXT; CLI-only context
selection does not configure Dockerode. The trusted host-agent may use the host
cgroup namespace; managed user workspaces may not. Required mounts and delegation
are installation prerequisites, not permissions granted to agent workloads.
A namespace-root process may be unable to authenticate directly to the native
host user's bus: the client-side UID and the bus's external authentication UID
can differ across user namespaces. The adapter must successfully execute real
systemctl --user queries and create/read back its slice; namespace arithmetic
is not a bus-access proof. When direct access is unavailable, an administrator
can set systemd.commandWrapper to a trusted absolute executable that forwards
only the fixed systemctl/systemd-run argument vector to an operator-provisioned
native-host user-bus bridge. The bridge and wrapper must be outside workload
mounts and inaccessible to application users. They are not automatically
installed, and no sudo escalation is attempted. Missing bridge/bus access fails
closed. The physical integration test exercises this combination; mocked tests
do not demonstrate that a particular host's bus accepts namespace-root.
A missing user bus, incomplete delegation, native ordinary-UID host-agent, unmapped account IDs, missing CAP_CHOWN, mismatched physical PID views, different Docker namespaces, or an unwritable data binding prevents hard certification.
The effective admission pool is the minimum of physical host capacity and every
ancestor's memory.max, CPU bandwidth and pids.max. A 2 GiB delegated parent is
not a 32 GiB execution pool merely because the physical server has 32 GiB RAM.
The adapter reports effective limits and usage at the limiting memory/PID
ancestor, including infrastructure processes outside the per-user subtree;
allocations exceeding an
ancestor ceiling fail closed. CPU/RAM maxima are upper limits, not dedicated
performance guarantees.
User workspace storage and CPU affinity
The server administrator configures each user's allocation for each execution host
in Server Users → user → Hosts and resource budgets. The first screen lists
all users, execution hosts, effective workspace directories, CPU sets and CPU/disk
budgets. An allocation may set workspaceStorageBase and cpuSet in addition to
the existing aggregate RAM, CPU, PID, disk and container budgets.
Storage bases belong to the execution server. Add optional
workspaceStorageBases: ["/srv/devctrl-workspaces", "/mnt/workspaces"] to the
trusted host capability configuration; the canonical DEVCTL_DATA_DIR remains
available. Provision each base first, with administrator ownership and no
group/world write access. When host-agent runs in Docker, mount each base
read-write at the identical absolute path on the host and inside host-agent.
The host validates physical mount visibility, real paths, ownership and every
tenant directory before attesting placement. Symlink aliases and unsafe shared
parents fail closed. Repository-controlled configuration cannot choose these roots.
New workspaces use <base>/users/<hashed-user>/workspaces/<workspace-id>/, with
separate repository and runtime directories. A changed default applies to
unplaced/new workspaces. Every placed workspace keeps its recorded directory,
including stopped workspaces with retained data; the overview lists those roots.
No files are moved and no session/container is restarted. Accounts, credentials
and file exchange keep their existing private controller-managed locations. Keep
retained bases mounted and in the host allowlist until their workspaces are retired;
removing an old base prevents a retained container start rather than moving its data.
There is no implicit storage migration operation.
A positive hard disk budget additionally requires all custom and retained roots to be included in the same user's administrator-provisioned filesystem project quota. Changing the workspace default does not create a filesystem quota or exclude old data from accounting. Missing coverage blocks hard starts and writes.
cpuSet, for example 2-3,7, selects actual CPU IDs from the delegated parent's
cpuset.cpus.effective, not indices into a CPU count. Affinity requires hard mode
and the cpuset controller in addition to the compute prerequisites above. The
host enables delegation, writes cpuset.cpus and cpuset.mems, reads back effective
placement, and still applies the user's aggregate cpu.max bandwidth cap. The CPU
quota cannot exceed the number of selected CPUs. Ancestor CPU sets also narrow
effective host capacity.
Affinity is a user/host boundary: all that user's containers and their sessions, including multiple sessions sharing one container, inherit it. An explicit administrator change updates the existing cgroup without killing/restarting sessions. Leaving the affinity field blank restores the delegated CPU set. CPU sets may overlap between users; this is CPU affinity, not exclusive-core reservation or guaranteed performance. Unsupported delegation, missing physical placement, stale proofs and generation mismatches block new work.
The API exposes administrator overview at GET /v1/server/users/overview and
allocation updates at PUT /v1/server/users/:id/hosts/:hostId/allocation. Only the
assigned authenticated execution host may attest placement/enforcement. Ordinary
users can read their own budgets, but cannot change quotas or select arbitrary
host paths. The recovery operator retains its legacy workspace behavior; configure
a regular server user for controlled tenant placement.
Disk prerequisites
Cgroups do not limit filesystem space. Configure a real XFS/ext4 filesystem with project quotas enabled and administrator-reserved project IDs. One project ID belongs to one application user; its roots cannot overlap another user's roots. All owned writable storage must be covered, including Docker upper/work layers, container logs, writable volumes/binds, retained stopped-container data, and private Git credential storage even when mounted read-only in the container. Shared immutable image layers are infrastructure storage.
Add this optional object to the compute configuration:
{
"diskQuota": {
"mount": "/srv",
"filesystem": "xfs",
"manage": true,
"dockerStorageRoots": ["/srv/docker"],
"users": {
"alice": {
"projectId": 240001,
"roots": ["/srv/devctl/users/<user-id>"]
}
}
}
}
Replace these roots with actual immutable user storage paths. A quota only on the
checkout directory is insufficient when writable Docker layers live under an
uncovered shared daemon directory. dockerStorageRoots permits dynamic coverage
of the actual exclusively owned container upper/work trees, log directories and
label-verified private volumes. Only those private directories receive the user's
project ID; the whole shared Docker pool does not. Cross-user private-path
ownership collisions, overlapping roots and unverified volume ownership fail
closed. Stopped-container private storage remains included. A dedicated per-user
Docker tree also works.
Roots must already exist. manage: false verifies administrator-prepared quotas
without changing them. manage: true requires actual initial-namespace host
permissions to assign inode project IDs and set quota limits; namespace-root alone
does not grant those host administrative permissions. Where needed, configure
diskQuota.commandWrapper as a trusted absolute executable forwarding the fixed
quota tool argument vectors and the fixed project-assignment helper to an
operator-provisioned initial-namespace helper. Python 3 is required for assignment.
That helper must validate allowed paths/project IDs and must be outside workload
mounts. Missing helper permissions fail closed. No automatic sudo is attempted.
Assignment on both filesystems uses a fixed Python helper with directory-relative
O_NOFOLLOW file descriptors and Linux FS_IOC_FSGETXATTR/FS_IOC_FSSETXATTR.
It sets project IDs and directory inheritance, rejects mount crossings, verifies
the inode identity and reads back every change. It refuses foreign project IDs
and unassigned hardlinks whose links are not all inside that user's owned roots.
It never performs broad recursive chattr or follows symlink targets. A privileged
bridge must whitelist this exact helper rather than permit arbitrary Python scripts.
Existing unassigned storage must be quiescent during administrative migration:
an inode's link-count check and its attribute ioctl are separate operations.
Normal background reconciliation refuses unassigned existing regular files.
Only the approved pending workspace creation path may assign the newly created
container's exclusive private Docker roots before any agent starts, while the
controller enforcement proof remains invalid. Existing user bind trees receive
no exception; populate them only after administrator-prepared inheritance.
After the first complete verification, a private provenance journal records the
root device/inode identities, verified project/inheritance policy and conservative
legacy symlink debit. It defaults to control/user-quota-provenance.json below the
host data directory; diskQuota.provenanceJournal may select other private host
storage outside all workspace/Docker roots. Never share that directory with agents.
Subsequent reconciliation checks each root's device/inode, project ID and inheritance
plus real kernel quota accounting; it does not walk every dependency again. This
optimization requires the actual non-initial user namespace boundary. Kernel
inheritance and cross-project link/rename restrictions then preserve attribution.
Replaced roots or changed project/inheritance flags fail closed. The conservative
initial symlink debit remains until a quiescent administrator rescan. Receipts
survive restart; there is no TTL-based substitution for actual enforcement checks.
XFS verification uses xfs_quota project -c, limit -p, state -p and numeric
block reports. It requires both accounting and enforcement, verifies every tree,
and reads back the actual hard limit. ext4 uses setquota -P, quotaon -p -P,
numeric CSV repquota, and inode checks with lsattr -p -d.
Symlinks are walked physically without following their targets. Their
allocated blocks are deduplicated by device/inode and conservatively debited from
the effective hard limit, even when already attributed to the project. Link
identity/size changes during verification fail closed and require a retry.
Regular inode IDs and directory inheritance remain mandatory. In a non-initial
user namespace agents cannot change project IDs/inheritance or move/link foreign
project inodes into the tree. A preexisting symlink cannot grow in place; replacing
it creates a new inode beneath an inherited project quota. Thus kernel quota plus
the conservative link debit bounds the owned tree; the debit can reduce usable
headroom. The UI distinguishes filesystem-quota+conservative-symlinks from pure
quota measurement. No unsupported tree receives a hard proof.
Limits are filesystem-accounted blocks, rounded down to whole KiB. Quota-reported use and the conservative link bound are reported separately; their sum is an upper bound, not exact logical file sizes or Docker image totals.
Hard project quotas require workloads in a non-initial user namespace, such as rootless Docker or userns-remap. On current Linux, initial-user-namespace file owners can change project quota IDs/inheritance. Dropping SYS_ADMIN alone does not establish the required boundary. The adapter checks physical owned workload UID maps and rejects the initial namespace for a hard disk proof.
Before cloning or storing a private Git credential, hard disk workspace creation invalidates the previous proof and verifies prepared storage. After creation, all new Docker storage must be reverified before any provider starts. Failed checks invalidate the controller proof immediately. Existing files remain; reducing a quota below currently occupied data requires an explicit drain. Disk measurements go to the aggregate user/host ledger and are not added again to workspace bind totals.
Configured hosts refresh enforcement through a separate nonoverlapping maintenance pump every ten seconds, independently of long lifecycle/image-build commands. Controller requests have a twenty-second maintenance deadline. Each allocation must freshly authenticate the same active controller authority before changing kernel policy and before publishing its proof; draining, fenced, standby and changed-epoch sources pause maintenance. Host inventory carries the timestamp of its required container/process observation, before optional nested accounting. Delayed delivery or fresh transport/host-metric samples cannot freshen that inventory evidence. Optional nested accounting has an eight-second deadline and reports unavailable usage on timeout; admission and termination keep their existing inventory freshness requirements. The process probes overlap primary container reads after storage collection. Completed readings older than 30 seconds are discarded before merging, leaving nested workload usage unknown. Nested HTTP accounting has a total eight-second deadline and does not publish partial results. Slow required storage can still delay an inventory cycle.
Verification
Fixture tests mock kernel files and tools. They do not modify real hosts. An opt-in integration test uses an explicitly prepared isolated host and config:
DEVCTL_RESOURCE_INTEGRATION_CONFIG=/private/isolated-cgroups.json \
DEVCTL_RESOURCE_INTEGRATION_ALLOCATION=/private/isolated-allocation.json \
pnpm --filter @devctl/host-agent exec vitest run src/server-user-resources.integration.test.ts
The allocation file contains userId, hostId, generation, mode: "hard", and
limits with ramBytes, cpuCores, pids, and diskBytes. This test performs
actual configured local operations; it is skipped in normal CI. No real-host
integration run or production migration is implied by fixture tests.
Sources
- Docker rootless prerequisites
- Docker rootless resource limits
- systemd-run options
- systemd delegation and resource control
- XFS project quotas and verification
- setquota project limits
- quotaon print-state
- quota-tools CSV implementation
- Linux project-ID/inheritance namespace restriction
- Linux project attribute ioctl ABI