Overnight server lifecycle
Overnight is a shutdown mode you explicitly enable before leaving a temporary cloud server unattended. Agents continue working after you disconnect. When every managed session on the host has finished or needs your input, devctrl turns the server off through its hosting adapter. You do not need to be online to issue the shutdown yourself; arming the mode expresses your intent to leave, rather than relying on automatic presence detection.
Workspace and session archival is the recovery feature that makes this shutdown safe: devctrl saves workspace data and native conversation histories to S3-compatible storage and verifies them before requesting power-off. Restore the saved work when you return. Archival remains required by the current shutdown workflow; presenting it separately does not make it optional.
This is separate from autonomous Agent Runs: Overnight manages the execution server's lifecycle. It does not invent tasks, reply to agents or choose a model.
Supported servers and boundaries
- The mode is provider-neutral: it uses an installed adapter's verified host identity and stop/resume capability, rather than a hardcoded provider name. The bundled Hetzner Cloud and DigitalOcean adapters currently supply that capability. Hostman, Linode and Vultr provisioning adapters do not yet supply Overnight power management; merely supporting VM creation does not enable it. A host must be explicitly linked to its real provider resource and marked temporary before Overnight can be armed.
- A regular SSH machine, a dedicated server or local Docker is not an eligible shutdown target merely because devctrl can connect to it.
- Boat disposable workspaces have a separate Delete when all sessions finish lifetime. It deletes the sandbox after all sessions have exited; waiting for input does not authorize deletion. The current Boat adapter intentionally disables snapshots and suspend/resume, so it is not a target for this VM power-off/restore workflow. See Boat lifetimes.
- The infrastructure operator configures and arms the host. Named server users, including tenant administrators, cannot shut down infrastructure shared with other users through this API.
- Power-off is not resource deletion. Providers can continue charging for the server, disks, IP addresses or archived objects while compute is stopped.
- The monitor runs on the controller. Closing TUI or the browser does not cancel an armed operation while that controller remains available.
Configure from TUI or web
Open Integrations → Overnight. Both interfaces use the same React panel and authenticated controller API.
- Provider token (
k): choose an installed stop/resume provider with the arrow keys and save its cloud API credential. It is encrypted on the controller and is never delivered to coding-agent sessions. - Configure S3 archive (
s): enter an HTTPS endpoint, bucket, region, object prefix and storage credentials. Private fields are masked and are sent only when the entire form is submitted. An optional base64-encoded 32-byte archive key can be provided; leaving it blank retains the saved key or generates one on first setup. Canceling the form discards its drafts. - When a configured hosting account supports automatic storage, choose
Configure storage automatically (
g). For DigitalOcean Spaces, choose the account, bucket and region, then explicitly choose an existing bucket or permit creation. The final confirmation creates bucket-scoped storage credentials and stores them in the controller; a hosting token is not copied to a session. Other providers require manually configured S3 credentials unless their adapter implements this capability. A DigitalOcean token with custom scopes needsspaces_key:read,spaces_key:createandspaces_key:deletefor setup and recovery of temporary setup keys. A compute-only token may need those scopes added first. The chosen Spaces location determines its endpoint; the adapter uses the documented AWS SDK signing regionus-east-1. - Link temporary server (
b): choose the provider and existing devctrl host, then enter the cloud server resource ID. devctrl requests a fresh identity probe from the authenticated host-agent and compares the cloud metadata ID (and public address where available) with the provider API resource. The provider-side name must also match the enrolled host alias. A matching name alone is insufficient, so a duplicate-name resource is rejected. Linking does not arm shutdown. - Select the linked server and choose Arm for tonight (
a). Use Disarm (d) to cancel the armed operation before shutdown.
Give archive credentials read/write access only to the selected bucket/prefix.
DigitalOcean Spaces setup selects the region through its HTTPS endpoint and uses
the official us-east-1 S3 signing region. Temporary bootstrap keys are revoked
after private bucket creation; crash recovery reconciles uncertain key creation
by its exact managed name and retries failed revocation.
Retain the encryption key and controller vault backup: an object-store copy
alone cannot decrypt encrypted archives if the original key has been lost.
Readiness and shutdown
The controller makes the decision; the client only displays it. One completed session never authorizes power-off while another is working. A parent waiting for input does not make its still-working subagents safe to interrupt.
| Observation | Overnight behavior |
|---|---|
| One session is finished, another is working | Keep the server on |
| All sessions have finished | Eligible for archival after readiness checks |
| All sessions are waiting for user input or permission | Eligible for archival; preserve their conversations for later replies |
| A mix of completed and waiting sessions | Eligible only when every session is settled |
| An active delegated task remains | Keep the server on |
| A session or its lifecycle observation is unknown/stale | Keep the server on; show a blocker |
| A session is attached, an autonomous Run is active, or another container workload is running | Keep the server on |
| Archive upload, checksum verification or index publication fails | Do not power off; report the failed stage |
| Cloud power-off fails | Retain the verified archive and report failure |
The controller reconciles every ten seconds and requires a continuous minute of
readiness before archival. The panel shows blockers, archive counts and operation stages: armed, waiting,
archiving, stopping, stopped, restoring, restored or failed. Overnight
is not a recurring shutdown schedule. A completed shutdown/restore disarms the
host; you opt in again for the next night.
Archives and recovery
S3 stores archives in a separate namespace for each owner:
<prefix>/owners/<owner-id>/index.json
<prefix>/owners/<owner-id>/<archive-run-id>/workspace.tar.aes256
Each versioned index.json catalog identifies the host and cloud resource,
archive IDs, timestamps, workspace IDs, native session IDs, root kinds, object
keys, byte counts and SHA-256 checksums. Catalog updates use conditional writes
so concurrent hosts do not silently overwrite each other's entries. Object checksums
are verified by reading back the uploaded data before the index marks an archive
available for restore. A failed or incomplete archive must not authorize power-off.
Choose Discover S3 archives (i) to read the current bucket's owner-scoped
indexes and import verified recovery metadata if its local controller receipt is
missing. Discovery is limited to this controller's known host/workspace owners;
an S3 index cannot create an arbitrary owner, server or destination path. Restore
reads a fresh catalog record instead of trusting the controller receipt alone.
Managed credential files and known secret paths, including .env, .agent.env,
agent authorization files and private secret directories, are excluded. Generated
dependencies/caches, sockets and symlinks are also excluded; install dependencies
again when the restored environment needs them. Existing runtime credential files
on the original disk are left untouched. Reauthorize expired accounts or redeliver
missing runtime secrets through the existing devctrl credential mechanisms.
Archive restore does not make an in-memory agent process or a browser
connection survive cloud power-off: resume uses saved native conversation context
on the restored workspace. Arbitrary repository files and conversation messages
are not scanned for secrets; archive encryption protects this potentially
sensitive project context.
Restore stages the archive and authenticates its encrypted data before promoting files to their original workspace and native-history roots. Fingerprint checks refuse to overwrite originals that changed unexpectedly after the saved snapshot.
If the controller and browser gateway run on the same server that stopped, their
API and public link are unavailable while it is off. Start that server using the
cloud provider or an independently running controller first. Reopen devctrl,
select the host in Integrations → Overnight and choose Restore saved work
(u). Select a saved coding session after the workspace is restored; an eligible
authorized account is required to resume it. A temporary public tunnel URL can
change when its gateway starts again; obtain the current console link instead of
assuming an old tunnel remains reachable.
The current UI restores to the same enrolled host under the same controller, with persistent cloud disks and the controller vault retained. If the controller stopped before recording provider confirmation, startup reconciliation makes the interrupted shutdown restore-ready without requesting power-off a second time. The portable archive engine accepts an encryption key and explicit owner/path mappings, but a new-controller/new-host recovery wizard is not exposed by this UI.
Keep the S3 catalog and encryption keys for as long as recovery is needed. This initial workflow restores saved work explicitly; it does not promise transparent live-process migration, automatic DNS recovery, external service backups or restoration of unmanaged databases and arbitrary host workloads.
Controller API
All configuration, lifecycle and archive endpoints require infrastructure operator authorization. Public responses include configured status and archive metadata, never storage or provider secret values.
| Method and route | Purpose |
|---|---|
GET /v1/overnight |
Redacted configuration, providers, eligible hosts, bindings and verified archives |
PUT /v1/overnight/archive |
Save manual S3 configuration and encrypted credentials |
GET /v1/overnight/storage-options |
Discover automatic storage setup supported by configured hosting accounts |
POST /v1/overnight/storage/automatic |
Configure existing storage or explicitly create it using the chosen adapter |
POST /v1/overnight/archives/discover |
Import verified recovery metadata from known owners' S3 indexes |
PUT /v1/overnight/providers/:id/credentials |
Save a cloud API token privately |
POST /v1/overnight/hosts |
Verify and link a temporary cloud resource to a host |
POST /v1/overnight/hosts/:hostId/arm |
Opt in to one shutdown cycle |
POST /v1/overnight/hosts/:hostId/disarm |
Cancel that cycle |
POST /v1/overnight/hosts/:hostId/restore |
Request verified archive restoration |
No production server is powered off by the normal test suite. Readiness tests use multiple-session and subagent fixtures; storage and cloud tests mock provider operations. The landing demo changes only local sample metadata and cannot call cloud APIs, write S3 objects or submit real credentials.