Overnight server lifecycle

Overnight is a shutdown mode you explicitly enable before leaving a temporary cloud server unattended. Agents continue working after you disconnect. When every managed session on the host has finished or needs your input, devctrl turns the server off through its hosting adapter. You do not need to be online to issue the shutdown yourself; arming the mode expresses your intent to leave, rather than relying on automatic presence detection.

Workspace and session archival is the recovery feature that makes this shutdown safe: devctrl saves workspace data and native conversation histories to S3-compatible storage and verifies them before requesting power-off. Restore the saved work when you return. Archival remains required by the current shutdown workflow; presenting it separately does not make it optional.

This is separate from autonomous Agent Runs: Overnight manages the execution server's lifecycle. It does not invent tasks, reply to agents or choose a model.

Supported servers and boundaries

Configure from TUI or web

Open Integrations → Overnight. Both interfaces use the same React panel and authenticated controller API.

  1. Provider token (k): choose an installed stop/resume provider with the arrow keys and save its cloud API credential. It is encrypted on the controller and is never delivered to coding-agent sessions.
  2. Configure S3 archive (s): enter an HTTPS endpoint, bucket, region, object prefix and storage credentials. Private fields are masked and are sent only when the entire form is submitted. An optional base64-encoded 32-byte archive key can be provided; leaving it blank retains the saved key or generates one on first setup. Canceling the form discards its drafts.
  3. When a configured hosting account supports automatic storage, choose Configure storage automatically (g). For DigitalOcean Spaces, choose the account, bucket and region, then explicitly choose an existing bucket or permit creation. The final confirmation creates bucket-scoped storage credentials and stores them in the controller; a hosting token is not copied to a session. Other providers require manually configured S3 credentials unless their adapter implements this capability. A DigitalOcean token with custom scopes needs spaces_key:read, spaces_key:create and spaces_key:delete for setup and recovery of temporary setup keys. A compute-only token may need those scopes added first. The chosen Spaces location determines its endpoint; the adapter uses the documented AWS SDK signing region us-east-1.
  4. Link temporary server (b): choose the provider and existing devctrl host, then enter the cloud server resource ID. devctrl requests a fresh identity probe from the authenticated host-agent and compares the cloud metadata ID (and public address where available) with the provider API resource. The provider-side name must also match the enrolled host alias. A matching name alone is insufficient, so a duplicate-name resource is rejected. Linking does not arm shutdown.
  5. Select the linked server and choose Arm for tonight (a). Use Disarm (d) to cancel the armed operation before shutdown.

Give archive credentials read/write access only to the selected bucket/prefix. DigitalOcean Spaces setup selects the region through its HTTPS endpoint and uses the official us-east-1 S3 signing region. Temporary bootstrap keys are revoked after private bucket creation; crash recovery reconciles uncertain key creation by its exact managed name and retries failed revocation. Retain the encryption key and controller vault backup: an object-store copy alone cannot decrypt encrypted archives if the original key has been lost.

Readiness and shutdown

The controller makes the decision; the client only displays it. One completed session never authorizes power-off while another is working. A parent waiting for input does not make its still-working subagents safe to interrupt.

Observation Overnight behavior
One session is finished, another is working Keep the server on
All sessions have finished Eligible for archival after readiness checks
All sessions are waiting for user input or permission Eligible for archival; preserve their conversations for later replies
A mix of completed and waiting sessions Eligible only when every session is settled
An active delegated task remains Keep the server on
A session or its lifecycle observation is unknown/stale Keep the server on; show a blocker
A session is attached, an autonomous Run is active, or another container workload is running Keep the server on
Archive upload, checksum verification or index publication fails Do not power off; report the failed stage
Cloud power-off fails Retain the verified archive and report failure

The controller reconciles every ten seconds and requires a continuous minute of readiness before archival. The panel shows blockers, archive counts and operation stages: armed, waiting, archiving, stopping, stopped, restoring, restored or failed. Overnight is not a recurring shutdown schedule. A completed shutdown/restore disarms the host; you opt in again for the next night.

Archives and recovery

S3 stores archives in a separate namespace for each owner:

<prefix>/owners/<owner-id>/index.json
<prefix>/owners/<owner-id>/<archive-run-id>/workspace.tar.aes256

Each versioned index.json catalog identifies the host and cloud resource, archive IDs, timestamps, workspace IDs, native session IDs, root kinds, object keys, byte counts and SHA-256 checksums. Catalog updates use conditional writes so concurrent hosts do not silently overwrite each other's entries. Object checksums are verified by reading back the uploaded data before the index marks an archive available for restore. A failed or incomplete archive must not authorize power-off.

Choose Discover S3 archives (i) to read the current bucket's owner-scoped indexes and import verified recovery metadata if its local controller receipt is missing. Discovery is limited to this controller's known host/workspace owners; an S3 index cannot create an arbitrary owner, server or destination path. Restore reads a fresh catalog record instead of trusting the controller receipt alone.

Managed credential files and known secret paths, including .env, .agent.env, agent authorization files and private secret directories, are excluded. Generated dependencies/caches, sockets and symlinks are also excluded; install dependencies again when the restored environment needs them. Existing runtime credential files on the original disk are left untouched. Reauthorize expired accounts or redeliver missing runtime secrets through the existing devctrl credential mechanisms. Archive restore does not make an in-memory agent process or a browser connection survive cloud power-off: resume uses saved native conversation context on the restored workspace. Arbitrary repository files and conversation messages are not scanned for secrets; archive encryption protects this potentially sensitive project context.

Restore stages the archive and authenticates its encrypted data before promoting files to their original workspace and native-history roots. Fingerprint checks refuse to overwrite originals that changed unexpectedly after the saved snapshot.

If the controller and browser gateway run on the same server that stopped, their API and public link are unavailable while it is off. Start that server using the cloud provider or an independently running controller first. Reopen devctrl, select the host in Integrations → Overnight and choose Restore saved work (u). Select a saved coding session after the workspace is restored; an eligible authorized account is required to resume it. A temporary public tunnel URL can change when its gateway starts again; obtain the current console link instead of assuming an old tunnel remains reachable.

The current UI restores to the same enrolled host under the same controller, with persistent cloud disks and the controller vault retained. If the controller stopped before recording provider confirmation, startup reconciliation makes the interrupted shutdown restore-ready without requesting power-off a second time. The portable archive engine accepts an encryption key and explicit owner/path mappings, but a new-controller/new-host recovery wizard is not exposed by this UI.

Keep the S3 catalog and encryption keys for as long as recovery is needed. This initial workflow restores saved work explicitly; it does not promise transparent live-process migration, automatic DNS recovery, external service backups or restoration of unmanaged databases and arbitrary host workloads.

Controller API

All configuration, lifecycle and archive endpoints require infrastructure operator authorization. Public responses include configured status and archive metadata, never storage or provider secret values.

Method and route Purpose
GET /v1/overnight Redacted configuration, providers, eligible hosts, bindings and verified archives
PUT /v1/overnight/archive Save manual S3 configuration and encrypted credentials
GET /v1/overnight/storage-options Discover automatic storage setup supported by configured hosting accounts
POST /v1/overnight/storage/automatic Configure existing storage or explicitly create it using the chosen adapter
POST /v1/overnight/archives/discover Import verified recovery metadata from known owners' S3 indexes
PUT /v1/overnight/providers/:id/credentials Save a cloud API token privately
POST /v1/overnight/hosts Verify and link a temporary cloud resource to a host
POST /v1/overnight/hosts/:hostId/arm Opt in to one shutdown cycle
POST /v1/overnight/hosts/:hostId/disarm Cancel that cycle
POST /v1/overnight/hosts/:hostId/restore Request verified archive restoration

No production server is powered off by the normal test suite. Readiness tests use multiple-session and subagent fixtures; storage and cloud tests mock provider operations. The landing demo changes only local sample metadata and cannot call cloud APIs, write S3 objects or submit real credentials.