Session search

Session search finds text in running, frozen and stopped agent sessions and their Task Graph tasks by words and by meaning. It combines a SQLite FTS5 full-text index with a sqlite-vec vector index and merges both rankings with reciprocal-rank fusion (RRF). It is off by default. Each owner turns it on in Settings → Session search by choosing an embedding source: a local CPU model that runs in its own container, or a provider model paid with one of the owner's own API keys.

Open Search in the console menu (terminal and web share one App), type a query in any language and press Enter.

Content and where it lives

Content Location and access today Indexed
Claude Code transcripts (JSONL) Host: <workspace sessions>/provider-state/claude/<account>/projects/**/<conversation>.jsonl. Read by the Host agent only, as an owned regular file (O_NOFOLLOW, exact uid, no symlinks), through the same parser that renders GET /v1/sessions/:id/output: visible user and assistant text only — no thoughts, tool calls/results, sidechains or bootstrap instructions. Yes
Codex rollouts (JSONL) Host: <account store>/codex/<account>/sessions/**/*-<conversation>.jsonl, identity checked against the first session_meta record. Yes
Earlier conversations of a session Controller session record nativeConversationHistory (conversation IDs only); files on the Host as above. Yes, without a history cursor
Frozen and stopped sessions Freezing and stopping keep native transcripts (freeze). When the container is stopped or removed, the Host finds the retained per-user session storage from the workspace's storage layout. Chunks already indexed stay searchable while the conversation remains planned, even if the file is later archived. Yes
Archived sessions Session archive keeps an encrypted S3 copy; the Host index keeps the session's chunks and vectors, also after its local history was freed. Results carry an archived marker; search never reads S3. Yes (indexed before archiving)
devctrl conversations (Project messages, delegations, handoffs) Controller database: session_messages, coordination:task:*, session handoffs. Owner-scoped reads already exist (agent conversations). Not yet
Task Graph tasks Controller database, agent-reported task titles, summaries and details (Task Graph). The controller sends each task with its session's owner to that session's Host, hash first and text only when the Host lacks it. Yes (opens the Task Graph at the task)
Control Room chats Native transcripts in each chat's private Control Room namespace on the Host (chats). No (not needed)
Retained archives Compressed workspace archives (retained archives). No

Privacy follows the existing rule that the controller is not a second transcript store: the index lives on each Host next to the transcripts it was built from, in $DEVCTL_DATA_DIR/session-search/index.sqlite (mode 0600). The controller keeps only the owner's settings. Text crosses the controller only in memory: query results (bounded snippets) and, for a provider source, the chunks it forwards to that provider.

Redaction

Index text never holds credentials. The Host runs the shared execution-output redaction (redactExecutionText: bearer and basic authorization values, credentials in URLs, password/secret/api_key/token assignments, common provider token formats and private-key blocks) over each whole message before it is chunked, so a secret that a chunk boundary would split is still caught. The controller redacts Task Graph text before it hashes it and sends it to the Host. Because the stored text is redacted, everything built from it is too: the FTS index, result snippets, the local embedding model's input and the chunks a provider receives.

Defence in depth for egress and mixed versions: the controller redacts every chunk again right before a provider request, redacts the owner's query text before a provider embeds it, and redacts each snippet a Host returns (dropping its highlight ranges when that changed the text, which only happens with an older Host). A Host also redacts pending chunks before handing them out for embedding.

Indexes built by earlier versions are rewritten in place: every indexing tick (also while the Host is busy) checks at most 5,000 stored chunks after a persisted cursor, replaces the text and FTS row of any chunk the redaction changes and drops that chunk's vectors so it is embedded again from the redacted text. Unchanged chunks keep their vectors, so re-embedding is limited to the few chunks that held a secret.

The local path redacts too, deliberately: the cost is one linear regex pass over text that is at most 12,000 characters per message (2,000 per task), small next to JSON parsing, the SQLite and FTS5 inserts and, above all, model inference for each chunk. It keeps credentials out of index.sqlite and its backups, and out of snippets shown in the console.

Opening a result still reads the native transcript through the session output view, which applies its own sanitizing; the index does not change that view.

Ownership

Model choice (local CPU)

Requirements: multilingual (Russian and English matter most; transcripts mix prose and code), CPU only, container RAM within about 2 GB, permissive license, pinned files.

Model Size (disk) Dims (Matryoshka) Max tokens License MMTEB mean (retrieval) Russian evidence
Qwen3-Embedding-0.6B (chosen) 595M params; Q8_0 GGUF 639 MB 1024 (32–1024) 32K Apache-2.0 64.34 (64.65); MTEB Code 74.57 not published separately
nomic-embed-text-v2-moe 475M total / 305M active; Q8_0 512 MB 768 (to 256) 512 Apache-2.0 not reported (BEIR 52.86, MIRACL 65.8) none found
BAAI/bge-m3 568M; ~2.2 GB f32 1024 (no) 8192 MIT 59.56 (54.6) best published: ruMTEB 60.8, retrieval 74.8; Encodechka 0.787
google/embeddinggemma-300m 308M 768 (512/256/128) 2048 Gemma (gated) 61.15 (62.5) none found
Alibaba-NLP/gte-multilingual-base 305M 768 (elastic) 8192 Apache-2.0 58.2 (56.5) none found; needs remote code
intfloat/multilingual-e5-base / small 278M / 118M 768 / 384 512 MIT 57.0 / 55.8 Encodechka 0.761 / 0.742
jinaai/jina-embeddings-v3 572M 1024 (32–1024) 8192 CC-BY-NC-4.0 58.37 (55.76) excluded by license

Sources: model cards on Hugging Face; the MMTEB table in the EmbeddingGemma paper (arXiv 2509.20354); ruMTEB (arXiv 2408.12503); Encodechka (github.com/avidale/encodechka).

Decision: Qwen3-Embedding-0.6B, Q8_0 GGUF, 1024 dimensions. It has the best multilingual retrieval and code scores of the models under 1B parameters, a long context, Apache-2.0, and a 639 MB file that fits a 1.5 GB container. Queries use its task instruction (Instruct: … Query:); documents are embedded as plain text with last-token pooling.

Why not the suggested nomic-embed-text-v2-moe: its hard 512-token context cuts long agent messages, it publishes no MMTEB result, and as a mixture of experts it keeps all 475M parameters in RAM while using 305M per token. It stays a reasonable fallback and is easy to add to the catalogue (sessionSearchLocalModels). bge-m3 has the strongest published Russian evidence and is the first alternative if a Russian check on real transcripts favours it; Qwen3 has no published ruMTEB score. Measure both on real Russian transcripts before changing the default.

Multilingual evaluation

apps/host-agent/src/fixtures/session-search-eval.json holds 58 passages and 60 queries across English, Russian, Ukrainian, German, Spanish, French, Polish, Czech, Portuguese, Italian, Chinese and Japanese: cross-language queries, transliterated Russian, messages that mix Russian or Ukrainian prose with English code terms and error messages, and code or stack traces with comments in other languages. Sixteen unrelated distractors and twenty hard negatives (same topic, different facts) are never relevant. apps/host-agent/scripts/session-search-eval.mjs <url> <model> prints recall@1, recall@5 and MRR per query→passage language pair and per category.

Measured on the server, each model in the production container shape (2 CPUs, 1.5 GB, read-only, no network), the same llama.cpp image:

Model (Q8_0 GGUF) R@1 R@5 MRR Cross-lingual MRR Transliteration Mixed Code 58 passages Query mean / max Container memory
Qwen3-Embedding-0.6B 0.98 1.00 0.992 1.000 1.000 0.938 1.000 10.7 s 272 / 627 ms 1.20 GiB
bge-m3 0.97 1.00 0.983 0.986 1.000 1.000 0.958 8.1 s 108 / 198 ms 0.16 GiB¹

¹ As reported by docker stats; bge-m3's weights are memory-mapped and largely counted as shared page cache.

The only misses were rank 2, never outside the top 5: Qwen3 ranked a Ukrainian passage second for one Russian mixed-language query (ru→uk); bge-m3 missed one de→fr code query and one ru→es query. Every other language pair had recall@1 of 1.0 for both. bge-m3 does not win clearly on this mix, so Qwen3-Embedding-0.6B stays the default. bge-m3 is pinned (gpustack/bge-m3-GGUF@2d48f17…, bge-m3-Q8_0.gguf, SHA-256 950f4a8e…821167, MIT) and can be chosen in Settings when lower latency and memory matter more. The fixture saturates both models, so it guards against regressions rather than ranking close models; a larger set from real transcripts would separate them better.

session-search-eval.test.ts checks the fixture's coverage and the metrics with a fake embedder in CI, and runs a 16-query slice against a real model when DEVCTL_SESSION_SEARCH_EVAL_URL (and optionally DEVCTL_SESSION_SEARCH_EVAL_MODEL) points at a running embedding service.

Runtime and container

The local model is served by the llama.cpp server image pinned by its multi-arch digest ghcr.io/ggml-org/llama.cpp:server-b12282@sha256:086942d6ac097c894d9d5bbaf8085c53af85e113fbb9d6d61b3748863c86aa5f (amd64 and arm64 in one index). Hugging Face text-embeddings-inference supports the same models but loads them as f32 on CPU (about 2.4 GB of weights for a 0.6B model) and publishes one image per architecture; Ollama is convenient but harder to pin.

The model file is pinned by repository revision and SHA-256 (Qwen/Qwen3-Embedding-0.6B-GGUF@370f27d…, Qwen3-Embedding-0.6B-Q8_0.gguf, 06507c7b…c3e439, 639,150,592 bytes). The Host agent downloads it once with a bounded streaming hash into $DEVCTL_DATA_DIR/session-search/models/ and mounts it read-only. A file that does not match is deleted and the install fails.

The container devctl-session-search-embeddings:

Provider embeddings

There is no generic "embedding account" in devctrl. Provider keys are owned controller-only env secrets in Secrets, the same eligibility rule the OpenRouter integration uses (not granted to any Project). Settings list the owner's eligible secrets; the key value is never returned. A secret may hold the bare key or one NAME=value line.

Provider Models (fixed dimensions) Endpoint
OpenAI text-embedding-3-small (1536), text-embedding-3-large (1024 via dimensions) POST https://api.openai.com/v1/embeddings
Gemini gemini-embedding-001 (768 via outputDimensionality, normalized) POST …/v1beta/models/{model}:batchEmbedContents
Voyage AI voyage-3.5, voyage-multilingual-2 (1024) POST https://api.voyageai.com/v1/embeddings
OpenRouter openai/text-embedding-3-small (1536) POST https://openrouter.ai/api/v1/embeddings

Anthropic has no embedding model and recommends Voyage AI. The Settings screen warns that session text is sent to the chosen provider. That text is redacted first (Redaction).

Index

One SQLite file per Host (node:sqlite, WAL):

sqlite-vec 0.1.9 is a Host agent dependency; its npm package ships prebuilt loadable extensions for linux-x64, linux-arm64 and macOS. The Host opens the database with allowExtension: true and loadExtension(getLoadablePath()). If the extension cannot be loaded, search continues with FTS only and the reason is shown in Settings.

When indexing runs

Indexing is background work at low priority; a server administrator can skip the gate on request (Index now). Every 20 s the Host samples:

Indexing pauses while a launch is in progress or RAM is short (and resumes after one more good sample), and while the Host is busy: load per core above 1.0, CPU pressure above 25 %, or no available core for two samples. A busy pause resumes only after three calm samples (load per core below 0.7 and pressure below 10 %); Settings shows "paused: host busy". Between the two bands indexing runs reduced: one batch at a time and half the container's CPU limit. Only a calm Host gets the full multi-core concurrency: as many embedding batches in parallel as there are available cores, at most the container's indexing slots. The container always has at least two slots and the indexer uses all but one, so a search query never waits behind document batches.

The container's CPU limit follows the band at once with docker update --cpus (all its CPUs while calm, half while reduced, one while paused or during a launch), always within the Host's indexing CPU budget: a fifth of the Host's cores (at least one) unless DEVCTL_SESSION_SEARCH_CPU_BUDGET on the Host agent sets whole cores. A 20-vCPU Host therefore gives indexing at most four CPUs while calm and two while reduced; before the budget, the embedding container used five cores at load 0.8 per core while Core waited for CPU. No restart or model reload is needed, and a long pass also stops between rounds when the Host turns busy. Automatic threads follow the available cores (at most half the CPUs while reduced, and within the budget) with hysteresis because a new thread count needs a container restart: the wanted count must hold for 15 passes (about five minutes) and the container is recreated at most every 30 minutes. Threads set in Settings are never changed.

Measured throughput

apps/host-agent/scripts/session-search-bench.mjs <url> <concurrency> [chunks] sends ~1,000-character fixture chunks in batches of 16 and reports tokens/s from llama.cpp's usage field. On the 20-core server (2026-11-11, one-minute load 18–22 from other workloads, 0–9 cores at least 80 % idle; 64 chunks, 19,483 tokens, Qwen3-Embedding-0.6B Q8_0, image b12282):

Container Priority Concurrent batches Tokens/s Batch p50 / p95
2 CPUs, 2 threads, 1 slot (previous) shares 128, nice 10 1 83 44 / 83 s
8 CPUs, 8 threads, 1 slot shares 128, nice 10 1 132 32 / 49 s
8 CPUs, 8 threads, 2 slots (default) shares 128, nice 10 1 141 33 / 38 s
8 CPUs, 8 threads, 4 slots shares 128, nice 10 1 109 43 / 59 s
8 CPUs, 8 threads, 4 slots shares 128, nice 10 4 111 97 / 175 s
16 CPUs, 16 threads, 8 slots shares 128, nice 10 8 136 67 / 143 s
8 CPUs, 2 threads, 4 slots shares 128, nice 10 4 — (a request exceeded 300 s) —
8 CPUs, 8 threads, 1 slot shares 1024, nice 0 1 152 31 / 34 s
8 CPUs, 8 threads, 4 slots shares 1024, nice 0 4 167 55 / 117 s

Threads raise throughput (83 → 132 tokens/s from 2 to 8); parallel slots barely do (±10 %, within the noise of the shared load), because embedding is prompt processing that already batches every token of a request, while extra slots multiply memory and latency. Low priority costs about 13 % against normal priority on this load. The defaults therefore use more threads and only two slots; slots and concurrency remain configurable for Hosts where they help. Document batches get a 10-minute timeout (queries 2 minutes) because a low-priority batch on a busy Host can take minutes.

A pass is bounded and its bounds grow with that concurrency (factor 1–8): at most 8 conversations and 6 MB in total per factor, 2 MB per conversation, complete JSONL records only (records over 1 MB are skipped), and four rounds of up to one 16-chunk batch per slot. Offsets are stored per pass, so work resumes after restarts. Provider batches are handed out only while indexing may run; they are sent one at a time.

Index now

On a busy Host the idle gate can hold indexing back for hours. A server administrator can start it at once: Index now on the Search screen (Shift+I in the terminal, the button in the terminal and the web console), devctl search index-now [--host <id>], the operator MCP tool session_search_index_now (hostId optional) or POST /v1/session-search/index-now ({} or { "hostId": "…" }).

Indexing statistics

Settings → Session search shows per Host, under the Host's status line:

The Search screen shows one compact line summed over the owner's Hosts: pending conversations, vectors of chunks, the waiting reason or the idle-core concurrency, and tokens today (with the estimated cost for a provider). Settings also shows the provider embeddings of the owner's search queries.

Token counts come from the response's usage field: usage.prompt_tokens (OpenAI, OpenRouter, llama.cpp), usage.total_tokens (Voyage) and Gemini's usageMetadata when present (its batch endpoint usually returns none). A response without usage is estimated as one token per four UTF-8 bytes, which overestimates Cyrillic and CJK text; estimated tokens are counted separately and shown with "estimated" or "~".

Before the owner switches to or enables a provider model, Settings shows the projected backlog: the chunks that would be embedded (pending chunks for the current model, every chunk for another model), their estimated tokens and cost at the pinned price, and how many planned conversations are not read yet and therefore not included. Saving such a change needs a second Enter that confirms it.

A failed provider batch with a retryable code (HTTP 429 or 5xx, timeout, network) is retried once after its Retry-After (at most 10 s, otherwise 1 s); the error and the retry are counted. Local failures are retried by the next pass.

Persistence and bounds: the Host keeps each owner's local ledger and re-embed count in its index (usage_stats, at most 64 owners, least recently updated dropped) and the wait time and last pass in host_stats. The controller keeps each owner's provider ledgers in session-search:usage:<owner> (one per Host, at most 32 Hosts, plus the query ledger). A ledger holds at most 96 day × model buckets, 128 latencies and 16 error or retry codes (further codes count as other); a stored value that does not match these bounds reads as empty.

Statistics travel in the existing GET /v1/session-search/settings answer, owner-scoped like the rest: the controller asks each Host with the owner's ID (status with stats: true, only for Hosts with the session-search-v2 capability; plans carry localResources only to those Hosts) and joins its own provider ledger for that owner and Host. A server user sees only their own corpus and usage.

Search statistics

Search → Statistics (Ctrl+Y or the tab above the query box; terminal and web) shows the owner's statistics for the last 24 hours, 7, 30 or 365 days (p changes the period, r refreshes, ↑/↓ and PgUp/PgDn scroll). The Search tab carries a one-line summary from recorded samples only: index size, the share of chunks with vectors and the searches of the last 24 hours.

Where it is recorded:

CLI and operator MCP return the same view: devctl search stats [24h|7d|30d|365d] [--sampled] and the operator tool session_search_statistics (period, sampled), which runs that CLI command. Both build the request with sessionSearchStatisticsRequest from @devctl/contracts/session-search.

Search and results

POST /v1/session-search/query takes query and optional filters: groupId (project), projectId (repository), agent, sessionId, from/to and states (running, frozen, stopped) and archived (any, only, exclude; see session archive). Filters narrow the owner's session allowlist before any Host is asked. The controller embeds the query with the provider (a local Host embeds it itself), asks each Host with matching sessions in parallel, fuses each Host's text and vector ranks with RRF (k = 60) and returns, per result: project, repository, agent, session, state, kind (transcript or task), role, time, the matched text with up to ~600 characters of context and highlight ranges, the matched channels, and two native cursors — one ending at the match, one ending two messages later — and archived ({ archivedAt, location: "s3", pruned }, or null).

Each result also carries reason, textRank and vectorRank. The controller computes the reason and the final highlight ranges from the query and the snippet (explainSessionSearchMatch in @devctl/contracts/session-search):

The console derives the same explanation itself when an older controller sends no reason.

The query route accepts an optional progressId query parameter (8–64 letters, digits, - or _). While the query runs, GET /v1/session-search/progress/:progressId returns its stage — embedding (a provider source only), searching with Hosts answered / unavailable of Hosts asked, ranking, done — and the planned stages. Progress is kept per owner in memory for one minute (at most 256 queries); another owner's ID reads as 404.

The Search screen (terminal and web) fills the content pane: the native list computes how many rows fit from the terminal rows left under the header lines, and the web list grows with the pane and scrolls.

The handoff seed is the owner's own matched text: POST /v1/sessions/:id/handoffs/prepare accepts an optional context (1–600 characters) that becomes the focus in the handoff's goal. The agent prompt is unchanged.

Routes (server-route-catalogue.ts; handler-scoped user routes, except index now, which is a handler-scoped administrator route):

Route Purpose
GET /v1/session-search/settings Settings, eligible secrets, per-Host index status and statistics
PUT /v1/session-search/settings { expectedRevision, embedding }; revision-checked
POST /v1/session-search/query Owner-scoped hybrid search; optional ?progressId=
GET /v1/session-search/progress/:progressId Stage of one of the owner's in-flight queries
GET /v1/session-search/statistics period (24h, 7d, 30d, 365d), live; the owner's statistics view
POST /v1/session-search/log/:logId/open { rank, kind, action }; counts an opened result of the owner's own logged search
POST /v1/session-search/index-now Administrator: { hostId? }; queues index now per Host (queued, unsupported, offline)

Limitations

Remaining steps

  1. Index devctrl's own messages, delegations and handoffs with the same owner rules.
  2. Grow the evaluation set from real (owner-approved) transcripts so close models can be ranked.