Session search
Session search finds text in running, frozen and stopped agent sessions and their Task Graph tasks by words and by meaning. It combines a SQLite FTS5 full-text index with a sqlite-vec vector index and merges both rankings with reciprocal-rank fusion (RRF). It is off by default. Each owner turns it on in Settings → Session search by choosing an embedding source: a local CPU model that runs in its own container, or a provider model paid with one of the owner's own API keys.
Open Search in the console menu (terminal and web share one App), type a query in any language and press Enter.
Content and where it lives
| Content | Location and access today | Indexed |
|---|---|---|
| Claude Code transcripts (JSONL) | Host: <workspace sessions>/provider-state/claude/<account>/projects/**/<conversation>.jsonl. Read by the Host agent only, as an owned regular file (O_NOFOLLOW, exact uid, no symlinks), through the same parser that renders GET /v1/sessions/:id/output: visible user and assistant text only — no thoughts, tool calls/results, sidechains or bootstrap instructions. |
Yes |
| Codex rollouts (JSONL) | Host: <account store>/codex/<account>/sessions/**/*-<conversation>.jsonl, identity checked against the first session_meta record. |
Yes |
| Earlier conversations of a session | Controller session record nativeConversationHistory (conversation IDs only); files on the Host as above. |
Yes, without a history cursor |
| Frozen and stopped sessions | Freezing and stopping keep native transcripts (freeze). When the container is stopped or removed, the Host finds the retained per-user session storage from the workspace's storage layout. Chunks already indexed stay searchable while the conversation remains planned, even if the file is later archived. | Yes |
| Archived sessions | Session archive keeps an encrypted S3 copy; the Host index keeps the session's chunks and vectors, also after its local history was freed. Results carry an archived marker; search never reads S3. |
Yes (indexed before archiving) |
| devctrl conversations (Project messages, delegations, handoffs) | Controller database: session_messages, coordination:task:*, session handoffs. Owner-scoped reads already exist (agent conversations). |
Not yet |
| Task Graph tasks | Controller database, agent-reported task titles, summaries and details (Task Graph). The controller sends each task with its session's owner to that session's Host, hash first and text only when the Host lacks it. | Yes (opens the Task Graph at the task) |
| Control Room chats | Native transcripts in each chat's private Control Room namespace on the Host (chats). | No (not needed) |
| Retained archives | Compressed workspace archives (retained archives). | No |
Privacy follows the existing rule that the controller is not a second
transcript store: the index lives on each Host next to the transcripts it was
built from, in $DEVCTL_DATA_DIR/session-search/index.sqlite (mode 0600).
The controller keeps only the owner's settings. Text crosses the controller only
in memory: query results (bounded snippets) and, for a provider source, the
chunks it forwards to that provider.
Redaction
Index text never holds credentials. The Host runs the shared execution-output
redaction (redactExecutionText: bearer and basic authorization values,
credentials in URLs, password/secret/api_key/token assignments, common
provider token formats and private-key blocks) over each whole message
before it is chunked, so a secret that a chunk boundary would split is still
caught. The controller redacts Task Graph text before it hashes it and sends it
to the Host. Because the stored text is redacted, everything built from it is
too: the FTS index, result snippets, the local embedding model's input and the
chunks a provider receives.
Defence in depth for egress and mixed versions: the controller redacts every chunk again right before a provider request, redacts the owner's query text before a provider embeds it, and redacts each snippet a Host returns (dropping its highlight ranges when that changed the text, which only happens with an older Host). A Host also redacts pending chunks before handing them out for embedding.
Indexes built by earlier versions are rewritten in place: every indexing tick (also while the Host is busy) checks at most 5,000 stored chunks after a persisted cursor, replaces the text and FTS row of any chunk the redaction changes and drops that chunk's vectors so it is embedded again from the redacted text. Unchanged chunks keep their vectors, so re-embedding is limited to the few chunks that held a secret.
The local path redacts too, deliberately: the cost is one linear regex pass
over text that is at most 12,000 characters per message (2,000 per task),
small next to JSON parsing, the SQLite and FTS5 inserts and, above all, model
inference for each chunk. It keeps credentials out of index.sqlite and its
backups, and out of snippets shown in the console.
Opening a result still reads the native transcript through the session output view, which applies its own sanitizing; the index does not change that view.
Ownership
- The controller is the ownership authority. Every planning cycle (60 s) it sends
each connected Host the exact list of conversations to index, each labelled
with its current owner (
ownerUserIdForSession). Only owners whose search is enabled are planned; Control Room sessions are excluded. - The Host stores the owner on every chunk and filters every read by owner. An ownership change relabels chunks; a complete plan purges conversations that are no longer listed (deleted sessions, an owner who turned search off). A plan truncated at 512 conversations per Host never purges.
- A query passes the owner and the allowlist of that owner's sessions after the project/session/state filters. The controller rechecks each returned session's owner before it builds a result. A server user therefore only ever sees their own sessions, even if a Host returned something else.
- Provider API keys never reach a Host. Provider embeddings are pumped by the controller: it asks the Host for pending chunks of that owner, re-reads the owner's settings, calls only the chosen provider with the owner's own secret, and returns vectors that the Host stores only in that owner's current space.
Model choice (local CPU)
Requirements: multilingual (Russian and English matter most; transcripts mix prose and code), CPU only, container RAM within about 2 GB, permissive license, pinned files.
| Model | Size (disk) | Dims (Matryoshka) | Max tokens | License | MMTEB mean (retrieval) | Russian evidence |
|---|---|---|---|---|---|---|
| Qwen3-Embedding-0.6B (chosen) | 595M params; Q8_0 GGUF 639 MB | 1024 (32–1024) | 32K | Apache-2.0 | 64.34 (64.65); MTEB Code 74.57 | not published separately |
| nomic-embed-text-v2-moe | 475M total / 305M active; Q8_0 512 MB | 768 (to 256) | 512 | Apache-2.0 | not reported (BEIR 52.86, MIRACL 65.8) | none found |
| BAAI/bge-m3 | 568M; ~2.2 GB f32 | 1024 (no) | 8192 | MIT | 59.56 (54.6) | best published: ruMTEB 60.8, retrieval 74.8; Encodechka 0.787 |
| google/embeddinggemma-300m | 308M | 768 (512/256/128) | 2048 | Gemma (gated) | 61.15 (62.5) | none found |
| Alibaba-NLP/gte-multilingual-base | 305M | 768 (elastic) | 8192 | Apache-2.0 | 58.2 (56.5) | none found; needs remote code |
| intfloat/multilingual-e5-base / small | 278M / 118M | 768 / 384 | 512 | MIT | 57.0 / 55.8 | Encodechka 0.761 / 0.742 |
| jinaai/jina-embeddings-v3 | 572M | 1024 (32–1024) | 8192 | CC-BY-NC-4.0 | 58.37 (55.76) | excluded by license |
Sources: model cards on Hugging Face; the MMTEB table in the EmbeddingGemma paper (arXiv 2509.20354); ruMTEB (arXiv 2408.12503); Encodechka (github.com/avidale/encodechka).
Decision: Qwen3-Embedding-0.6B, Q8_0 GGUF, 1024 dimensions. It has the best
multilingual retrieval and code scores of the models under 1B parameters, a long
context, Apache-2.0, and a 639 MB file that fits a 1.5 GB container. Queries use
its task instruction (Instruct: … Query:); documents are embedded as plain
text with last-token pooling.
Why not the suggested nomic-embed-text-v2-moe: its hard 512-token context cuts
long agent messages, it publishes no MMTEB result, and as a mixture of experts it
keeps all 475M parameters in RAM while using 305M per token. It stays a reasonable
fallback and is easy to add to the catalogue (sessionSearchLocalModels).
bge-m3 has the strongest published Russian evidence and is the first
alternative if a Russian check on real transcripts favours it; Qwen3 has no
published ruMTEB score. Measure both on real Russian transcripts before
changing the default.
Multilingual evaluation
apps/host-agent/src/fixtures/session-search-eval.json holds 58 passages and
60 queries across English, Russian, Ukrainian, German, Spanish, French, Polish,
Czech, Portuguese, Italian, Chinese and Japanese: cross-language queries,
transliterated Russian, messages that mix Russian or Ukrainian prose with
English code terms and error messages, and code or stack traces with comments
in other languages. Sixteen unrelated distractors and twenty hard negatives
(same topic, different facts) are never relevant.
apps/host-agent/scripts/session-search-eval.mjs <url> <model> prints recall@1,
recall@5 and MRR per query→passage language pair and per category.
Measured on the server, each model in the production container shape (2 CPUs, 1.5 GB, read-only, no network), the same llama.cpp image:
| Model (Q8_0 GGUF) | R@1 | R@5 | MRR | Cross-lingual MRR | Transliteration | Mixed | Code | 58 passages | Query mean / max | Container memory |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-Embedding-0.6B | 0.98 | 1.00 | 0.992 | 1.000 | 1.000 | 0.938 | 1.000 | 10.7 s | 272 / 627 ms | 1.20 GiB |
| bge-m3 | 0.97 | 1.00 | 0.983 | 0.986 | 1.000 | 1.000 | 0.958 | 8.1 s | 108 / 198 ms | 0.16 GiB¹ |
¹ As reported by docker stats; bge-m3's weights are memory-mapped and largely
counted as shared page cache.
The only misses were rank 2, never outside the top 5: Qwen3 ranked a
Ukrainian passage second for one Russian mixed-language query (ru→uk); bge-m3
missed one de→fr code query and one ru→es query. Every other language pair had
recall@1 of 1.0 for both. bge-m3 does not win clearly on this mix, so
Qwen3-Embedding-0.6B stays the default. bge-m3 is pinned
(gpustack/bge-m3-GGUF@2d48f17…, bge-m3-Q8_0.gguf, SHA-256 950f4a8e…821167,
MIT) and can be chosen in Settings when lower latency and memory matter more.
The fixture saturates both models, so it guards against regressions rather
than ranking close models; a larger set from real transcripts would separate
them better.
session-search-eval.test.ts checks the fixture's coverage and the metrics with
a fake embedder in CI, and runs a 16-query slice against a real model when
DEVCTL_SESSION_SEARCH_EVAL_URL (and optionally DEVCTL_SESSION_SEARCH_EVAL_MODEL)
points at a running embedding service.
Runtime and container
The local model is served by the llama.cpp server image pinned by its
multi-arch digest
ghcr.io/ggml-org/llama.cpp:server-b12282@sha256:086942d6ac097c894d9d5bbaf8085c53af85e113fbb9d6d61b3748863c86aa5f
(amd64 and arm64 in one index). Hugging Face text-embeddings-inference supports
the same models but loads them as f32 on CPU (about 2.4 GB of weights for a
0.6B model) and publishes one image per architecture; Ollama is convenient but
harder to pin.
The model file is pinned by repository revision and SHA-256
(Qwen/Qwen3-Embedding-0.6B-GGUF@370f27d…, Qwen3-Embedding-0.6B-Q8_0.gguf,
06507c7b…c3e439, 639,150,592 bytes). The Host agent downloads it once with a
bounded streaming hash into $DEVCTL_DATA_DIR/session-search/models/ and mounts
it read-only. A file that does not match is deleted and the install fails.
The container devctl-session-search-embeddings:
- is labelled
io.devctl.managed=session-search-embeddingsplus model and Host labels, and its ID is recorded insession-search/local-service.json; removal checks the recorded ID and the label beforedocker rm; - runs read-only as
65532:65532with all capabilities dropped,no-new-privileges, 128 pids and a 64 MB/tmp; - runs at low CPU priority:
--cpu-shares 128(Docker's default is 1024) andnice -n 10in front of/app/llama-server, so it uses idle cores and yields to sessions under contention; withDEVCTL_WORKLOADS_CGROUP_PARENT=devctl-workloads.sliceon the Host agent it also runs in that low-weight slice (Host performance); - has configurable resources (Settings → Session search, local model): the CPU
limit (
--cpus), llama.cpp threads (--threads,--threads-batch) and parallel slots (--parallel,--cont-batching). Each value is optional; the Host chooses absent ones from its core count: half the cores, at most 8 CPUs; threads = CPUs; two slots (a 20-core server gets 8 CPUs, 8 threads and 2 slots). With more than one slot, one stays free so a search query never waits behind document batches; the indexer uses the others. Each slot keeps a 4,096-token context (--ctx-size= 4,096 × slots,--ubatch-size4,096; a chunk is at most about 1,000 characters). Memory is 1 GiB plus 512 MiB per slot, without swap. The resolved values are a container label; a change recreates the owned container. The container is shared by the Host's local owners; like the model, the first planned local owner's resources apply; - has a Docker health check on
/health; - joins an internal Docker network
devctl-session-searchwith no egress, which the Host agent's own container is connected to. A Host agent on the host network instead reaches it through127.0.0.1:18089; - starts only when an owner with sessions on that Host chooses the local source, and is removed (the model file is kept) when no owner on the Host uses it.
Provider embeddings
There is no generic "embedding account" in devctrl. Provider keys are owned
controller-only env secrets in Secrets, the same eligibility rule the
OpenRouter integration uses (not granted to any Project). Settings list the
owner's eligible secrets; the key value is never returned. A secret may hold the
bare key or one NAME=value line.
| Provider | Models (fixed dimensions) | Endpoint |
|---|---|---|
| OpenAI | text-embedding-3-small (1536), text-embedding-3-large (1024 via dimensions) |
POST https://api.openai.com/v1/embeddings |
| Gemini | gemini-embedding-001 (768 via outputDimensionality, normalized) |
POST …/v1beta/models/{model}:batchEmbedContents |
| Voyage AI | voyage-3.5, voyage-multilingual-2 (1024) |
POST https://api.voyageai.com/v1/embeddings |
| OpenRouter | openai/text-embedding-3-small (1536) |
POST https://openrouter.ai/api/v1/embeddings |
Anthropic has no embedding model and recommends Voyage AI. The Settings screen warns that session text is sent to the chosen provider. That text is redacted first (Redaction).
Index
One SQLite file per Host (node:sqlite, WAL):
chunks: session, conversation, owner, source, native message ID, role, time, the record's end offset and the chunk text. Messages are split into ~1,000-character chunks with 120 characters of overlap on paragraph, line, sentence or word boundaries. A repeated native message ID (streamed assistant updates) replaces its earlier chunks.chunks_fts: external-content FTS5 table,unicode61 remove_diacritics 2(case folding for Cyrillic and Latin). A query becomes quoted prefix terms joined with OR and ranked by BM25. Words are only tokenized; nothing is classified by pattern.vec_<hash(model)>: one sqlite-vecvec0table per embedding space with cosine distance and the owner as partition key.spacesrecords each owner's current model; choosing another model drops that owner's old vectors and the chunks become pending again (re-embedding).sources: per conversation, the file identity (device, inode, birth time) and the byte offset indexed so far. Indexing resumes from the offset; a replaced or truncated file is re-indexed from the start.
sqlite-vec 0.1.9 is a Host agent dependency; its npm package ships prebuilt
loadable extensions for linux-x64, linux-arm64 and macOS. The Host opens the
database with allowExtension: true and loadExtension(getLoadablePath()).
If the extension cannot be loaded, search continues with FTS only and the
reason is shown in Settings.
When indexing runs
Indexing is background work at low priority; a server administrator can skip the gate on request (Index now). Every 20 s the Host samples:
- whether a managed session launch is in progress on this Host;
- available RAM (
MemAvailable; at least 25 % must remain); - the one-minute load average per core and, where the kernel reports it, CPU
pressure (
/proc/pressure/cpu, PSIsome avg10); - idle capacity: the kernel's per-core CPU counters (
/proc/staton Linux, read throughos.cpus()), sampled over one second, summed as each core's idle share. Six cores 70 % idle are about 4.2 cores of capacity even though none is fully idle. One core is kept in reserve: available cores = ⌊Σ idle share − 1⌋.
Indexing pauses while a launch is in progress or RAM is short (and resumes after one more good sample), and while the Host is busy: load per core above 1.0, CPU pressure above 25 %, or no available core for two samples. A busy pause resumes only after three calm samples (load per core below 0.7 and pressure below 10 %); Settings shows "paused: host busy". Between the two bands indexing runs reduced: one batch at a time and half the container's CPU limit. Only a calm Host gets the full multi-core concurrency: as many embedding batches in parallel as there are available cores, at most the container's indexing slots. The container always has at least two slots and the indexer uses all but one, so a search query never waits behind document batches.
The container's CPU limit follows the band at once with docker update --cpus
(all its CPUs while calm, half while reduced, one while paused or during a
launch), always within the Host's indexing CPU budget: a fifth of the Host's
cores (at least one) unless DEVCTL_SESSION_SEARCH_CPU_BUDGET on the Host
agent sets whole cores. A 20-vCPU Host therefore gives indexing at most four
CPUs while calm and two while reduced; before the budget, the embedding
container used five cores at load 0.8 per core while Core waited for CPU. No
restart or model reload is needed, and a long pass also stops
between rounds when the Host turns busy. Automatic threads follow the
available cores (at most half the CPUs while reduced, and within the budget) with hysteresis because
a new thread count needs a container restart: the wanted count must hold for
15 passes (about five minutes) and the container is recreated at most every
30 minutes. Threads set in Settings are never changed.
Measured throughput
apps/host-agent/scripts/session-search-bench.mjs <url> <concurrency> [chunks]
sends ~1,000-character fixture chunks in batches of 16 and reports tokens/s
from llama.cpp's usage field. On the 20-core server (2026-11-11, one-minute
load 18–22 from other workloads, 0–9 cores at least 80 % idle; 64 chunks,
19,483 tokens, Qwen3-Embedding-0.6B Q8_0, image b12282):
| Container | Priority | Concurrent batches | Tokens/s | Batch p50 / p95 |
|---|---|---|---|---|
| 2 CPUs, 2 threads, 1 slot (previous) | shares 128, nice 10 | 1 | 83 | 44 / 83 s |
| 8 CPUs, 8 threads, 1 slot | shares 128, nice 10 | 1 | 132 | 32 / 49 s |
| 8 CPUs, 8 threads, 2 slots (default) | shares 128, nice 10 | 1 | 141 | 33 / 38 s |
| 8 CPUs, 8 threads, 4 slots | shares 128, nice 10 | 1 | 109 | 43 / 59 s |
| 8 CPUs, 8 threads, 4 slots | shares 128, nice 10 | 4 | 111 | 97 / 175 s |
| 16 CPUs, 16 threads, 8 slots | shares 128, nice 10 | 8 | 136 | 67 / 143 s |
| 8 CPUs, 2 threads, 4 slots | shares 128, nice 10 | 4 | — (a request exceeded 300 s) | — |
| 8 CPUs, 8 threads, 1 slot | shares 1024, nice 0 | 1 | 152 | 31 / 34 s |
| 8 CPUs, 8 threads, 4 slots | shares 1024, nice 0 | 4 | 167 | 55 / 117 s |
Threads raise throughput (83 → 132 tokens/s from 2 to 8); parallel slots barely do (±10 %, within the noise of the shared load), because embedding is prompt processing that already batches every token of a request, while extra slots multiply memory and latency. Low priority costs about 13 % against normal priority on this load. The defaults therefore use more threads and only two slots; slots and concurrency remain configurable for Hosts where they help. Document batches get a 10-minute timeout (queries 2 minutes) because a low-priority batch on a busy Host can take minutes.
A pass is bounded and its bounds grow with that concurrency (factor 1–8): at most 8 conversations and 6 MB in total per factor, 2 MB per conversation, complete JSONL records only (records over 1 MB are skipped), and four rounds of up to one 16-chunk batch per slot. Offsets are stored per pass, so work resumes after restarts. Provider batches are handed out only while indexing may run; they are sent one at a time.
Index now
On a busy Host the idle gate can hold indexing back for hours. A server
administrator can start it at once: Index now on the Search screen
(Shift+I in the terminal, the button in the terminal and the web console),
devctl search index-now [--host <id>], the operator MCP tool
session_search_index_now (hostId optional) or POST /v1/session-search/index-now ({} or { "hostId": "…" }).
- Who: administrator only (operator token or a server user with the
administrator role), like the other on-request Host passes (retained archive
rescan, Docker storage refresh). The run spends shared Host CPU outside the
gate that protects every user's sessions, so a server user is refused (403)
and the console offers the action disabled with that explanation
(
GET /v1/session-search/settingscarriescanIndexNow; an older Core omits it and the console does not offer the action). Core audits each request per Host (session-search.index-now.requested, with the outcome and command ID). - Delivery: Core queues the Host command
session-search.index-nowthrough the ordinary Host command queue to every Host with session search (or the one named). A Host without thesession-search-index-now-v1capability is reported asunsupported(an older Host) and a disconnected one asoffline; neither gets a command. A pending request of the same Host younger than five minutes is reused. The command answers at once:started, oralready_runningwhen a run or a working idle pass is in progress (no second pass is ever started; a pass that is only sampling the Host is awaited first). - Where it runs: in the Host agent process that holds the session search plan and
index. With split Host channels that is the terminal process (it answers
session.search.rpc), while Main executes Host commands: Main hands the command to the terminal process over its IPC channel and returns its answer. A process that has not received a plan yet (just restarted) waits up to two minutes for Core's next planning cycle instead of reporting nothing to do. - What runs: the same bounded passes, the same reading, redaction before chunking and embedding code, repeated back to back. Each pass skips the CPU part of the gate (load per core, CPU pressure, idle cores, the settling sample and the busy check between embedding rounds) but still waits while a session launch is in progress or RAM is short. Automatic threads are not retuned during a run.
- CPU bound: a run uses the calm band, never more: the container's CPU
limit is the Host's indexing CPU budget (a fifth of the cores, or
DEVCTL_SESSION_SEARCH_CPU_BUDGET, and at most the container's CPUs) and the concurrency is the indexing slots (all but the one kept for queries) within that budget. The container keeps its low CPU shares andnice, so sessions still come first. Provider batches are handed out during the run regardless of load. - End: when every planned conversation has been read to its end and no
chunk waits for a vector (
caught-up), after 30 minutes or 360 passes (limit; press Shift+I again to continue), after three failed passes in a row (error), or when the Host agent stops (stopped). The idle gate applies again afterwards. - Progress: each Host reports its current or last run in its statistics
(status RPC
indexNow: true, only to capable Hosts): state, start and end, passes, the owner's chunks read and vectors embedded during the run (never another owner's), the waiting cause and how it ended. The Search screen shows one line per Host: queued, started, already indexing, not supported (older Host), offline or failed, then the run's progress and its final chunk counts; it refreshes the statistics every few seconds while a run is active. Settings → Session search lists the run under each Host. - Host log: one JSON line on the Host agent's stderr for the start
(
session-search.index-now.started: whether a plan was received, targets, owners, CPU budget), each pass (session-search.index-now.pass:worked,empty,blockedwith its cause,failedwith its code, orno-plan; bytes read, chunks, vectors, concurrency, CPU limit, local model state), the end (session-search.index-now.finished: stop, reason, passes, duration and totals) and a refused second request (…already-running). Lines carry counts, codes and times only, never owners, sessions or text.
Indexing statistics
Settings → Session search shows per Host, under the Host's status line:
- Corpus: conversations planned, read to the end (indexed) and pending; chunks, vectors, Task Graph tasks; the index file size on disk (database plus WAL, shared by the Host's owners); time of the last completed pass.
- Embedding work for the owner's current source: inputs embedded, tokens,
batches, throughput (tokens/s and chunks/s from wall time, so parallel
batches count once) and p50/p95 latency of the last 128 batches; errors and
retries by code (
http_<status>,timeout,network,shape,other); chunks re-embedded because the owner chose another model. - Usage and cost: tokens today, over 7 and over 30 days and, for provider
models, the estimated cost from the pinned price table
(
packages/contracts/src/session-search-prices.ts, USD per 1M input tokens, with the date the prices were checked). Every cost is marked as an estimate. - Idle gate: total time indexing waited and the current reason (session launch, RAM, hysteresis).
- Local model: the Host's idle capacity and the concurrency used ("idle
capacity ≈ 4.4 of 20 cores → using 1 slot"), the configured or automatic slots, threads
and CPU limit, and the container's CPU and RAM from one
docker statssample, at most once a minute while it embeds, with the peak since the Host agent started. - Index now: the current or last index-now run with this owner's chunk and vector counts.
The Search screen shows one compact line summed over the owner's Hosts: pending conversations, vectors of chunks, the waiting reason or the idle-core concurrency, and tokens today (with the estimated cost for a provider). Settings also shows the provider embeddings of the owner's search queries.
Token counts come from the response's usage field: usage.prompt_tokens
(OpenAI, OpenRouter, llama.cpp), usage.total_tokens (Voyage) and Gemini's
usageMetadata when present (its batch endpoint usually returns none). A
response without usage is estimated as one token per four UTF-8 bytes, which
overestimates Cyrillic and CJK text; estimated tokens are counted separately
and shown with "estimated" or "~".
Before the owner switches to or enables a provider model, Settings shows the projected backlog: the chunks that would be embedded (pending chunks for the current model, every chunk for another model), their estimated tokens and cost at the pinned price, and how many planned conversations are not read yet and therefore not included. Saving such a change needs a second Enter that confirms it.
A failed provider batch with a retryable code (HTTP 429 or 5xx, timeout,
network) is retried once after its Retry-After (at most 10 s, otherwise 1 s);
the error and the retry are counted. Local failures are retried by the next
pass.
Persistence and bounds: the Host keeps each owner's local ledger and re-embed
count in its index (usage_stats, at most 64 owners, least recently updated
dropped) and the wait time and last pass in host_stats. The controller keeps
each owner's provider ledgers in session-search:usage:<owner> (one per Host,
at most 32 Hosts, plus the query ledger). A ledger holds at most 96 day × model
buckets, 128 latencies and 16 error or retry codes (further codes count as
other); a stored value that does not match these bounds reads as empty.
Statistics travel in the existing GET /v1/session-search/settings answer,
owner-scoped like the rest: the controller asks each Host with the owner's ID
(status with stats: true, only for Hosts with the session-search-v2
capability; plans carry localResources only to those Hosts) and joins its
own provider ledger for that owner and Host. A server user sees only their
own corpus and usage.
Search statistics
Search → Statistics (Ctrl+Y or the tab above the query box; terminal and
web) shows the owner's statistics for the last 24 hours, 7, 30 or 365 days
(p changes the period, r refreshes, ↑/↓ and PgUp/PgDn scroll). The Search
tab carries a one-line summary from recorded samples only: index size, the
share of chunks with vectors and the searches of the last 24 hours.
- Index: bytes on disk per Host and in total, split into the full-text
(FTS5) and vector (sqlite-vec) stores, chunk texts and the rest (from
SQLite's
dbstat, cached for a minute; the WAL counts as rest); the owner's chunks, vectors and sessions per Host; an index-size trend. - Coverage: indexed versus pending sessions (a session counts as indexed when every planned conversation was read to the end and every chunk has a vector), chunks with and without vectors, and transcript bytes read and unread, in total and by project, repository, agent and source (live, frozen, stopped, Task Graph). A conversation that was never opened has no known size; it keeps its session pending and is counted separately.
- Indexing activity: chunks indexed and embedded per bucket (minute, hour or day, depending on the period), tokens, embedding time, passes and paused time, plus a weekday × hour heatmap of embedded chunks over the last 7 days (UTC): the "when does indexing happen" chart.
- Pauses and limits: pauses by cause (session launch, RAM, host busy) with their count and duration, and a timeline of pauses, resumes and changes of the embedding container's CPU limit and threads. A pause's duration is counted when it ends.
- Embedding errors by code and chunks re-embedded after a model change.
- Search log: the owner's searches with average and p95 time, opened results, errors and the lexical / vector / both hit mix; recent, most frequent and slowest queries with their stage times (query embedding, Hosts, ranking).
Where it is recorded:
- Host indexer. Each pass writes counts and short codes, never text, to an
outbox in its index (
activity, at most 5,000 rows):pass,indexandembed(per owner),embed_error,reembed,pause,resumeandlimits. Core drains it in every planning cycle with anactivityRPC (epoch and sequence cursor kept in Core, at most 4 × 500 entries per cycle) and every five minutes asks for index size and per-owner counts. Hosts announce this with thesession-search-metrics-v1capability; older Hosts are not asked. Owner entries are accepted only for owners Core planned on that Host. - Core. Provider embeddings are counted where the controller makes them; each query is measured while it runs (provider call, Host calls, ranking) and logged when it returns, including failures.
- Storage. Everything lands in Core's metrics store (source access required): owner
data in the owner's scope (labelled with the Host), index size, passes,
pauses and limit changes in
host:<id>and shown only to owners with sessions on that Host. Rollups are kept for 7 days (minutes), 30 days (hours) and 365 days (days); pause and limit events for 90 days. - Search log privacy. A query is stored as an event in the owner's own scope for 30 days: its text with secrets redacted (credential patterns and long letter-and-digit tokens), its length, up to 12 lowercased terms, an owner-salted hash that groups repeated queries, the names of the filters used, the mode, result count, hit mix, stage times and Host counts. Counts and timings remain in the rollups for a year without text. No other user can read or annotate it.
- Opened results. The console reports which result of a logged search the
owner opened (detail, go to, restore, hand off, Task Graph) with
POST /v1/session-search/log/:logId/open; the query answer carrieslogId. Coverage is computed live when the view opens (coverageRPC per Host);live=falseanswers from recorded samples only.
CLI and operator MCP return the same view: devctl search stats [24h|7d|30d|365d] [--sampled] and the operator tool session_search_statistics (period,
sampled), which runs that CLI command. Both build the request with
sessionSearchStatisticsRequest from @devctl/contracts/session-search.
Search and results
POST /v1/session-search/query takes query and optional filters: groupId
(project), projectId (repository), agent, sessionId, from/to and
states (running, frozen, stopped) and archived (any, only, exclude; see
session archive). Filters narrow the owner's session
allowlist before any Host is asked. The controller embeds the query with the
provider (a local Host embeds it itself), asks each Host with matching sessions
in parallel, fuses each Host's text and vector ranks with RRF (k = 60) and
returns, per result: project, repository, agent, session, state, kind
(transcript or task), role, time, the matched text with up to ~600
characters of context and highlight ranges, the matched channels, and two native
cursors — one ending at the match, one ending two messages later — and
archived ({ archivedAt, location: "s3", pruned }, or null).
Each result also carries reason, textRank and vectorRank. The
controller computes the reason and the final highlight ranges from the query and
the snippet (explainSessionSearchMatch in @devctl/contracts/session-search):
- the Host's FTS5 ranges are kept;
- every query term is matched against the snippet's words again, so a vector hit is highlighted too: exact (same word, case-insensitive), word form (a prefix, a shared stem or folded diacritics, e.g. "миграция" → "миграции"), or transliteration (Cyrillic ↔ Latin, standard and look-alike spellings, e.g. "емр" → "EMR", "doker" → "Докер");
- a result with no term match is meaning only (semantic). The snippet sentence that shares the most character trigrams with the query is marked as the closest span; when none shares any, nothing is marked. Sentence embeddings are not computed per result, so this span is an approximation.
The console derives the same explanation itself when an older controller sends
no reason.
The query route accepts an optional progressId query parameter (8–64 letters,
digits, - or _). While the query runs, GET /v1/session-search/progress/:progressId returns its stage — embedding (a
provider source only), searching with Hosts answered / unavailable of Hosts
asked, ranking, done — and the planned stages. Progress is kept per owner
in memory for one minute (at most 256 queries); another owner's ID reads as
404.
The Search screen (terminal and web) fills the content pane: the native list computes how many rows fit from the terminal rows left under the header lines, and the web list grows with the pane and scrolls.
- a query box and a filter bar that always shows the current values with their
keys:
sstate,ttime,pproject,yrepository,aagent,varchived (buttons in the web console). The lists come from the user's own projects, repositories and agents. Keys follow the physical key on Russian and Ukrainian layouts (ыiss); - results are grouped by project / repository, with a count per group, in the
order of each group's best result.
←/→(or Enter on a header) collapse and expand a group;↑/↓,PgUp/PgDnand the mouse wheel move; - each result shows agent · session · state · age, then the role (or "task"),
the matched channels with their ranks, the reason and the fused score, then up
to three lines of the match with the highlighted spans;
ishows how hybrid search ranks results; - a running query shows its stage, step and elapsed time, and
Esc(or Cancel in the web console) cancels it. With a Core that cannot report progress, the line shows the number of online Hosts being searched. Editing the query drops the previous results and any query still in flight at once; a late answer to an older query is ignored; - when vectors are missing or conversations are not read yet, a line such as "Index: 4,978 of 28,356 chunks embedded, results may be incomplete" warns that results may be missing;
- Enter opens the exchange around a transcript match (the messages before it, the match marked, and the next two messages) or the Task Graph at a task;
ogoes to the session (attaches when it is running; otherwise it says to restore it),rrestores it through the existing flows (Thaw for a frozen session, honouring the Thaw target default; resume for a stopped one),hhands it off through the existing handoff flow, seeded with the matched text as the handoff's focus. Hand-off is offered only for Claude and Codex sessions or sessions with a saved handoff; otherwise the reason is shown.
The handoff seed is the owner's own matched text: POST /v1/sessions/:id/handoffs/prepare accepts an optional context (1–600
characters) that becomes the focus in the handoff's goal. The agent prompt is
unchanged.
Routes (server-route-catalogue.ts; handler-scoped user routes, except index
now, which is a handler-scoped administrator route):
| Route | Purpose |
|---|---|
GET /v1/session-search/settings |
Settings, eligible secrets, per-Host index status and statistics |
PUT /v1/session-search/settings |
{ expectedRevision, embedding }; revision-checked |
POST /v1/session-search/query |
Owner-scoped hybrid search; optional ?progressId= |
GET /v1/session-search/progress/:progressId |
Stage of one of the owner's in-flight queries |
GET /v1/session-search/statistics |
period (24h, 7d, 30d, 365d), live; the owner's statistics view |
POST /v1/session-search/log/:logId/open |
{ rank, kind, action }; counts an opened result of the owner's own logged search |
POST /v1/session-search/index-now |
Administrator: { hostId? }; queues index now per Host (queued, unsupported, offline) |
Limitations
- Indexed sources are Claude and Codex native transcripts and Task Graph tasks. devctrl's own messages and retained archives are not indexed; other agent CLIs have no supported transcript parser. Control Room chats and CLI/MCP search are out of scope; CLI/MCP read the search statistics and start index now only.
- A Host indexes at most 512 conversations and 2,000 tasks per plan (running sessions first, then the most recently active); the rest wait.
- Only the current conversation of a session has history cursors; results from earlier conversations open the session.
- A frozen session archived to S3, or a session whose history was freed after a session archive, keeps its already indexed chunks but is not indexed further until it is restored.
- FTS uses prefix matching without stemming; the vector channel covers inflection and cross-language matches once vectors are ready.
- Cancelling a query in the console stops waiting for it; the controller still finishes the bounded Host fan-out it already started.
- The local model is installed per Host; a Host without sessions of a local owner does not install it.
Remaining steps
- Index devctrl's own messages, delegations and handoffs with the same owner rules.
- Grow the evaluation set from real (owner-approved) transcripts so close models can be ranked.