Decisions API: automatic result review and statistics

The controller can use TypeSafe Jev for launch suggestions and automatic, read-only review of new results and session handoffs. This integration is optional. Without TYPESAFE_API_KEY, or when the optional SDK or TypeSafe API is unavailable, the controller and every existing launch, AgentRun, verification, and provider-session flow keep working. The advisor returns not_configured or unavailable; it never blocks those flows.

Configure TYPESAFE_API_KEY on the controller process, not in a Project container or agent prompt. The local Compose stack passes through a non-empty TYPESAFE_API_KEY from the developer-machine environment when it is started or recreated; leaving it unset preserves existing behavior. Treat the Docker container environment as sensitive (users with Docker admin access can inspect it). No key is sent to Claude/Codex. The controller uses the official JavaScript SDK @typesafe-ai/sdk and TypeSafeClient.systemOne with jev-latest; SDK import is deferred until a manual request or a newly queued automatic result review. Calls have a five-second timeout and no retries. Jev calls may incur TypeSafe charges; the automatic worker checks its local durable queue every second. It does not make inference calls when the queue is empty.

Operator commands:

devctl launch advise <project> --provider codex --task "Fix the failing build"
devctl run triage <run-id>
devctl run triage-list [--project <project>] [--limit 1-10]

The equivalent admin-only routes are POST /v1/projects/:id/launch-advice (fixed {provider, task} body), POST /v1/runs/:id/result/triage, and POST /v1/result-capsules/triage (newest 1–10 capsules, optional project filter and maxAgeDays 1–3650). The batch response is ordered by Jev's attention probability; it does not inspect older capsules beyond the requested limit.

Launch advice chooses only an existing Project Launch Profile for the requested provider and an effectively enabled Skill from global/template/Project bindings plus the selected profile's already configured packs. none is a valid answer. Jev receives the bounded task summary and candidate names/descriptions, not grants, credentials, session transcripts, or arbitrary questions. Suggestions do not create bindings, grants, Services, accounts, or sessions. The existing deterministic host/account launch recommendations and all eligibility checks remain separate and authoritative.

Result triage sends a bounded projection of an already created Result Capsule: goal, recorded outcome, verification status and short check summaries, blocker summaries, and counts of files/artifacts. It asks two fixed questions: attention probability (noul) and a suggested next human step (choice). The response is advice, not a replacement for Result Capsule outcome, completion verification, VERIFIED, AgentRun state, NightPermit, or any lifecycle/permission decision. It performs no lifecycle actions. The controller stores bounded recommendation metadata in its local database (see telemetry below). Low confidence is displayed as-is; operators must inspect the capsule before acting. Do not put secrets in task summaries or result metadata that will be evaluated externally.

The local admin devctrl MCP offers the same fixed operations as jev_launch_advice (project, provider, task), jev_result_triage (runId), and jev_result_triage_list (optional project, limit 1–10). These are operator tools on the developer machine, not session-scoped tools. They call the CLI and return its result; each request can incur TypeSafe charges when configured. They do not accept a model, arbitrary state, or arbitrary questions. These operator triage tools are not exposed to remote sessions. The separate session-scoped decide_subagent_model operation serves managed launch hooks with validated session identity and bounded model/task inputs; it is not arbitrary advisor access or an authorization grant.

The question packs and candidate validation live in apps/controller/src/jev-advisor.ts. The controller rejects invented profile, Skill, or next-action labels and treats malformed responses as unavailable. There is deliberately no arbitrary-state/arbitrary-questions API.

Decisions API screen and feedback

Open Integrations → Decisions API in TUI/web (Tab moves focus into the screen). Reading or refreshing this screen makes no TypeSafe calls. It distinguishes key configuration from the last actual API attempt; a configured key is not proof of a working connection.

Click Settings or press w to choose what Decisions API may evaluate. The Settings button is visible in every Decisions view in both TUI and web. Its persistent controller checkboxes cover: subagent/execution model selection, launch profile/skill advice, approval tags and danger verification, result/handoff review, and CLI release importance. All are enabled by default. Existing provider/project budgets still apply. Disabling a use prevents new calls through its manual API and automatic path. GET/PUT /v1/decisions/usages serves the same settings; only the controller operator can change them.

Project/session badges say Optional report advice. An attention probability estimates whether a previously recorded result or handoff merits human review; it is not current agent state, an assigned task, or an executed action. Percentages are kept in full details. t from Projects opens a short summary of the source report, reported blocker, suggested check and next step. e expands the full report and API metrics. Reports can be old; their dates are shown. This advice does not send prompts, pause/resume the provider or grant access. Dashboard Attention opens the exact report with Enter or a click. Mark reviewed (x on the Dashboard, i in report details) saves a separate controller reviewedAt acknowledgment. It removes that report from Attention without deleting history or changing the useful/needs-attention calibration labels. A newer report is evaluated independently. Restore attention notice (i in the same report) clears the acknowledgment. The existing feedback endpoint accepts {reviewed: true} or {reviewed: false}; timestamps are assigned by the controller. Native subagent model routing is separate: it can return a model override before a delegated task starts; its result and cost forecast are displayed separately.

The Overview starts with projects, followed by providers. Press j or click Projects in the browser for a dedicated project breakdown: requests, automatic and manual reviews, errors, attention flags, latency, tokens, estimated cost, and monthly budget. Each project also shows its provider split. Projects with no requests remain visible. p, d, and f filter project, date range, and provider; request metrics follow these filters, while monthly spend always covers all providers and the entire calendar month. Press b to edit budgets.

The project view also identifies each recorded purpose: launch profile/skill selection, session handoff review, or result review. It lists suggestions and source outcomes, operator ratings, and net savings from comparable measured pairs, including the advice cost. A source outcome describes the reviewed report; it does not prove the suggestion was followed. Without paired measurements, savings stay unknown. The recorded model and tokens belong to the advisor, not the Claude/Codex workload. Press m for optional execution-model selection and an API-equivalent cost forecast (see below).

Native subagent routing follows operator automatic-review/policy settings; manual comparisons are available separately. A cheaper-model forecast identifies its assumed token volumes and comparison model; it remains an estimate until comparable-quality runs provide measurements. Claude/Codex subscription quota is not a per-token cash charge. Saving weekly quota and saving dollars are separate metrics; neither can be derived from the advisor's own token usage or from a model name alone.

Native subagent model selection

Managed PreToolUse hooks route Claude Agent/legacy Task and Codex spawn_agent calls before the child starts. The hook uses the existing session-scoped Unix-socket MCP broker; provider/controller credentials never enter the child environment. It returns the official updatedInput.model override while preserving every other tool argument. The parent session's model is unchanged. Codex requires permissionDecision: "allow" alongside updatedInput; omitting it causes the rewrite to be rejected. Claude accepts the input rewrite without that decision and retains its permission flow. Model selection is no longer a step in the parent launch wizard.

Routing requires a configured Decisions provider, Overview automatic review ON, and subagent routing ON. Press m (Models) to inspect or toggle the subagent policy (o), edit eligible models, token forecasts and minimum confidence, and save (s). Defaults are a 100000-input / 10000-output workload with a 20000-token maximum prompt and 0.8 minimum confidence: software policy assumptions, not observed usage. The task/baseline fields below the policy controls are an optional manual comparison, not an instruction to change a parent session.

Codex full-history forks (including omitted fork_turns, "all", and currently numeric legacy values) inherit the parent model and reject overrides. They are recorded as full_history_inherits_model without a paid decision call. For an independent child, use fork_turns: "none" and include the necessary task/context in its message. devctl never silently drops a fork's history. Unsupported explicit reasoning-effort overrides also keep their original model. Claude per-invocation models take precedence over subagent frontmatter/defaults, but organization model allowlists or forced-model policies can still substitute the effective model; returned overrides are not proof of actual child consumption or quality.

For example, a parent on Sol can ask an independent child to rename a variable and run a specified test; the hook may select Luna for that child. "none" omits the parent conversation, not the repository files. Include the relevant paths, requirements and expected result in the child's message. A child asked to continue a prior investigation with "all" retains that conversation and its inherited model in the current integration. See the official Codex subagent model configuration.

The regression matrix in apps/controller/src/decisions-subagent-routing.test.ts runs shipped lifecycle/routing scripts through a Unix socket and the real session-scoped MCP handler for both Claude and Codex. It covers simple edits selecting the smallest model, bounded changes selecting a middle model, complex tasks retaining the baseline, confidence at/below the threshold, invalid replies and provider failures. It also verifies full-history forks make no paid decision call, preserves all other launch arguments and parent settings, and checks forecasts separately from subscription savings. Only the external evaluator is mocked; these tests do not assert real Jev classification quality or launch paid child agents.

The hook obtains the baseline from the requested child model, an independent Codex child's configured defaultSubagentModel, or known current parent model (Claude's bounded transcript tail, Codex's official hook model field). An unknown baseline leaves the original launch untouched and makes no paid request or forecast. Child task text goes to the configured provider but is not persisted in Decisions telemetry. Hook tool-use IDs are deduplicated. Monthly project/provider budgets apply; low confidence, invalid replies, missing keys, errors and deadlines retain the original model. Controller routing has a 1.5s abort deadline (forwarded to supported adapters); the native bridge is bounded at 2.2s. An interrupted request can still have been billed: unknown cost remains visible rather than being counted as free or retried automatically.

Receipts retain a redacted native task label and task fingerprint, original and chosen models, suitability confidence, reason and cost assumptions. A choice is initially recommended. Supported native PostToolUse results link the child identity and confirm its execution arguments; Claude's resolvedModel also records any provider substitution as different_model. Unsupported result shapes remain unverified. Confirmation makes no claim about task quality or total child token use. Receipt updates do not charge for another decision. Automatic choices consider only the operator's permitted, same-provider models with a known lower workload price; no_cheaper_candidate incurs no advisor call. Missing optional advisor configuration also incurs no budget reservation and preserves normal native launches.

Temporary model capacity

An exact native child capacity failure marks that model busy for ten minutes. Claude/native launch failures use PostToolUseFailure; Codex children that fail after a successful spawn are linked from their persisted native error and terminal-turn events to the confirmed launch receipt. A subsequent authorized child spawn excludes the recently busy model and asks the configured Decisions provider whether a cheaper permitted model suits that specific child task. The receipt links the failed decision and distinguishes next_child_model, wait_for_baseline, and inherited-context limitations. Low confidence or an unsuitable cheaper model keeps the baseline. The cooldown expires, is scoped to the same managed session/process generation, and does not grant account access.

The integration cannot remotely switch a running parent or retry a native child directly. It preserves full-history forks and needs a new authorized independent child launch to apply the fallback. It never discards context to force a cheaper model. Parent sessions can instead retry their interrupted turn through the existing managed input path. This server-side scheduler needs no open client. It admits only an exact Codex capacity error followed by native turn termination (or explicit no-retry evidence), or Claude's official StopFailure: overloaded. Generic 429/503, quota, authentication, billing and noncapacity errors are excluded.

Retries wait at least 30, 60, 120 and 300 seconds, with four total attempts and a 20-minute episode limit. Attempts persist before input delivery; repeated error observations and server restarts cannot replay the same attempt. A native busy turn, active/unknown children, pending user question/approval, autonomous Run, unready account, unauthorized compute owner or controlling input owner blocks admission. Read-only watchers do not own input. The host rechecks native evidence and requires an unattached live provider pane. Every terminal input operation is fenced by an expiring lease inside the container; ambiguous delivery stops retries. Positive later Stop evidence resolves the busy status.

In Models, c toggles capacity retries; Overview automatic review and model selection must also be enabled. Sessions show MODEL BUSY, the next retry time and attempt count. Model decisions show recommendation/launch/failure evidence and API-equivalent forecast savings separately from measured consumption.

Existing managed sessions that already registered devctl's wildcard lifecycle command receive an atomic script update through host MCP recovery, without a provider or Project restart. Ordinary tools do not spawn the routing bridge. The host writer uses Linux FD-pinned directories and no-follow, exclusive creation; unsupported host platforms do not pretend a hot install succeeded. New managed launches materialize the same hook. Unmanaged/custom hook layouts need explicit integration; the host does not overwrite arbitrary commands.

The native contract is documented in Claude PreToolUse hooks, Claude subagent model precedence, Codex hooks, and the official Codex full-fork implementation.

The forecast is baseline API cost - selected API cost - actual priced advice cost. Missing tariffs, token assumptions or advisor metering leave the result unknown. Negative values mean extra cost. Both execution models are assumed to use the same total tokens; successful completion, equivalent quality, retries, prompt caching, tools, regional premiums and speed tiers are not measured. Long-context prices use the supplied maximum prompt tier for the whole forecast. For subscriptions this is only an API-equivalent comparison, not cash or weekly quota savings. Actual before/after measurements remain separate.

Model selection is downgrade-only. Both the manual selection API and native subagent router may retain the baseline or select a strictly lower capability tier with lower known token tariffs. A cheaper newer model in the same or a higher tier is not a downgrade. Unknown tiers/prices keep the baseline. Stored receipts are checked again before applying them, including against a changed session default; old advice cannot silently upgrade it. This restriction does not prevent an operator from explicitly choosing another model in Claude/Codex.

Opt-in tests against real Jev

pnpm test:live:jev runs eight synthetic cases against the actual configured Jev adapter using TYPESAFE_API_KEY from the test process environment. The suite exercises the shipped session router and request/response parsing without mocking Jev: two mechanical tasks must select a cheaper lower-tier model, while an unsafe migration and an ambiguous security redesign must retain the baseline, for both Claude and Codex. It also checks call metering and forecasts.

These tests are skipped in the ordinary suite, even when a key is present. Only the explicit command (or DEVCTL_TEST_LIVE_JEV=1) enables them; enabling without a key fails clearly. Tests run sequentially with no retries and incur eight Decisions API requests. They never start Claude/Codex, send real project data, or change the production controller/history. Synthetic forecasts are not production savings or a measurement of downstream task quality. Native hook rewriting, provider failures and confidence boundaries remain covered by the deterministic tests in decisions-subagent-routing.test.ts.

Admin API: GET /v1/decisions/models?executorProvider=claude|codex and POST /v1/projects/:id/model-route with executorProvider, task, baselineModel, optional candidateModels, optional estimatedTokens: {input,output,maxPrompt}, billingMode, and optional minimumConfidence. The result supplies a receipt ID, selected model, confidence, fallback reason, typed {model} override for comparison, and forecast assumptions. Task text is sent only for selection; it is not saved in Decisions telemetry. The request receipt records the selection and forecast, separately from advisor API cost. GET/PUT /v1/decisions/subagents/settings configure the operator policy; session-scoped MCP route_subagent_model accepts toolUseId, task, optional requestedModel, parentModel, forkTurns and reasoningEffort. The controller derives the Project, account and provider from authenticated session identity.

The catalog/tariffs were checked on 2026-10-08 against the official Claude model catalog, Claude API pricing, Codex model catalog, and OpenAI API pricing.

All existing launch-advice and result-triage routes now retain the latest 500 requests in the controller SQLite database, surviving restarts. History starts with this version; earlier calls cannot be reconstructed. Records contain IDs, status, recommendation, model, local latency, SDK call attempts and reported token usage, recorded outcome, a rules baseline and operator ratings. Task text, raw model responses, credentials, transcripts and capsule check/blocker text are not copied into telemetry. Latency is per advisory request (launch advice may make two API calls). Failed attempts count too. Counters cover retained history, not lifetime usage. Token totals show the number of metered attempts alongside all attempts; missing usage is unknown, not zero usage. Dollar cost is estimated from the versioned public tariff, separately from paired workflow savings (below).

Select a row with arrows: y useful, n wrong, a needs human attention, z does not need attention, u clear ratings, r refresh. Attention labels apply only to successful result triage. p cycles projects and d cycles 7 days (default), 30 days, and all time. t, then t again sends up to five matching newest Result Capsules to TypeSafe; the confirmation explains the external data transfer and possible API charges. Launch recommendations remain accessible through the existing CLI/MCP commands and appear in this history as well.

The comparison uses the latest labelled successful advice per capsule, to avoid counting repeated evaluations as independent results. Jev flags attention at probability >= 0.5. The explicit baseline flags any non-verified outcome, blockers, missing/failed verification or stale verification. Both are compared against the same human attention labels. Handoffs have no independently verified completion, so their rules baseline requires attention. Missed problems and false alarms are counts within this labelled sample; no labels means no evidence, and these are not automatic quality or token-saving claims. Useful/wrong ratings are a separate subjective measure. The baseline is an evaluation rule, not the live Dashboard ATTENTION block.

Admin-only API: GET /v1/jev returns configuration and retained history; POST /v1/jev/:id/feedback accepts optional useful and humanAttention booleans (or null to clear). No endpoint exposes raw prompts or credentials. Automatic calls are controlled separately by the persisted setting below.

Add a key in TUI

In Decisions API, focus the content with Tab and press k. Paste the API key into masked input; Enter saves, Esc cancels, Ctrl+U clears. k also replaces a key. The controller encrypts it with AES-256-GCM using its existing master key, in a controller-only credentials table, separate from grantable Project secrets. The stored key overrides TYPESAFE_API_KEY, survives controller restarts, and applies immediately without restarting sessions. Saving makes no test request and does not verify validity; pending new automatic reviews can start once a key is available. No API returns the key. Admin-only PUT /v1/jev/key accepts {key} and returns only configuration status.

Cost and paired savings

The direct TypeSafe tariff for jev-1.13.0, verified 2026-09-29 at https://docs.typesafe.ai/models, is $0.042 per million input tokens; output is free. Each SDK response is priced using its resolved versioned model, and price/source/date are retained with request history. Unknown models and responses missing usage are not silently priced at zero. The screen shows priced/total calls and the sum of known estimated costs, not an invoice or account balance. Old one-call records with complete usage can be estimated with this published tariff; ambiguous old multi-call records remain unpriced. Prices are versioned code data, not refreshed by a paid API call. Update the tariff when published pricing changes.

c on a successful history row opens a paired-measurement form. Supply baseline and with-Jev run/report references, USD costs and elapsed seconds, then confirm whether task, conditions and outcome quality are comparable. These are explicitly operator-supplied measurements, not automatically collected experimental results. With-Jev cost must include all workflow spending except the selected Jev request; its estimated cost is added automatically. With-Jev elapsed time must already include the advisor overhead. Net savings = baseline USD - with-Jev USD - selected Jev request cost; time saved = baseline seconds - with-Jev seconds. Negative values are retained. Only fully priced, comparable pairs count; duplicate reference pairs within a project count once (newest first). x clears the selected comparison. No pair is manufactured from unrelated historical tasks. No paired data means no measured savings; advisory-only use adds API cost without automatically replacing work. The quality table versus deterministic rules remains a separate comparison.

All totals cover the retained history and selected project. The period filter limits displayed history/statistics by request date and manual batch candidates by report date. Capsule dates are shown on rows to distinguish an old result from a recent evaluation. The batch uses up to five concurrent calls, unrelated to the 500-record storage cap. An empty time window sends zero calls and prompts the operator to choose another scope.

Provider configuration and availability are documented in Decisions API providers (source access required). In the interface press v to configure providers, and f to filter history/statistics by provider.

Automatic integration

Automatic review defaults to enabled when this integration is deployed. With a configured provider, each new Result Capsule or persisted Session Handoff queues an evaluation. Agents publish a bounded structured checkpoint through session MCP report_session_handoff on task completion, blockers or handoff. It uses the existing managed handoff lifecycle, retains agent claims as unverified, and deduplicates identical reports per session across reconnects. Writing an arbitrary Markdown file in Exchange alone does not publish a managed checkpoint; older installations that only instructed agents to write files therefore produced no automatic reviews. No transcript is scraped and old files are not silently backfilled or billed. Ordinary terminal activity/idle detection does not send a transcript and does not imply task completion. Old reports are not backfilled on upgrade, key entry or re-enabling the setting.

The controller worker performs at most two automatic calls concurrently. Result creation and agent completion do not wait for inference. Durable source receipts prevent duplicate automatic attempts across controller restarts. A changed capsule is a new revision; a stale in-flight reply cannot replace the newer revision's status. Interrupted attempts are shown explicitly and are not retried, because the provider may already have billed them. Failures do not change session status, verification or permissions. Manual evaluation remains available.

o toggles automatic review in Decisions API. PUT /v1/jev/settings with {"automatic":false} does the same for administrators. Disabling skips queued work, allows in-flight calls to finish, and suppresses new queued work. Re-enabling does not replay skipped work. Pending new reports wait if no provider key exists.

The latest advice appears next to sessions in Projects (list and mosaic), inside Result Capsule detail and in Dashboard attention. It carries a timestamp and a suggested human action; it is not the current agent's lifecycle state. Marking a flag as a false alarm (z) removes it from Dashboard attention. A handoff's tests are reported claims, never silently promoted to independent verification.

Detailed statistics

In Decisions API, s opens statistics, arrows scroll, and s returns to history. Enter toggles the selected request detail. p selects the project, d changes 7 days / 30 days / all retained dates. Metadata refreshes every five seconds; refreshing is free and never triggers an evaluation.

Statistics cover request counts, automatic/manual calls, successful/unavailable responses, attention flags, mean/p50/p95/max latency, input/output token totals, metering coverage, known cost and pricing coverage. Breakdowns include projects, UTC days, providers and models. Queue receipts show pending/running/success/failure/interrupted and skipped counts separately from the request-date filter.

Quality statistics show useful ratings, true positives/negatives, false alarms, missed issues, precision, recall and accuracy against explicit human labels. No labels means unknown, not perfect accuracy. Repeated ratings are deduplicated by source and revision. Savings require comparable operator-measured pairs; API cost and quality estimates are not evidence of saved Claude/Codex tokens or money.

Statistics are bounded by the last 500 retained requests, not lifetime totals. Incomplete token or pricing coverage remains explicitly visible. Request detail shows source/session, report and evaluation times, model, action confidence, recorded outcome, cost source and feedback without exposing prompt text or keys.

Session badges open their exact retained decision with t or a web click. Details identify the Project, repository and session and show the source report’s known issues and next step. When opened from Projects/Sessions, Esc returns to the original view and selected session. The advice does not assign a task automatically.