Decisions API: automatic result review and statistics
The controller can use TypeSafe Jev for launch suggestions and automatic, read-only review of new results and session handoffs. This integration is optional. Without TYPESAFE_API_KEY, or when the optional SDK or TypeSafe API is unavailable, the controller and every existing launch, AgentRun, verification, and provider-session flow keep working. The advisor returns not_configured or unavailable; it never blocks those flows.
Configure TYPESAFE_API_KEY on the controller process, not in a Project container or agent prompt. The local Compose stack passes through a non-empty TYPESAFE_API_KEY from the developer-machine environment when it is started or recreated; leaving it unset preserves existing behavior. Treat the Docker container environment as sensitive (users with Docker admin access can inspect it). No key is sent to Claude/Codex. The controller uses the official JavaScript SDK @typesafe-ai/sdk and TypeSafeClient.systemOne with jev-latest; SDK import is deferred until a manual request or a newly queued automatic result review. Calls have a five-second timeout and no retries. Jev calls may incur TypeSafe charges; the automatic worker checks its local durable queue every second. It does not make inference calls when the queue is empty.
Operator commands:
devctl launch advise <project> --provider codex --task "Fix the failing build"
devctl run triage <run-id>
devctl run triage-list [--project <project>] [--limit 1-10]
The equivalent admin-only routes are POST /v1/projects/:id/launch-advice (fixed {provider, task} body), POST /v1/runs/:id/result/triage, and POST /v1/result-capsules/triage (newest 1–10 capsules, optional project filter and maxAgeDays 1–3650). The batch response is ordered by Jev's attention probability; it does not inspect older capsules beyond the requested limit.
Launch advice chooses only an existing Project Launch Profile for the requested provider and an effectively enabled Skill from global/template/Project bindings plus the selected profile's already configured packs. none is a valid answer. Jev receives the bounded task summary and candidate names/descriptions, not grants, credentials, session transcripts, or arbitrary questions. Suggestions do not create bindings, grants, Services, accounts, or sessions.
The existing deterministic host/account launch recommendations and all eligibility checks remain separate and authoritative.
Result triage sends a bounded projection of an already created Result Capsule: goal, recorded outcome, verification status and short check summaries, blocker summaries, and counts of files/artifacts. It asks two fixed questions: attention probability (noul) and a suggested next human step (choice). The response is advice, not a replacement for Result Capsule outcome, completion verification, VERIFIED, AgentRun state, NightPermit, or any lifecycle/permission decision. It performs no lifecycle actions. The controller stores bounded recommendation metadata in its local database (see telemetry below). Low confidence is displayed as-is; operators must inspect the capsule before acting. Do not put secrets in task summaries or result metadata that will be evaluated externally.
The local admin devctrl MCP offers the same fixed operations as jev_launch_advice (project, provider, task), jev_result_triage (runId), and jev_result_triage_list (optional project, limit 1–10). These are operator tools on the developer machine, not session-scoped tools. They call the CLI and return its result; each request can incur TypeSafe charges when configured. They do not accept a model, arbitrary state, or arbitrary questions. These operator triage tools are not exposed to remote sessions. The separate session-scoped decide_subagent_model operation serves managed launch hooks with validated session identity and bounded model/task inputs; it is not arbitrary advisor access or an authorization grant.
The question packs and candidate validation live in apps/controller/src/jev-advisor.ts. The controller rejects invented profile, Skill, or next-action labels and treats malformed responses as unavailable. There is deliberately no arbitrary-state/arbitrary-questions API.
Decisions API screen and feedback
Open Integrations → Decisions API in TUI/web (Tab moves focus into the screen). Reading or refreshing this screen makes no TypeSafe calls. It distinguishes key configuration from the last actual API attempt; a configured key is not proof of a working connection.
Click Settings or press w to choose what Decisions API may evaluate.
The Settings button is visible in every Decisions view in both TUI and web.
Its persistent controller checkboxes cover:
subagent/execution model selection, launch profile/skill advice, approval tags
and danger verification, result/handoff review, and CLI release importance.
All are enabled by default. Existing provider/project budgets still apply.
Disabling a use prevents new calls through its manual API and automatic path.
GET/PUT /v1/decisions/usages serves the same settings; only the controller
operator can change them.
Project/session badges say Optional report advice. An attention probability
estimates whether a previously recorded result or handoff merits human review;
it is not current agent state, an assigned task, or an executed action.
Percentages are kept in full details. t from Projects opens a short summary
of the source report, reported blocker, suggested check and next step. e
expands the full report and API metrics. Reports can be old; their dates are shown.
This advice does not send prompts, pause/resume the provider or grant access.
Dashboard Attention opens the exact report with Enter or a click. Mark reviewed
(x on the Dashboard, i in report details) saves a separate controller
reviewedAt acknowledgment. It removes that report from Attention without deleting
history or changing the useful/needs-attention calibration labels. A newer report
is evaluated independently. Restore attention notice (i in the same report)
clears the acknowledgment. The existing feedback endpoint accepts {reviewed: true} or {reviewed: false}; timestamps are assigned by the controller.
Native subagent model routing is separate: it can return a model override before
a delegated task starts; its result and cost forecast are displayed separately.
The Overview starts with projects, followed by providers. Press j or click
Projects in the browser for a dedicated project breakdown: requests, automatic
and manual reviews, errors, attention flags, latency, tokens, estimated cost, and
monthly budget. Each project also shows its provider split. Projects with no
requests remain visible. p, d, and f filter project, date range, and provider;
request metrics follow these filters, while monthly spend always covers all
providers and the entire calendar month. Press b to edit budgets.
The project view also identifies each recorded purpose: launch profile/skill
selection, session handoff review, or result review. It lists suggestions and
source outcomes, operator ratings, and net savings from comparable measured
pairs, including the advice cost. A source outcome describes the reviewed report;
it does not prove the suggestion was followed. Without paired measurements,
savings stay unknown. The recorded model and tokens belong to the advisor, not
the Claude/Codex workload. Press m for optional execution-model selection and
an API-equivalent cost forecast (see below).
Native subagent routing follows operator automatic-review/policy settings; manual comparisons are available separately. A cheaper-model forecast identifies its assumed token volumes and comparison model; it remains an estimate until comparable-quality runs provide measurements. Claude/Codex subscription quota is not a per-token cash charge. Saving weekly quota and saving dollars are separate metrics; neither can be derived from the advisor's own token usage or from a model name alone.
Native subagent model selection
Managed PreToolUse hooks route Claude Agent/legacy Task and Codex
spawn_agent calls before the child starts. The hook uses the existing
session-scoped Unix-socket MCP broker; provider/controller credentials never
enter the child environment. It returns the official updatedInput.model
override while preserving every other tool argument. The parent session's model
is unchanged. Codex requires permissionDecision: "allow" alongside
updatedInput; omitting it causes the rewrite to be rejected. Claude accepts
the input rewrite without that decision and retains its permission flow.
Model selection is no longer a step in the parent launch wizard.
Routing requires a configured Decisions provider, Overview automatic review ON,
and subagent routing ON. Press m (Models) to inspect or toggle the subagent
policy (o), edit eligible models, token forecasts and minimum confidence, and
save (s). Defaults are a 100000-input / 10000-output workload with a 20000-token
maximum prompt and 0.8 minimum confidence: software policy assumptions, not
observed usage. The task/baseline fields below the policy controls are an optional
manual comparison, not an instruction to change a parent session.
Codex full-history forks (including omitted fork_turns, "all", and currently
numeric legacy values) inherit the parent model and reject overrides. They are
recorded as full_history_inherits_model without a paid decision call. For an
independent child, use fork_turns: "none" and include the necessary task/context
in its message. devctl never silently drops a fork's history. Unsupported explicit
reasoning-effort overrides also keep their original model. Claude per-invocation
models take precedence over subagent frontmatter/defaults, but organization model
allowlists or forced-model policies can still substitute the effective model;
returned overrides are not proof of actual child consumption or quality.
For example, a parent on Sol can ask an independent child to rename a variable
and run a specified test; the hook may select Luna for that child. "none"
omits the parent conversation, not the repository files. Include the relevant
paths, requirements and expected result in the child's message. A child asked
to continue a prior investigation with "all" retains that conversation and
its inherited model in the current integration. See the
official Codex subagent model configuration.
The regression matrix in apps/controller/src/decisions-subagent-routing.test.ts
runs shipped lifecycle/routing scripts through a Unix socket and the real
session-scoped MCP handler for both Claude and Codex. It covers simple edits
selecting the smallest model, bounded changes selecting a middle model, complex
tasks retaining the baseline, confidence at/below the threshold, invalid replies
and provider failures. It also verifies full-history forks make no paid decision
call, preserves all other launch arguments and parent settings, and checks
forecasts separately from subscription savings. Only the external evaluator
is mocked; these tests do not assert real Jev classification quality or launch
paid child agents.
The hook obtains the baseline from the requested child model, an independent
Codex child's configured defaultSubagentModel, or known current
parent model (Claude's bounded transcript tail, Codex's official hook model
field). An unknown baseline leaves the original launch untouched and makes no
paid request or forecast. Child task text goes to the configured provider but is
not persisted in Decisions telemetry. Hook tool-use IDs are deduplicated. Monthly
project/provider budgets apply; low confidence, invalid replies, missing keys,
errors and deadlines retain the original model. Controller routing has a 1.5s
abort deadline (forwarded to supported adapters); the native bridge is bounded
at 2.2s. An interrupted request can still have been billed: unknown cost remains
visible rather than being counted as free or retried automatically.
Receipts retain a redacted native task label and task fingerprint, original and
chosen models, suitability confidence, reason and cost assumptions. A choice is
initially recommended. Supported native PostToolUse results link the child
identity and confirm its execution arguments; Claude's resolvedModel also
records any provider substitution as different_model. Unsupported result
shapes remain unverified. Confirmation makes no claim about task quality or
total child token use. Receipt updates do not charge for another decision.
Automatic choices consider only the operator's permitted, same-provider models
with a known lower workload price; no_cheaper_candidate incurs no advisor call.
Missing optional advisor configuration also incurs no budget reservation and
preserves normal native launches.
Temporary model capacity
An exact native child capacity failure marks that model busy for ten minutes.
Claude/native launch failures use PostToolUseFailure; Codex children that fail
after a successful spawn are linked from their persisted native error and
terminal-turn events to the confirmed launch receipt. A subsequent authorized
child spawn excludes the recently busy model and asks the configured Decisions
provider whether a cheaper permitted model suits that specific child task.
The receipt links the failed decision and distinguishes next_child_model,
wait_for_baseline, and inherited-context limitations. Low confidence or an
unsuitable cheaper model keeps the baseline. The cooldown expires, is scoped to
the same managed session/process generation, and does not grant account access.
The integration cannot remotely switch a running parent or retry a native child
directly. It preserves full-history forks and needs a new authorized independent
child launch to apply the fallback. It never discards context to force a cheaper
model. Parent sessions can instead retry their interrupted turn through the
existing managed input path. This server-side scheduler needs no open client.
It admits only an exact Codex capacity error followed by native turn termination
(or explicit no-retry evidence), or Claude's official StopFailure: overloaded.
Generic 429/503, quota, authentication, billing and noncapacity errors are excluded.
Retries wait at least 30, 60, 120 and 300 seconds, with four total attempts and a
20-minute episode limit. Attempts persist before input delivery; repeated error
observations and server restarts cannot replay the same attempt. A native busy
turn, active/unknown children, pending user question/approval, autonomous Run,
unready account, unauthorized compute owner or controlling input owner blocks
admission. Read-only watchers do not own input. The host rechecks native evidence
and requires an unattached live provider pane. Every terminal input operation is
fenced by an expiring lease inside the container; ambiguous delivery stops
retries. Positive later Stop evidence resolves the busy status.
In Models, c toggles capacity retries; Overview automatic review and model
selection must also be enabled. Sessions show MODEL BUSY, the next retry time
and attempt count. Model decisions show recommendation/launch/failure evidence
and API-equivalent forecast savings separately from measured consumption.
Existing managed sessions that already registered devctl's wildcard lifecycle command receive an atomic script update through host MCP recovery, without a provider or Project restart. Ordinary tools do not spawn the routing bridge. The host writer uses Linux FD-pinned directories and no-follow, exclusive creation; unsupported host platforms do not pretend a hot install succeeded. New managed launches materialize the same hook. Unmanaged/custom hook layouts need explicit integration; the host does not overwrite arbitrary commands.
The native contract is documented in Claude PreToolUse hooks, Claude subagent model precedence, Codex hooks, and the official Codex full-fork implementation.
The forecast is baseline API cost - selected API cost - actual priced advice cost. Missing tariffs, token assumptions or advisor metering leave the result
unknown. Negative values mean extra cost. Both execution models are assumed to
use the same total tokens; successful completion, equivalent quality, retries,
prompt caching, tools, regional premiums and speed tiers are not measured.
Long-context prices use the supplied maximum prompt tier for the whole forecast.
For subscriptions this is only an API-equivalent comparison, not cash or weekly
quota savings. Actual before/after measurements remain separate.
Model selection is downgrade-only. Both the manual selection API and native subagent router may retain the baseline or select a strictly lower capability tier with lower known token tariffs. A cheaper newer model in the same or a higher tier is not a downgrade. Unknown tiers/prices keep the baseline. Stored receipts are checked again before applying them, including against a changed session default; old advice cannot silently upgrade it. This restriction does not prevent an operator from explicitly choosing another model in Claude/Codex.
Opt-in tests against real Jev
pnpm test:live:jev runs eight synthetic cases against the actual configured Jev
adapter using TYPESAFE_API_KEY from the test process environment. The suite
exercises the shipped session router and request/response parsing without
mocking Jev: two mechanical tasks must select a cheaper lower-tier model, while
an unsafe migration and an ambiguous security redesign must retain the baseline,
for both Claude and Codex. It also checks call metering and forecasts.
These tests are skipped in the ordinary suite, even when a key is present.
Only the explicit command (or DEVCTL_TEST_LIVE_JEV=1) enables them; enabling
without a key fails clearly. Tests run sequentially with no retries and incur
eight Decisions API requests. They never start Claude/Codex, send real project
data, or change the production controller/history. Synthetic forecasts are not
production savings or a measurement of downstream task quality. Native hook
rewriting, provider failures and confidence boundaries remain covered by the
deterministic tests in decisions-subagent-routing.test.ts.
Admin API: GET /v1/decisions/models?executorProvider=claude|codex and
POST /v1/projects/:id/model-route with executorProvider, task,
baselineModel, optional candidateModels, optional estimatedTokens:
{input,output,maxPrompt}, billingMode, and optional minimumConfidence.
The result supplies a receipt ID, selected model, confidence, fallback reason,
typed {model} override for comparison, and forecast assumptions. Task text is
sent only for selection; it is not saved in Decisions telemetry. The request
receipt records the selection and forecast, separately from advisor API cost.
GET/PUT /v1/decisions/subagents/settings configure the operator policy;
session-scoped MCP route_subagent_model accepts toolUseId, task, optional
requestedModel, parentModel, forkTurns and reasoningEffort. The controller
derives the Project, account and provider from authenticated session identity.
The catalog/tariffs were checked on 2026-10-08 against the official Claude model catalog, Claude API pricing, Codex model catalog, and OpenAI API pricing.
All existing launch-advice and result-triage routes now retain the latest 500 requests in the controller SQLite database, surviving restarts. History starts with this version; earlier calls cannot be reconstructed. Records contain IDs, status, recommendation, model, local latency, SDK call attempts and reported token usage, recorded outcome, a rules baseline and operator ratings. Task text, raw model responses, credentials, transcripts and capsule check/blocker text are not copied into telemetry. Latency is per advisory request (launch advice may make two API calls). Failed attempts count too. Counters cover retained history, not lifetime usage. Token totals show the number of metered attempts alongside all attempts; missing usage is unknown, not zero usage. Dollar cost is estimated from the versioned public tariff, separately from paired workflow savings (below).
Select a row with arrows: y useful, n wrong, a needs human attention,
z does not need attention, u clear ratings, r refresh. Attention labels apply
only to successful result triage. p cycles projects and d cycles 7 days (default), 30 days, and all time.
t, then t again sends up to five matching newest Result Capsules to TypeSafe; the confirmation explains the external data transfer
and possible API charges. Launch recommendations remain accessible through the
existing CLI/MCP commands and appear in this history as well.
The comparison uses the latest labelled successful advice per capsule, to avoid counting repeated evaluations as independent results. Jev flags attention at probability >= 0.5. The explicit baseline flags any non-verified outcome, blockers, missing/failed verification or stale verification. Both are compared against the same human attention labels. Handoffs have no independently verified completion, so their rules baseline requires attention. Missed problems and false alarms are counts within this labelled sample; no labels means no evidence, and these are not automatic quality or token-saving claims. Useful/wrong ratings are a separate subjective measure. The baseline is an evaluation rule, not the live Dashboard ATTENTION block.
Admin-only API: GET /v1/jev returns configuration and retained history;
POST /v1/jev/:id/feedback accepts optional useful and humanAttention booleans
(or null to clear). No endpoint exposes raw prompts or credentials. Automatic calls are controlled separately by the persisted setting below.
Add a key in TUI
In Decisions API, focus the content with Tab and press k. Paste the API key into
masked input; Enter saves, Esc cancels, Ctrl+U clears. k also replaces a key.
The controller encrypts it with AES-256-GCM using its existing master key,
in a controller-only credentials table, separate from grantable Project secrets.
The stored key overrides TYPESAFE_API_KEY, survives controller restarts, and
applies immediately without restarting sessions. Saving makes no test request
and does not verify validity; pending new automatic reviews can start once a key is available. No API returns the key. Admin-only
PUT /v1/jev/key accepts {key} and returns only configuration status.
Cost and paired savings
The direct TypeSafe tariff for jev-1.13.0, verified 2026-09-29 at
https://docs.typesafe.ai/models, is $0.042 per million input tokens; output is free.
Each SDK response is priced using its resolved versioned model, and price/source/date
are retained with request history. Unknown models and responses missing usage are
not silently priced at zero. The screen shows priced/total calls and the sum of
known estimated costs, not an invoice or account balance. Old one-call records
with complete usage can be estimated with this published tariff; ambiguous old
multi-call records remain unpriced. Prices are versioned code data, not refreshed
by a paid API call. Update the tariff when published pricing changes.
c on a successful history row opens a paired-measurement form. Supply baseline
and with-Jev run/report references, USD costs and elapsed seconds, then confirm
whether task, conditions and outcome quality are comparable. These are explicitly
operator-supplied measurements, not automatically collected experimental results.
With-Jev cost must include all workflow spending except the selected Jev request;
its estimated cost is added automatically. With-Jev elapsed time must already
include the advisor overhead. Net savings = baseline USD - with-Jev USD - selected
Jev request cost; time saved = baseline seconds - with-Jev seconds. Negative values
are retained. Only fully priced, comparable pairs count; duplicate reference pairs
within a project count once (newest first). x clears the selected comparison.
No pair is manufactured from unrelated historical tasks. No paired data means no
measured savings; advisory-only use adds API cost without automatically replacing
work. The quality table versus deterministic rules remains a separate comparison.
All totals cover the retained history and selected project. The period filter limits displayed history/statistics by request date and manual batch candidates by report date. Capsule dates are shown on rows to distinguish an old result from a recent evaluation. The batch uses up to five concurrent calls, unrelated to the 500-record storage cap. An empty time window sends zero calls and prompts the operator to choose another scope.
Provider configuration and availability are documented in Decisions API providers (source access required). In the interface press v to configure providers, and f to filter history/statistics by provider.
Automatic integration
Automatic review defaults to enabled when this integration is deployed. With a
configured provider, each new Result Capsule or persisted Session Handoff
queues an evaluation. Agents publish a bounded structured checkpoint through session MCP
report_session_handoff on task completion, blockers or handoff. It uses the existing
managed handoff lifecycle, retains agent claims as unverified, and deduplicates identical
reports per session across reconnects. Writing an arbitrary Markdown file in Exchange
alone does not publish a managed checkpoint; older installations that only instructed
agents to write files therefore produced no automatic reviews. No transcript is scraped
and old files are not silently backfilled or billed. Ordinary terminal activity/idle detection does not send a
transcript and does not imply task completion. Old reports are not backfilled on
upgrade, key entry or re-enabling the setting.
The controller worker performs at most two automatic calls concurrently. Result creation and agent completion do not wait for inference. Durable source receipts prevent duplicate automatic attempts across controller restarts. A changed capsule is a new revision; a stale in-flight reply cannot replace the newer revision's status. Interrupted attempts are shown explicitly and are not retried, because the provider may already have billed them. Failures do not change session status, verification or permissions. Manual evaluation remains available.
o toggles automatic review in Decisions API. PUT /v1/jev/settings with
{"automatic":false} does the same for administrators. Disabling skips queued
work, allows in-flight calls to finish, and suppresses new queued work. Re-enabling
does not replay skipped work. Pending new reports wait if no provider key exists.
The latest advice appears next to sessions in Projects (list and mosaic), inside
Result Capsule detail and in Dashboard attention. It carries a timestamp and a
suggested human action; it is not the current agent's lifecycle state. Marking a
flag as a false alarm (z) removes it from Dashboard attention. A handoff's tests
are reported claims, never silently promoted to independent verification.
Detailed statistics
In Decisions API, s opens statistics, arrows scroll, and s returns to history.
Enter toggles the selected request detail. p selects the project, d changes
7 days / 30 days / all retained dates. Metadata refreshes every five seconds;
refreshing is free and never triggers an evaluation.
Statistics cover request counts, automatic/manual calls, successful/unavailable responses, attention flags, mean/p50/p95/max latency, input/output token totals, metering coverage, known cost and pricing coverage. Breakdowns include projects, UTC days, providers and models. Queue receipts show pending/running/success/failure/interrupted and skipped counts separately from the request-date filter.
Quality statistics show useful ratings, true positives/negatives, false alarms, missed issues, precision, recall and accuracy against explicit human labels. No labels means unknown, not perfect accuracy. Repeated ratings are deduplicated by source and revision. Savings require comparable operator-measured pairs; API cost and quality estimates are not evidence of saved Claude/Codex tokens or money.
Statistics are bounded by the last 500 retained requests, not lifetime totals. Incomplete token or pricing coverage remains explicitly visible. Request detail shows source/session, report and evaluation times, model, action confidence, recorded outcome, cost source and feedback without exposing prompt text or keys.
Session badges open their exact retained decision with t or a web click. Details identify the Project, repository and session and show the source report’s known issues and next step. When opened from Projects/Sessions, Esc returns to the original view and selected session. The advice does not assign a task automatically.