Commit Graph
17 Commits
Author SHA1 Message Date
Jiayuan Zhangandmultica-agent 231dedb552 MUL-8034: chore: clean historical comments across the codebase (#9274)
* chore: clean historical comments (MUL-8034)

Co-authored-by: multica-agent <[email protected]>

* docs: preserve pending Composio expiry contract

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: multica-agent <[email protected]>
2026-10-15 21:54:33 +08:00
e7742b052b MUL-7776: fix(api): stop returning raw error text in 500 responses (#8905)
* fix(api): stop returning raw error text in 500 responses

39 handlers answered a 500 with the message followed by err.Error(),
sending database and store errors (constraint names, SQL states, Redis
errors) to API clients, while the other 720 500 responses in the handler
package send a fixed message. These now send their fixed message too and
log the error with the request's attributes, as the handlers that already
logged it do.

* fix(api): drop raw error text from the task callbacks and skill import 500s

completeTask and failTask sent err.Error() as the whole 500 message, and
the structured skill import put it in the result's Reason on its three
500s. They now send fixed messages and log the error. The task callbacks
keep their 500, so the daemon still retries them: isTransientError reads
the status code, never the body.

* fix(api): close remaining raw error leaks

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: Yushen <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-10-10 12:17:42 +08:00
d1c7ec25b1 feat(agents): add live model catalog refresh (#7919)
Co-authored-by: Lambda <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-09-03 15:05:11 +08:00
a075e58b80 MUL-6961 feat(agent): discover Claude Code's model catalog from the CLI (#7917)
* feat(agent): discover Claude Code's model catalog from the CLI

Claude was the last first-party runtime answering "what models do you
have?" from a hand-maintained list. A static catalog cannot know what the
CLI in front of it can run, so every new Anthropic model opened a window
where Multica offered a model that 400s:

    API Error: 400 Claude Code 2.1.246 does not support this model;
    version 2.1.251 or newer is required.

Claude Code does have a discovery hook, just not on the command line: the
stream-json control protocol answers a `list_models` control request with
the rows its own /model picker renders. Ask the installed binary and the
window closes at the source — a model it cannot run is either absent or
flagged disabled, computed against the real CLI version and the logged-in
account.

Verified against 2.1.223/228/246/258. The three older builds return Fable
5.1 as a disabled row carrying "Update to 2.1.255+ to use Fable 5.1";
2.1.258 returns it as a normal pick. Those rows are the reason no
per-model minimum-version table is needed here: upstream already computes
one, including the remedy copy, which we forward verbatim and grey out.

No version gate guards the request. An unsupported subtype is answered,
not ignored — every build tested replies "Unsupported control request
subtype" in ~2s and exits 0 — so an old CLI costs one cheap round trip and
falls back. Gating would mean hand-maintaining exactly the kind of number
this change removes.

Notes:

- The static catalog stays, now as the fallback and as the server-side
  accept list. It is flagged Catalog.Fallback so a discovery failure is
  not pinned in the server's day-long cache (MUL-5549).
- Discovery replaces rather than merges, as with every other dynamic
  provider. The picker narrows to the CLI's own list; agents already
  pinned to an older model keep working, and manual entry still reaches
  them.
- Effort levels now come from each row instead of scraping `claude
  --help` and narrowing through the hand-kept claudeModelEffortAllow.
- Catalog ids keep the context-window tag, because `claude-opus-5[1m]` is
  what the CLI would really run for that row. That exposed a latent bug
  in two capability lookups, which normalized the query side but not the
  catalog side; both now normalize both, or a model would fail to find
  its own effort picker.

MUL-6961

Co-authored-by: multica-agent <[email protected]>

* fix(agent): keep unrunnable Claude models out of the selectable catalog

Review found the first cut carried "this CLI cannot run it" as a flag on
the model itself. That was wrong in a way a flag cannot fix.

Only the create-agent dropdown learned the flag. The inspector picker and
the agent builder read the same list and kept offering the row, so
editing an existing agent still reached the 400 this change exists to
prevent. And the part no amount of teaching fixes: an already installed
desktop client has never heard of the field, so it renders the row as an
ordinary model and persists a model id the CLI rejects — exactly what the
repo's mixed-version API rule forbids.

Unavailable models now travel in their own list, end to end: daemon
report, store, catalog cache, and the client result. `models` contains
only selectable entries. The inspector and the builder needed no changes
to become correct — they read `models`, and the rows are no longer in it
— and an old client is correct for the same reason. Both pickers render
the advisory rows through one shared component that takes no click
handler, so nothing can turn them back into controls.

Two more review findings:

- The default row was dropped when no sibling row named the same model,
  which an org-restricted install can produce. That is not one missing
  picker entry: ValidateThinkingLevelWith resolves an empty model — the
  common "follow the CLI default" configuration — through the entry
  flagged Default, and fails closed without one, silently discarding the
  user's effort on every such task. It is now materialised, keeping the
  row's own capabilities. The test that claimed to cover this asserted
  the opposite of its own comment; rewritten, plus a case proving an
  unrunnable default is never materialised.

- "One cheap round trip" for a CLI without list_models was one round trip
  PER TASK: every Claude task carrying a thinking_level loads the catalog
  before it starts, and cachedDiscovery refuses to memoise a fallback.
  The structured unsupported-subtype answer is now remembered per
  (command, CLI version) — a property of the binary, not a bad moment —
  so an old CLI is probed once. Transient failures stay uncached and
  retryable, and the version in the key makes an upgrade take effect at
  once rather than after a TTL. Detecting it costs ~0.01s against the
  ~1.7s it saves, and the fallback path already paid for it.

The discovery timeout keeps its 20s ceiling, now with the measurements
that justify it: 1.57-2.19s over five runs against 2.1.258, 1.64-1.79s
against 2.1.246. The ceiling is for a CLI that never answers, not a slow
one.

Verified against real binaries: 2.1.246 offers four selectable models and
one unavailable row carrying upstream's own upgrade hint, 2.1.258 offers
Fable 5.1 normally, a missing binary falls back to the flagged static
catalog.

MUL-6961

Co-authored-by: multica-agent <[email protected]>

* docs(agent): record the downgrade case as an accepted boundary

Dynamic discovery governs selection, and only selection. An agent already
configured with a model whose CLI is later downgraded still starts and
still gets the upstream 400 — discovery cannot reach a value already
persisted.

That is deliberate, not an oversight, so it is written down next to the
code that invites the wrong fix. Clearing the configured model turns a
legible error into a silent config change the user did not make;
substituting a runnable one is worse, because it succeeds using a model
nobody chose. The downgrade is rare and the upstream error is accurate,
so failing loudly at task start is a fine way to report it.

MUL-6961

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-09-02 15:48:38 +08:00
mrlonelyandL 108407088c MUL-6836: fix(codex): allow explicit standard speed (#7756)
* fix(codex): allow explicit standard speed

* fix(codex): negotiate explicit standard support

---------

Co-authored-by: L <[email protected]>
2026-09-01 13:04:32 +08:00
eddb0dc2ad MUL-6884 feat(metrics): attribute agent_runtime lookups by product source (#7822)
* feat(metrics): attribute agent_runtime lookups by product source

MUL-6884 asked which product API is behind the hot GetAgentRuntime query.
It has no single answer: a dozen unrelated paths issue the same primary-key
read, so pg_stat_statements sums them into one row and can report that the
statement is busy without saying which behaviour is driving it. Deciding
between "raise the heartbeat interval" and "back off the 500ms browser polls"
needs that split, and today it is a guess.

Add multica_agent_runtime_lookup_total{source,result} and route every
production read through service.RuntimeLookup, which carries the source with
the connection and classifies pgx.ErrNoRows as not_found rather than as an
error — a deleted runtime is the daemon's self-heal signal, not an outage.

The labels are a closed enum (15 sources x 3 results, all prewarmed at zero)
because this fires on the hottest read in the system; an unclassified call
site degrades to "other" instead of minting a series. No query, no timing
histogram, and no extra SQL on /metrics — this PR only makes the existing
calls countable.

An AST lint test fails CI on any direct Queries.GetAgentRuntime call outside
the wrapper, since a bypassed call site would quietly undercount whichever
source it belonged to and send the follow-up work after the wrong caller.

Co-authored-by: multica-agent <[email protected]>

* refactor(metrics): drop the standalone handler lookup file

The two helpers it held are not an endpoint — no route, no HTTP handler, no
new file under cmd/ — but a new agent_runtime_lookup.go under internal/handler
reads like one, and a file name that has to be opened before it can be ruled
out is a bad file name.

Move both into runtime.go next to requireRuntimeReadAccess, where the rest of
the runtime read helpers already live, and name the metrics test after what it
tests. No behaviour change.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-08-31 17:46:56 +08:00
8442504201 fix(runtimes): hide private runtimes from non-owners (#7435)
* fix(runtimes): enforce private runtime access

* fix(runtimes): gate agent model discovery

* refactor(runtimes): reuse isRuntimeUsableForUser on runtime detail

The detail page re-derived the owner-or-public rule inline instead of
calling the shared helper every other surface in this PR uses. The two
disagreed on an ownerless runtime: the helper is fail-closed (matching
the server's read gate, which reuses canUseRuntimeForAgent), while the
inline copy rendered Usage for a public one and could only 404.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: m2keloop <[email protected]>
Co-authored-by: Bohan-J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-08-26 01:48:44 +08:00
25afaaf153 fix(agent): MUL-6606 surface hermes model-discovery failures instead of a silent empty catalog (#7464)
* fix(agent): surface hermes model-discovery failures instead of an empty catalog

Hermes model discovery swallowed every failure into an empty list with no
error, so the daemon reported "completed with 0 models" and the picker rendered
an authoritative empty dropdown — no reason, no hint, no prompt to type a model
in by hand. Hermes' own message already explains the common case ("No LLM
provider configured. Run `hermes model` ..."), and it was being discarded on
the discovery path.

- discoverHermesModels sets strictErrors. Unlike grok/codebuddy/copilot, hermes
  has no static catalog to degrade to, so the failure must reach the UI rather
  than be replaced by a stand-in list.
- Add a discovery-path annotation for "no LLM provider configured". It points
  at the daemon's own environment, which is the actual cause when a GUI-launched
  daemon cannot see credentials the user's shell exports. Deliberately not the
  task path's HERMES_HOME/custom_env advice: discovery is per-runtime and reads
  neither, so that remedy cannot populate the picker.
- Give acpDiscoveryProvider a per-provider timeout and set hermes to 40s.
  Measured against hermes 0.20.0: a healthy session/new answers in ~2s, but a
  hermes that cannot resolve its provider takes ~25s to say so. Under the shared
  15s cap that diagnosis was replaced by "context deadline exceeded" — the one
  failure text that tells the user nothing.
- Raise the client poll budget to the server's own 60s modelListRunningTimeout.
  Below that the client abandoned requests the server would still have answered.
  Manual entry stays live throughout the poll, so waiting costs the user nothing.
- Render the reason in the picker, next to the manual-entry input.

Also reconciles three comments that contradicted each other about which
providers report supported=false. ModelSelectionSupported names qwenpaw and
mcode; runtime_models.go claimed hermes, which if "aligned" to would have cost
hermes its model picker entirely.

Verified against the local hermes 0.20.0 CLI. Before: models=0, err=nil at the
15s cap. After: the full "No LLM provider configured" message plus the hint.

Co-authored-by: multica-agent <[email protected]>

* fix(agent): address review — poll budget window and Windows test portability

Two blockers from review, both confirmed against the code:

1. The client poll budget did not cover the server's full request lifetime.
   The pending and running windows are sequential, not overlapping: pending is
   measured from CreatedAt (30s to claim), running from RunStartedAt, which
   PopPending sets at claim time. Heartbeat pickup therefore does NOT consume
   the 60s running window, as the previous comment claimed. A request claimed
   at ~29s that then runs hermes' ~40s discovery reports at ~69s — past the 60s
   client budget, while the server still holds the record open. The user got a
   generic client-side "timed out" instead of the daemon's reason, which is the
   exact failure mode this PR exists to remove.

   POLL_TIMEOUT_MS is now derived from the two server constants plus slack, so
   the invariant is visible rather than asserted in prose. Letting the server's
   own timeout win is also strictly better copy ("daemon did not respond within
   30 seconds" beats "model discovery timed out"). Added two fake-timer tests:
   one for a late claim plus a full discovery, one proving the poll still
   terminates once both server windows have closed.

2. The new backend regression tests broke `go test ./pkg/agent` on Windows.
   They write and execute `#!/bin/sh` fixtures, but the file carried no build
   tag, and the Windows writeTestExecutable can only drop bytes on disk. Split
   along the same line as codebuddy_discovery_fallback_test.go: process-based
   cases move to hermes_model_discovery_unix_test.go (//go:build !windows), and
   the assertions needing no subprocess — hint wording, classification
   inertness, the timeout invariant — stay cross-platform against a literal of
   the ACP error string.

Also corrects the Go timeout invariant itself, which had encoded the same wrong
model of the running window, and refreshes two daemon.go comments that still
described discovery as capped at ~15s and as uniformly returning empty lists on
failure.

Verified: GOOS=windows go test -c builds, with the shell-fixture tests excluded
from the Windows binary and all four portable tests present. Both new core tests
fail against the previous 60s budget.

Co-authored-by: multica-agent <[email protected]>

* test(agent): make ACP timeout fixture deadline-safe

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-08-24 13:42:39 +08:00
44ce16d9b8 MUL-5549: fix(agent): stop reporting a failed model discovery as a real catalog (#6196)
* fix(agent): stop reporting a failed model discovery as a real catalog (MUL-5549)

Selecting the CodeBuddy runtime showed a model list that shares no IDs with
what the CLI actually supports, so every pick was an ID codebuddy rejects
(GH #6180). The list in the report is codebuddyStaticModels() verbatim: the
daemon had fallen back, but nothing downstream could tell.

discoverCodebuddyModels returned (staticModels, nil) on all three failure
paths, and copilot/cursor/grok do the same. A failed discovery therefore
arrived as a successful one, which defeated every guard built to catch it:
the daemon reported status "completed", the picker's discovery_failed hint
only renders on isError, and cacheableModelCatalog — whose own comment says
an empty list means transient failure — waves through a non-empty stand-in
and stores it as last-known-good for the full 24h serve window. One blip got
pinned as the answer for a day.

Discovery now returns a Catalog carrying a Fallback marker, which the daemon
forwards as an additive `fallback` field (older servers ignore it; an older
daemon omitting it keeps the previous behaviour). A fallback catalog is still
rendered — the picker stays populated and manual entry still works — but it
is kept out of both the daemon's 60s discovery cache and the server's catalog
cache. On the server it maps to Keep rather than Drop: a stand-in is no
grounds to evict a real catalog, matching how a `failed` report is treated.

Also stop codebuddyHelpOutput swallowing the exec error. CombinedOutput folds
in stderr, so a codebuddy whose `#!/usr/bin/env node` interpreter is missing
from a GUI-launched daemon's PATH had `env: node: No such file or directory`
parsed as help text — and cached as such for 60s.

Verified against CodeBuddy CLI v2.130.0: the parser itself is fine (16 models
from real --help), so this fixes the reporting of the failure, not the parse.

Co-authored-by: multica-agent <[email protected]>

* fix(agent): run codebuddy --help at most once per model-list request (MUL-5549)

Review catch on the previous commit. Model discovery and effort discovery both
read `codebuddy --help`, and the effort pass called it independently. That was
free while a failed --help was (wrongly) memoised, but once failures correctly
stopped being cached, the failure path ran the 35s command twice in a single
request — past the server's 60s running timeout, so the request timed out and
the late report was then discarded as stale. The user got nothing, not even the
fallback list the previous commit exists to preserve.

discoverCodebuddyModels now owns the thinking annotation, so the one help
capture feeds both catalogs, and the failure path uses codebuddyFallbackCatalog
to apply the static effort levels without exec'ing at all: whatever broke
--help for the model catalog breaks it for the effort catalog too.

Also strengthen the handler tests. They decoded into a struct declared in the
test rather than calling ReportModelListResult, so a wrong JSON tag or a
mis-wired cache branch would have passed. They now drive the real endpoint with
daemon auth and chi params, covering: a fallback report leaving a previously
discovered catalog intact, an older daemon omitting the field still warming the
cache, and an authoritative empty catalog still dropping the snapshot.

Both fixes are mutation-tested — reverting either makes the new tests fail.

Co-authored-by: multica-agent <[email protected]>

* docs(agent): correct codebuddy --help comments after the single-capture refactor (MUL-5549)

Review nit. The comments still described the pre-refactor call graph, where
both discoverCodebuddyModels and codebuddyEffortSuperset called
codebuddyHelpOutput and the cache was what stopped the duplicate run. The
effort parser now takes an already-captured string, and the single-invocation
guarantee is structural rather than cache-dependent — which matters, because a
failed --help is deliberately not cached, so a second caller would re-run the
full 35s timeout.

Also note on codebuddyHelpOutput that it has exactly one caller and why a new
one would reintroduce the bug.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: Bohan-J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-07-30 20:34:01 +08:00
c25a82eee0 perf(agents): fast model discovery on runtime switch (MUL-5444) (#6098)
* perf(agents): make runtime model discovery fast on runtime switch (MUL-5444)

Switching runtime in the agent creation form left the model picker
spinning for ~8-20s. Two costs stacked up:

- the list-models request sat in the store until the daemon's next
  scheduled heartbeat (0-15s, avg 7.5s of pure dead wait), and
- the daemon then enumerated the catalog locally (static for claude,
  but a CLI/ACP round trip up to ~15s for everyone else).

Both are addressed with the two standard techniques for a slow,
low-frequency, read-only operation: push instead of poll, and
stale-while-revalidate.

Push (removes the heartbeat wait):
- new additive `daemon:pending_work` hint, runtime-scoped, delivered
  through the existing daemon WS hub and the Redis relay so the API node
  holding the socket does the delivery.
- the daemon answers a hint with ONE immediate heartbeat and dispatches
  what it claimed. The hint deliberately carries no work, so nothing has
  to be un-claimed when delivery fails and a duplicate hint cannot
  duplicate work - PopPending stays the atomic claim.
- per-runtime coalescing plus a 1s floor keeps a caller-triggered hint
  from becoming a heartbeat amplifier.

Cache (removes the discovery wait on repeat opens):
- server-side per-runtime catalog cache (in-memory single-node, Redis
  multi-node) written on every successful report.
- a snapshot younger than 15min answers the POST immediately as an
  already-completed request; older than 60s it also enqueues a
  background refresh that only warms the cache.
- only supported, non-empty catalogs are cached; a completed-but-empty
  report invalidates instead, while a failed report keeps serving the
  last known good list.

Frontend: staleTime 60s -> 5min and gcTime 30min, so a runtime revisited
in the same session renders from cache and revalidates in the background
instead of showing the spinner again.

Compatibility: every wire change is additive. Old daemons ignore the
unknown hint type and keep using the scheduled heartbeat; new daemons
against an old server simply never receive one. The cached response is
shaped exactly like a completed live discovery apart from the optional
`cached` / `cached_at` markers.

Co-authored-by: multica-agent <[email protected]>

* fix(agents): address review on model discovery SWR (MUL-5444)

Sol-Boy's review on #6098 found the client cache could outlive the
server's own staleness promise, and that the two changed endpoints were
still cast rather than validated.

Must-fix 1 — client freshness now derives from the served answer.
`staleTime` was a flat 5min, so a 14-minute-old snapshot (which the
server returns while queueing its own refresh) was held as fresh for
another 5min: observable staleness became server window + client window,
and the refreshed catalog never reached the tab that triggered the
refresh. `staleTime` is now a function of the query data: a `cached`
answer is stale on arrival (bound stays the server's window alone, and
the next mount/focus picks up the refreshed snapshot), while a live
discovery — which just measured the truth — is trusted for the full 5min
so a cold runtime is never re-enumerated inside one form session.
`gcTime` stays 30min, so a revisited runtime still renders from cache and
revalidates in the background; the pickers gate their spinner on
`isLoading`, which stays false throughout.

Must-fix 2 — both model-discovery responses go through a zod schema.
`POST /api/runtimes/{id}/models` and its poll companion were casting
network JSON to `RuntimeModelListRequest`, which the root CLAUDE.md
API-compatibility rules forbid. Added a lenient schema (`status` stays
`z.string()`, `supported` defaults to true, `.loose()` keeps unknown
fields) plus a fallback record whose `status` is `failed`: a malformed
body now surfaces "discovery failed" with manual entry still usable
instead of a fabricated empty catalog or an endless spinner.
`resolveRuntimeModels` was tightened to match — only an explicit
`completed` is a catalog, so an unrecognised status is an error rather
than a silent empty list, and `supported` can no longer be `undefined`.

Nit — the in-memory catalog cache now deep-copies each entry's
`Thinking` (and its level slice) and `ServiceTiers`, so it delivers the
independent value its comment promises and matches the Redis backend's
JSON round-trip semantics.

Tests: staleTime policy for cached/live/no-data; a QueryObserver test
proving the refreshed catalog reaches the same client with no blank
loading state; unknown-status and omitted-`supported` handling; schema
tests for live, cached, old-backend and nine malformed shapes; client
tests that both endpoints degrade to an explicit failure; nested-field
mutation isolation for the cache.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: Eve <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-07-29 16:03:26 +08:00
6992c58de3 MUL-5185: add Codex Fast mode (#5821)
* feat(agents): add Codex fast mode (MUL-5185)

Co-authored-by: multica-agent <[email protected]>

* fix(agents): make Codex Fast override authoritative

Co-authored-by: multica-agent <[email protected]>

* fix(agents): remove Codex Fast config conflicts

Co-authored-by: multica-agent <[email protected]>

* chore: refresh checks after conflict resolution

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: Eve <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-07-23 15:40:18 +08:00
Bohan Jiangandmultica-agent 2bec2221d2 feat(agent): per-agent thinking_level for claude + codex (MUL-2339) (#2865)
* feat(agent): persist thinking_level per agent (MUL-2339)

Adds a nullable `thinking_level` column to the `agent` table so the
backend can route a runtime-native reasoning/effort token (e.g. Claude's
`xhigh`, Codex's `minimal`) through to the agent CLI on every dispatch.

The column is intentionally TEXT rather than an enum — Claude and Codex
publish overlapping but distinct vocabularies and we want the persisted
value to round-trip exactly through whichever CLI receives it. NULL is
the "use runtime default" sentinel that every downstream consumer reads
as "do not inject --effort / reasoning_effort".

This commit is just the storage layer (migration + sqlc); subsequent
commits wire it through the API, daemon, and agent backends.

Co-authored-by: multica-agent <[email protected]>

* feat(agent-backend): inject reasoning effort for claude + codex (MUL-2339)

Extends ExecOptions with a runtime-native ThinkingLevel string and wires
it into the Claude and Codex backends. Discovery is driven by the local
CLI so the daemon advertises whatever the host install supports rather
than a hand-maintained list that goes stale.

Per Elon's PR1 review:
- Claude: parses `claude --help` to learn the `--effort` superset and
  projects through a per-model allow-list (xhigh is Opus-only; max is
  session-only on the smaller models). Falls back to a conservative
  static list when the binary is missing or help drift hides the line.
- Codex: drives `codex debug models --output json` so per-model
  reasoning subsets and the documented default come directly from the
  CLI. The older config-error probe trick is gone — the JSON path is
  stable and doesn't pollute stderr with an intentional misconfig.
- Cache key includes (provider, executablePath, cliVersion) so a CLI
  upgrade invalidates entries that referenced the older help / catalog.

Per Trump's PR1 constraint, all three Codex injection points
(thread/start.config, thread/resume.config, turn/start.effort) flow
through one helper (`applyCodexReasoningEffort`) so they cannot drift
independently. The shared `codexReasoningCases` fixture in
`thinking_test.go` asserts the same value→{shape, key} contract at
each site for every level the runtimes know about.

Claude's `--effort` is also added to `claudeBlockedArgs` so a user
custom_args entry can't silently outvote the daemon-injected value.

Co-authored-by: multica-agent <[email protected]>

* feat(api): wire thinking_level through API + daemon contract (MUL-2339)

End-to-end plumbing for the per-agent reasoning/effort setting:

- AgentResponse / TaskAgentData now carry `thinking_level`; the daemon's
  claim response includes it and the daemon's executor passes it through
  to agent.ExecOptions, where the Claude and Codex backends already know
  what to do with it.
- ModelEntry on the runtime-models wire format gains a `thinking` block
  carrying `supported_levels` + `default_level` per model so the UI can
  render a runtime-aware picker without the server having to know about
  the local CLI install. `handleModelList` projects the agent-package
  catalog (including the new Thinking field) into the wire shape.
- CreateAgent / UpdateAgent gate the field with a synchronous provider
  enum check (claude / codex only today). UpdateAgent is tri-state:
  field omitted = no change, "" = explicit clear (new
  `ClearAgentThinkingLevel` query, mirrors the existing mcp_config null
  pattern), non-empty = validate then set.

Per Trump's PR1 review, the API NEVER auto-clears on a runtime/model
swap and ALWAYS returns 400 on an unknown literal value — same shape
across CreateAgent, UpdateAgent, and combined patches that move
runtime + level in one request. Per-model combination failures (e.g.
`xhigh` against a model that only supports up to `high`) surface as a
daemon-side task error, not a silent server-side rewrite.

TS types follow the same shape: `Agent.thinking_level`,
`CreateAgentRequest`/`UpdateAgentRequest` add the field, `RuntimeModel`
grows a `thinking` block. Older backends omit the field, which the
front-end treats as "no picker for this model" — installed desktop
builds keep working.

Co-authored-by: multica-agent <[email protected]>

* fix(agent): correct codex debug models argv + pin via runner test (MUL-2339)

`codex debug models --output json` is rejected by codex-cli 0.131.0 —
the subcommand emits JSON on stdout by default and has no `--output`
flag. Drop the flag and add `--bundled` to skip the network refresh
discovery doesn't need. Move the argv to a package-level var and add
a test that runs a fake `codex` to assert the binary actually
receives exactly `debug models --bundled`, so the contract can't
silently drift on the next refactor.

Also teach ValidateThinkingLevel to resolve an empty model to the
provider's default model entry. Without this, every default-model
task with a persisted thinking_level would be misjudged "unknown
model" by the daemon guard.

Co-authored-by: multica-agent <[email protected]>

* fix(api): reject runtime switch that would leave invalid thinking_level (MUL-2339)

A PATCH that changed `runtime_id` without touching `thinking_level`
used to silently keep the existing value, so a Claude agent storing
`max` could land on a Codex runtime where `max` is not a recognised
token at all, and the daemon would receive a literal-invalid level.

Hold the same "always 400 on literal-invalid, never silent coerce"
rule on this implicit path. When runtime_id changes and the existing
value is not in the new provider's enum, return 400 with the
recovery options (clear via `thinking_level=""` or re-set in the
same PATCH).

Add coverage for both the kept-when-still-valid and the rejected
cases, plus the two recovery paths (clear and replace).

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): guard runTask with per-model thinking_level validator (MUL-2339)

ValidateThinkingLevel existed but had no call site — `task.Agent.
ThinkingLevel` flowed straight into ExecOptions, so `xhigh` configured
on a non-Opus Claude model, or API-side stale values that escaped the
provider enum gate, would be injected anyway.

Run the validator before building ExecOptions. Invalid combinations
log a warning and drop the level instead of failing the task: the
agent still runs, just at the runtime's default reasoning effort.
Discovery errors fail open (keep the level, let the CLI surface any
objection) so a transient `claude --help` failure can't strand work.

Empty model is forwarded as-is; the validator resolves it to the
provider's default model internally per the cross-package contract.

Co-authored-by: multica-agent <[email protected]>

* chore(agent): drop stale `--output json` comments + unused scanner (MUL-2339)

Codex CLI's `debug models` subcommand emits JSON without an `--output`
flag, and `parseCodexDebugModels` never read from the bufio.Scanner.
Sync the comments with the actual invocation and remove the dead init.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: multica-agent <[email protected]>
2026-05-20 12:30:10 +08:00
Bohan Jiangandmultica-agent ba5b7db78e fix(server): persist ModelListStore across replicas via Redis (#2022)
* fix(server): persist ModelListStore across replicas via Redis

The model picker uses a pending-request pattern: the frontend POSTs to
create a request, the daemon pops it on its next heartbeat, runs
agent.ListModels locally, and reports back. Until now the store was a
plain in-memory map per Handler instance.

That works for self-hosted single-instance deploys but fails in any
multi-replica environment (Multica Cloud). Each replica has its own
map, so:

  POST /runtimes/:id/models               → request stored in replica A
  GET  /runtimes/:id/models/<requestId>   → polls land on B/C → 404
  daemon heartbeat                        → only A sees PendingModelList
  POST .../<requestId>/result             → daemon's report has to land on A

Success probability ~1/N². The visible symptom is "No models available"
in the picker for every provider, even those (Claude/Codex) whose
catalog is statically populated end-to-end.

Same shape of bug, same Redis-backed fix as multica-ai/multica#1557 did
for LocalSkillListStore / LocalSkillImportStore. Reuse the operational
playbook (namespaced keys, ZSET-backed pending queue, atomic
ZREM+SET-running via the shared Lua script) so we don't introduce a
second concurrency model for the same primitive.

Changes:
- Convert ModelListStore from struct to interface with context-aware
  methods. Add HasPending for cheap heartbeat-side probing.
- InMemoryModelListStore — single-node fallback, used when REDIS_URL
  is unset (self-hosted dev / tests).
- RedisModelListStore — multi-node implementation using the same key
  layout and Lua atomic claim as RedisLocalSkillListStore.
- Use RunStartedAt (not UpdatedAt) as the running-timeout reference
  point, matching the local-skill stores so subsequent UpdatedAt
  bumps don't reset the running clock.
- Heartbeat now uses the probe-then-pop pattern for the model queue
  (matching local-skills) so a slow Redis can't stall every connected
  daemon. Extends heartbeatMetrics + slow-log with probe_model_ms /
  pop_model_ms / probe_model_timed_out for parity.
- Wire the Redis backend in NewRouterWithOptions when rdb != nil.
- Tests for both backends. Redis tests gate on REDIS_TEST_URL so
  laptop runs without Redis still pass; CI provides it.

Co-authored-by: multica-agent <[email protected]>

* fix(server): persist RunStartedAt + retry model report on transient failures

Two follow-ups from PR #2022 review:

1. RedisModelListStore was dropping ModelListRequest.RunStartedAt on
   persistence — the field is tagged json:"-" so it doesn't leak into
   the HTTP response, which made plain json.Marshal(req) silently
   discard it. Across-node readers saw RunStartedAt=nil and
   applyModelListTimeout's running branch became a no-op, so the 60s
   running-timeout escape hatch never fired. CI's
   TestRedisModelListStore_RunningTimeout was failing on this exact
   case. Fix mirrors RedisLocalSkillImportStore's envelope pattern —
   wrap in an internal struct that re-promotes the field. HTTP shape
   stays clean. Adds a no-Redis unit test that pins the round trip.

2. Daemon's handleModelList called d.client.ReportModelListResult
   directly and swallowed any 5xx, leaving the pending request
   stranded in "running" until its 60s server-side timeout — exactly
   the failure mode the multi-node store fix was meant to eliminate.
   Generalize the existing local-skill retry helper into
   reportRuntimeResultWithRetry (kind: model_list / local_skill_list /
   local_skill_import) and wire handleModelList through a new
   reportModelListResult helper. Renames the test-overridable
   var localSkillReportBackoffs → runtimeReportBackoffs to match.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: multica-agent <[email protected]>
2026-05-03 11:13:34 +08:00
Bohan JiangandEve f628e48775 refactor(server): error-returning ParseUUID to prevent silent data loss
* refactor(server): make ParseUUID error-returning to prevent silent data loss (MUL-1410)

util.ParseUUID previously swallowed errors and returned a zero pgtype.UUID
on invalid input. When this zero UUID reached a write query (DELETE/UPDATE),
the SQL matched zero rows and the handler returned 2xx success — producing
silent data corruption. #1661 (DeleteIssue with identifier-style ID) was the
visible symptom; PR #1680 patched that one site, this commit closes the
class of bug.

Changes:

- util.ParseUUID now returns (pgtype.UUID, error). Add util.MustParseUUID
  for trusted round-trips that should panic on invalid input.
- handler/handler.go: parseUUID wrapper now calls MustParseUUID — any
  unguarded user-input string reaching it surfaces as a recovered panic
  (chi middleware.Recoverer → 500) instead of silently corrupting data.
  Add parseUUIDOrBadRequest(w, s, fieldName) for handler entry points.
- Convert every Queries.Delete*/Update* call site reachable from raw user
  input (autopilot, comment, project, skill, skill_file, label, pin,
  attachment, feedback, issue assignee, daemon runtime, workspace) to
  validate UUIDs explicitly with parseUUIDOrBadRequest, returning 400 on
  invalid input. Where a resolved entity.ID is already in scope, write
  queries now use it directly instead of re-parsing the URL string.
- Update getWorkspaceMember + loadIssueForUser to handle invalid UUIDs
  gracefully (404/400 instead of panic).
- Update util/middleware/cmd-level callers (subscriber_listeners,
  notification_listeners, activity_listeners, scope_authorizer,
  middleware/workspace) to use the error-returning API.
- Add server/internal/util/pgx_test.go covering valid/invalid input and
  the MustParseUUID panic contract.
- Add TestDeleteIssueByIdentifier + TestDeleteIssueRejectsInvalidUUID
  regression tests in handler_test.go (the original #1661 bug + the
  invalid-input case).
- Document the handler UUID parsing convention in CLAUDE.md so the rule
  is enforceable in future PR review.

* fix(server): address GPT-Boy review of #1748

P1 fixes from PR #1748 review:

1. Migrate remaining request-boundary UUIDs to parseUUIDOrBadRequest so
   malformed input returns 400 instead of panic/500. Was missing on:
   - issue.go: workspace_id in CreateIssue/ChildIssueProgress/ListIssues/
     SearchIssues/BatchUpdateIssues/BatchDeleteIssues; project_id /
     parent_issue_id / lead_id / assignee_id / assignee_ids / creator_id
     filters; batch issue_ids and assignee/parent/project fields in
     BatchUpdateIssues (skip on bad input via util.ParseUUID, matching
     the existing per-row continue semantics).
   - project.go: project id + workspace_id in GetProject/UpdateProject/
     DeleteProject; lead_id in CreateProject/UpdateProject;
     workspace_id in ListProjects + SearchProjects.
   - handler.go: resolveActor now uses util.ParseUUID for X-Agent-ID /
     X-Task-ID headers; invalid UUID falls back to "member" (matches
     pre-existing semantics) instead of panicking.
   - issue.go: validateAssigneePair returns 400 on invalid workspace_id
     instead of panicking.

2. Fix issue:deleted WS event payloads to emit uuidToString(issue.ID)
   instead of the raw URL string. After an identifier-path delete
   ("MUL-7"), the previous payload would have leaked the identifier to
   subscribers, leaving stale entries in frontend caches that key by
   UUID. Updated DeleteIssue (issue.go:1341) and BatchDeleteIssues
   (issue.go:1641). The slog "issue deleted" log line also now records
   the resolved UUID so logs match the WS payload.

3. Extend TestDeleteIssueByIdentifier to subscribe to the bus and
   assert issue:deleted.payload.issue_id is the resolved UUID, not
   the identifier.

* fix(server): validate remaining reviewed UUID inputs

* fix(server): validate remaining handler UUID inputs

* fix(server): finish request boundary UUID audit

* fix(server): validate remaining request body UUIDs

* fix(server): validate runtime path UUIDs

* fix(server): validate remaining audit UUID inputs

---------

Co-authored-by: Eve <[email protected]>
2026-04-28 14:50:28 +08:00
Bohan Jiang 683ff132ca fix(server/heartbeat): probe/claim split + slow-log + model-list running timeout (#1644)
Mitigates #1637 and the related model-discovery failure in MUL-1397 by bounding the /api/daemon/heartbeat hot path with an ack-safe probe/claim split, adding structured slow-log attribution, and closing the ModelListStore running-state gap. See PR description for details.
2026-04-25 02:06:00 +08:00
Bohan Jiang 6fd1255873 feat(runtimes): remove Test Connection / runtime ping feature (#1554)
* feat(runtimes): remove Test Connection / runtime ping feature

The Test Connection action invoked a real single-turn agent run to verify
runtime connectivity. In practice it was expensive (reuses none of the
normal task exec env, so it also gave misleading results) and low value —
daemon heartbeat + Online status already covers the "is the runtime
alive" question. Dropping the whole end-to-end probe path:

- deletes server handler and in-memory PingStore
- drops pending_ping from the heartbeat response and daemon poll loop
- removes daemon.handlePing, PendingPing, ReportPingResult
- removes the CLI `multica runtime ping` command
- removes the PingSection UI block and RuntimePing types / api methods

* docs: fix runtime CLI subcommand list in product-overview
2026-04-23 16:18:21 +08:00
Jiayuan Zhang b291db11c2 feat(agents): add per-agent model field with provider-aware dropdown (#1399)
Adds a first-class `model` field on agents so users can pick the LLM model from the create / settings UI instead of editing `custom_env` / `custom_args`. Each provider's dropdown is populated from the live CLI when possible (`opencode models`, `pi --list-models`, `openclaw agents list --json`, `cursor-agent --list-models`, hermes ACP `session/new` → `SessionModelState`), with a static catalog for providers that don't enumerate.

Daemon resolves the runtime model as `agent.model → MULTICA_<PROVIDER>_MODEL → ""` — empty passes through so each backend's CLI picks its own default, avoiding static-guess drift.

Per-provider honouring:
- Claude / Codex / OpenCode / Cursor / Gemini / Pi / Copilot — CLI `--model` / thread payload.
- OpenClaw — `opts.Model` is mapped to `--agent <name>` (the CLI rejects `--model`).
- Hermes — `session/set_model` ACP RPC; stderr is sniffed for provider-level errors so HTTP 4xx from the configured LLM surfaces instead of "empty output"; explicit-model failures mark the task `failed`.

Supporting changes: migration 050 adds `agent.model`; daemon ↔ server heartbeat piggyback carries a model-discovery request; new REST endpoints under `/api/runtimes/{id}/models`; `multica agent create --model` / `update --model`; shared `ModelDropdown` in `packages/views/agents` (searchable, creatable, provider-grouped, default-badge, runtime-supported gate).
2026-04-21 00:06:34 +08:00