mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-09 01:46:46 +08:00
admin-workspace-create2
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Younes Beriane
|
262dcb5fa7 |
MUL-7816 fix(taskfailure): classify a 403 usage limit as provider quota, not auth (#8967)
* fix(taskfailure): classify a 403 usage limit as provider quota, not auth Kimi Code reports an exhausted usage window as HTTP 403. The bare 403 auth rule ran before the quota rule, so the failure was filed as provider_auth_or_access. Match the usage-limit witness before the auth rule. The concurrent-request-limit case still comes first, and a plain 403 still classifies as auth. Fixes #8965 * fix(taskfailure): upgrade old-daemon 403 usage limits and beat the token rule Move the usage-limit check into a shared isUsageLimit403 helper that requires HTTP 403. Classify checks it right after the concurrent-request case, so an access-token-prefixed 403 usage limit no longer lands in context_overflow. NormalizeDaemonReason upgrades the same message from older daemons (context_overflow, provider_auth_or_access, unknown, agent_error) to provider_quota_limit. |
||
|
mrlonely
|
afedc6f76d | fix(cursor): preserve resumed sessions on connect timeouts (#8527) | ||
|
mrlonely
|
9d3613653e |
MUL-7164 fix(tasks): classify concurrent request rejections (#8200)
A provider rejection carrying the concurrent request limit witness is classified as provider_capacity_or_rate_limit ahead of the broad token/context and generic 403/auth rules, so members are no longer told to sign in again or reduce context for a transient concurrency rejection. NormalizeDaemonReason upgrades the same narrow witness from older daemons. Classification only: retryableReasons, retry delays and task creation are unchanged, and a regression pins provider_capacity_or_rate_limit as non-auto-retryable. The auto-retry proposal from #8180 is deliberately not included. Split from #8180 after review. |
||
|
|
a2d819ea40 |
MUL-6962 fix(daemon): give environment preparation failures their own failure reason (#7915)
* fix(daemon): give environment preparation failures their own failure reason A failed execenv.Prepare / Reuse is a filesystem error on the daemon's own host, but handleTask routed it through taskfailure.Classify — a classifier written to read agent and provider output — so a full disk, a denied permission or a directory held open by another process all landed in agent_error.*: the catchall today, provider_server_error on the report that opened the issue and sent hours of diagnosis at an LLM vendor. The daemon now tags those errors structurally (environmentSetupError) and reports the new platform-side reason environment_prepare_failed. The four existing sentinels still win: task identity, the prepare budget, skill downloads and the OpenClaw CLI deadline each name a specific cause inside the same phase. Installed daemons update on their own cadence, and they report a non-empty reason that FailTask deliberately leaves alone, so NormalizeDaemonReason upgrades any agent_error.* value whose error text opens with the prepare / reuse wrapper. That witness proves the run died before an agent existed, so every label in that namespace is wrong by construction. Not retryable and resume-safe: a full disk reproduces identically on the next attempt, and no session was touched. Closes #7913 Co-authored-by: multica-agent <[email protected]> * fix(mobile): parse failure_reason as an open string, and match copy to real scope Review follow-up on #7915. AgentTaskSchema pinned failure_reason to the six pre-MUL-1949 coarse values with `.catch("")`, so every refined reason was erased to undefined before any surface read it. run-row's badge map was widened in MUL-5370 and has been unreachable ever since: agent_error.*, skill_bundle_unavailable and the new environment_prepare_failed all rendered a bare "Failed". The field is an open string on the wire (packages/core/types/agent.ts) and in the chat message schema beside it; the task schema now matches, and the badge map moved to lib/run-failure-badge.ts so mobile's node-only vitest lane can cover the lookup a component test cannot reach. execenv.Prepare is not only filesystem work — it also writes and validates the per-provider local config (Codex home, Hermes overlay, Cursor MCP, and an OpenClaw config it fails closed on), so a malformed local config lands on this reason too. The Go docs, the chat copy in four locales and the docs table said "disk, permission, directory in use", which would send the reader past half the real causes — the same misdirection this reason exists to end. They now point at the raw error first and name the config case, with regression tests for a non-filesystem preparation failure on both the daemon and the legacy wire path. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
mikyan
|
fe59262188 |
feat(agent): add Huawei Cloud CodeArts support (#6985)
* feat(agent): add Huawei Cloud CodeArts support * fix(migrations): resolve CodeArts prefix collision * fix(agent): address CodeArts review feedback * fix(agent): tighten CodeArts integration scope * fix(agent): resolve CodeArts follow-up review blockers * fix(agent): redact CodeArts launch arguments * docs: sync supported runtime counts * fix(agent): own CodeArts model probe process tree |
||
|
LinYushen
|
3b1018e9c9 |
Revert "MUL-6016: feat(task): retry exact Codex capacity failures (#6747)" (#7586)
This reverts commit
|
||
|
|
8fda4f22d5 |
MUL-6016: feat(task): retry exact Codex capacity failures (#6747)
* docs: record Codex capacity retry design * docs: add Codex capacity retry implementation plan * fix(taskfailure): recognize exact Codex capacity error * feat(task): retry exact Codex capacity failures * feat(server): configure Codex capacity retries * docs: expose Codex capacity retry setting * fix(task): align capacity retry comments --------- Co-authored-by: cuijs <[email protected]> Co-authored-by: Claude <[email protected]> |
||
|
|
acecb7dd60 |
fix(pi): keep turn error when process exits 1 [MUL-6585] (#7451)
* fix(pi): keep turn error when process exits 1 Pi (and pi-print-clean-exit) exits 1 after stopReason=error. The backend then overwrote lastTurnError with "pi exited with error: exit status 1", so Classify filed a LiteLLM Connection error / Request timed out as non-retryable process_failure. Preserve the provider message and classify those strings as provider_network so auto-retry can resume the session. * fix(taskfailure): scope Pi network error matching Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: buihongduc132 <[email protected]> Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Bohan Jiang
|
4d5a679b9b |
fix(openclaw): unblock the OpenClaw runtime on slow hosts (MUL-6321) (#7142)
Fixes #7112. Task preparation ran `openclaw config file` and `config get agents.list --json` with a hardcoded 5s deadline; on a 2020 Intel MacBook Pro those take 8.4-10.9s and 4.3s every time, so the OpenClaw runtime was unusable there with no way to raise the limit. - Deadline 5s -> 30s, overridable via MULTICA_OPENCLAW_CLI_TIMEOUT / backends.openclaw.cli_timeout, clamped to [1s, 60s] so the worst case (4 serial calls) stays inside the 5-minute preparation budget. - Config discovery cached per daemon profile (on disk, since preparation runs in a one-shot helper process), invalidated by binary identity, OpenClaw path env, active config stat, and a 60s TTL. Only complete successes are cached; any cache fault degrades to a live CLI run. - The timeout is classified structurally into a new platform-side, non-retryable reason (runtime_cli_timeout) instead of being text-matched into agent_error.provider_network, which showed "check your network" copy and auto-retried a deterministic local failure. Copy added for web/desktop/mobile; a NormalizeDaemonReason shim upgrades reports from older daemons. |
||
|
|
22db0b0a2c |
MUL-5651: preserve tasks through runtime network partitions (#7129)
* fix(runtime): preserve tasks through network partitions Co-authored-by: multica-agent <[email protected]> * fix(runtime): bound reconnect retry waiting Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
f51ee62b53 |
MUL-6164 fix(agent): diagnose an unrunnable agent CLI instead of leaking "exec format error" (#6963)
* fix(agent): diagnose an unrunnable agent CLI instead of leaking "exec format error" (MUL-6164) npm-distributed agent CLIs ship a placeholder text stub at their bin entry point and rely on a postinstall script to overwrite it with the platform-native binary. npm 12 blocks install scripts unless allowlisted, pnpm 10 needs `pnpm approve-builds`, and --ignore-scripts / --omit=optional do the same — all of them leave the stub in place with its executable bit set. exec.LookPath accepts it, execve rejects it, so the CLI reads as installed everywhere the daemon looks and only fails at launch with a bare "start claude: fork/exec <path>: exec format error". Add ExplainExecError: on ENOEXEC it names the path, the owning npm package, and the exact repair command read out of that package's own scripts.postinstall. The executable path is recovered from the error itself, so this is one provider-agnostic call at each boundary — the shared version probe and the daemon's single backend.Execute site — rather than a wrap in every backend. claude and opencode ship this layout today; a CLI added later is covered without a code change. Also classify "exec format error" as runtime_missing_executable: a stub that cannot exec means this host has no usable CLI, and that reason already tells the daemon a fresh-session retry cannot help. Co-authored-by: multica-agent <[email protected]> * fix(agent): keep the ENOEXEC diagnosis to one copy (MUL-6164 review) The openclaw preflight explained the error, then returned it through openclawBackend.Execute into the daemon's launch boundary, which explained it again — %w keeps ENOEXEC reachable, so the second pass fired and the user got the same paragraph twice. Drop the openclaw wrap: its error already travels through the daemon boundary, so the extra site bought nothing. Make ExplainExecError idempotent as well, so the next nested boundary cannot reintroduce the same regression silently. Also quote the package root properly in the repair command — it is meant to be pasted into a shell, and a path holding a quote or a space has to survive that. The chat copy for runtime_missing_executable said the CLI "couldn't be found / check it's on PATH", which is the opposite of this failure: the file is right there on PATH and cannot run. Reworded to cover both across all four locales; the compact labels ("Runner CLI not installed") are left for the follow-up that reworks the reason's presentation. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
85045228a1 |
MUL-6102: fix(daemon): surface which HERMES_HOME a Hermes task actually read (#6892)
* fix(daemon): surface which HERMES_HOME a Hermes task actually read A Hermes task runs against a per-task HERMES_HOME overlay, seeded from a source home the daemon resolves from its own process environment. When that resolution lands on a home holding no config.yaml, the overlay is built anyway from nothing: the derived config keeps only skills.external_dirs, so the user's provider and credentials are gone and Hermes refuses every session/new with "No LLM provider configured". Nothing on that path said so. The overlay was seeded silently, no line logged the resolved source home, and the failure surfaced Hermes' own remedy -- run `hermes model` -- which edits whichever home the user's shell points at, not the one the task read. GH #6872 is what that costs: eight documented workarounds, none of which could have worked, and a root cause filed against an unrelated resume bug. Three additions, no behaviour change: - Warn when the source home is missing, or exists without a config.yaml. Kept a warning rather than a hard failure because config-less is a legitimate setup: an image that ships Hermes and injects OPENAI_API_KEY through the environment resolves a provider fine. - Log the resolved source home once per task, at Info. - Append the overlay and source homes to a "no LLM provider configured" failure, so the task itself names the directory that was read. The annotation runs after every classifier has read the error text. It embeds two user-chosen paths and Classify matches substrings anywhere, so annotating earlier would let a directory name pick the failure_reason -- a home under /srv/token-cache/limit/ satisfies rule 1 (context overflow), which is evaluated before rule 2 (missing config) and feeds retry eligibility and resume blacklisting. The phrase rule 2 already matched is now a shared const so the new predicate and the classifier cannot drift apart. Co-authored-by: multica-agent <[email protected]> * fix(daemon): keep the Hermes hint out of resume-guard input Review caught that the annotation did not stop at the daemon. It travels to the server as the task's error, is persisted in agent_task_queue.error, and is re-scanned there for the life of the row by service.ResumeUnsafeFailure and by the ILIKE/regex guards in GetLastTaskSession / GetLastChatTaskSession. Both interpolated paths are user-controlled -- HERMES_HOME comes from the agent's custom_env, the overlay root from MULTICA_WORKSPACES_ROOT -- so a legal directory name got a vote on session recovery. Measured, with a source home under /srv/400-invalid_request_error/: raw error ResumeUnsafeFailure = false annotated with the paths ResumeUnsafeFailure = true annotated with this hint ResumeUnsafeFailure = false That true is a healthy chat session pointer being cleared on a failure that has nothing to do with it, so "no resume semantics change" was not true as written. The hint is now fixed prose with nothing interpolated, and names the daemon log line that does carry the paths. This removes the vector rather than containing it: no wire or schema change, and it is correct against old servers too, so there is no mixed-version window where an un-upgraded server misreads a new daemon's text. Tests cover the boundary the previous ones missed: classification AND service.ResumeUnsafeFailure agree with the unannotated text across five real failure shapes, plus a check against the SQL guard phrases, which no unit test can execute. Also softens the missing-config warning, which asserted the task runs without credentials when the environment may well supply them -- the config-less-but-working setup this PR deliberately keeps supported. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Jamison-J
|
d582dca062 |
MUL-6025: fix(daemon): resolve Windows installer junctions (#6802)
Windows standalone Codex exposes its installer bin directory as an IO_REPARSE_TAG_MOUNT_POINT junction, which filepath.EvalSymlinks cannot traverse, so the daemon launched the shim path and Codex could not find its sandbox helpers. Discovery now keeps the stable entry point and the final target is resolved per launch via GetFinalPathNameByHandleW, so installer retargets take effect while old releases remain installed. Resolution failures fail closed on both the initial and rediscovery paths. Refs #6762 |
||
|
|
37e2b07094 |
MUL-5831 fix(agent): fail opencode runs that end on an empty step (#6545)
* fix(agent): fail opencode runs that end on an empty step (MUL-5831) `opencodeBackend.processEvents` required a positive terminal signal but only failed closed on two shapes: EOF while a step was still open, and a step that finished with reason "tool-calls" whose continuation never started. A final step that opens, emits only a reasoning part, and closes with reason "unknown" and an all-zero token block matches neither, so `opencode run` exited 0 and the daemon reported `completed` with an empty `agent_error` — a run with no deliverable that leaves the issue stalled until something re-wakes the agent. Zero input tokens means the provider round-trip never happened; that is a dead stream wearing a clean finish. Track whether each step produced anything at all — text, a tool call, or any non-zero token counter — and fail closed when the run ends on a step that produced none of them. The criterion is deliberately "this step produced nothing", not "the run produced no text", so a task whose only deliverable is a tool side effect stays green. The finish reason is not consulted: a missing or unrecognised reason must stay terminal for protocol compatibility, and voidness is orthogonal to it. The run now enters the existing failure path with a diagnosable error, so the task's retry rules apply instead of a silent false-green completion. Refs #6522 — this fixes the first of its three symptoms; see the PR body. Co-authored-by: multica-agent <[email protected]> * fix(agent): count every opencode usage field as round-trip evidence (MUL-5831) The void-step guard claimed "any non-zero token counter keeps the step productive", but only checked input, output and cache. OpenCode's protocol keeps reasoning and the aggregate total in fields of their own and reports cost as a sibling of the token block, and none of the three were parsed at all — so a real step that landed with reasoning or cost positive while input and output happened to be zero would be called void and fail a healthy run. Parse reasoning, total and cost, and move the check into stepReportedUsage so the criterion is stated once. Reasoning and total are read as evidence only: total is derived, and TokenUsage has no reasoning bucket, so folding either into the usage accumulator would change billing figures rather than fix a bug. Also renames the guard's error text to "no reported usage", which is what it now means. Co-authored-by: multica-agent <[email protected]> * fix(taskfailure): make opencode stream-ended failures retryable (MUL-5831) Turning a false-green completion red only helps if the failure then retries. It did not: none of the three errors the OpenCode terminal-signal guard raises was on the retry allowlist. The two "terminal signal" variants matched rule 13's "signal" substring by accident and landed in agent_error.process_failure; the new empty-step one fell through to agent_error.unknown. Either way the task died on the first attempt and max_attempts never applied. All three mean one thing — the provider stream died and `opencode run` still exited 0 — which is agent_error.provider_network by definition. Route them there on the shared "opencode stream ended" prefix. That bucket is retryable and resume-safe, so the retry child inherits the session and continues the truncated conversation instead of redoing the work already paid for. Covered end to end: the classifier cases, and a service-level test that walks each guard error through Classify into retryEligible — retrying with an attempt left, and still terminating at the ceiling so a deterministically broken provider cannot loop. Co-authored-by: multica-agent <[email protected]> * fix(taskfailure): upgrade legacy opencode stream-ended reasons (MUL-5831) Rule 7 only decides where these failures land when THIS server classifies them, and FailTask classifies only when the daemon sent no reason at all. An installed daemon predating that entry reports a non-empty agent_error.process_failure instead — its rule 13 matching the word "signal" in "terminal signal" — so the empty-reason branch skips it and the run stays off the retry allowlist. The fix reached only hosts that happened to update, which is the installed-daemon cadence problem NormalizeDaemonReason exists for. Upgrade process_failure / unknown / coarse agent_error to provider_network when the error opens with the guard's own prefix. Wider than the context-overflow rule above it on purpose: that one leaves refined reasons alone because a phrase somewhere inside an error blob says less than the bucket an earlier rule picked, whereas this witness is the guard's message from its first character, so the old bucket cannot be describing a better-identified cause. The retry test becomes a matrix over both wire shapes — current daemon (server classifies) and each legacy reason — since the previous version called Classify directly and never crossed the normalize boundary where the gap was. A negative case pins the prefix: an unrelated crash that merely mentions the phrase keeps the daemon's label. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Bohan Jiang
|
c1bd4f995c |
fix(agent): fail a run whose provider session ran out of context (MUL-5739) (#6422)
Detect provider context exhaustion from Claude Code's structured terminal_reason rather than from its error prose, so a saturated session is retired and the next task cold-starts instead of resuming a conversation that can no longer answer. Refs #6402 — does not close it: the captured 2.1.220/2.1.221 frames report is_error alongside terminal_reason, so the specific `completed` escape the issue reports is not reproduced. The structured check is the load-bearing fix and is proven present on the reported CLI version; the composite text predicate is a bounded backstop for uncaptured shapes. |
||
|
|
79c5832e1d |
MUL-5708 fix(taskfailure): classify response-side context-window overflow (#6366)
* fix(taskfailure): classify response-side context-window overflow (MUL-5708)
Claude Code 2.1.x reports an exhausted context window on the response,
not as a 400 on the request: the turn ends with stop_reason
"model_context_window_exceeded" and the CLI prints
API Error: The model has reached its context window limit.
That string carries none of the phrases rule 1 matched and no "token",
so it classified as agent_error.unknown. Unknown is absent from the
resume blacklists (resumeUnsafeFailureReason, GetLastTaskSession,
GetLastChatTaskSession), so the over-full session stayed pinned as the
resume pointer for the (agent, issue) pair and every later comment on
the issue resumed the same transcript and overflowed again.
Match the CLI copy and the raw stop reason so the failure lands in
agent_error.context_overflow, which those blacklists already exclude —
the next comment then starts from a fresh session instead of replaying
the overflow.
Co-authored-by: multica-agent <[email protected]>
* fix(taskfailure): upgrade an old daemon's catchall on context overflow (MUL-5708)
Installed daemons update on their own cadence, and FailTask only
re-classifies when the caller supplied no reason. A daemon whose rule 1
predates the response-side wordings reports agent_error.unknown, which
is on no resume blacklist — so until every host updates, the over-full
session stays pinned as the (agent, issue) resume pointer and every
later comment replays the same overflow. One un-upgraded host means a
permanently stuck issue, not just a missing label.
Recognise the two witnesses server-side in NormalizeDaemonReason, next
to the MUL-5370 skill-bundle rule it mirrors, so the retirement lands
the moment the server deploys. The accepted legacy set is narrower than
that rule's: only the catchall and the pre-MUL-1949 coarse agent_error.
A refined reason means the old daemon matched an earlier rule on the
same text, which says more about what ended the run than a witness
appearing somewhere in the blob does.
The witnesses move into one shared var so Classify and the normalizer
cannot drift apart.
Co-authored-by: multica-agent <[email protected]>
---------
Co-authored-by: Bohan-J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
|
||
|
|
4fe94a6d40 |
revert: "MUL-5637 fix(mcp): agent mcp_config is an authoritative allowlist (#6292)" (#6314)
This reverts commit
|
||
|
|
aa349fed02 |
MUL-5637 fix(mcp): agent mcp_config is an authoritative allowlist (#6292)
* fix(mcp): treat agent mcp_config as an authoritative allowlist
An agent's saved mcp_config was silently widened with the runtime host's
own user-level MCP servers, so an explicitly empty `{"mcpServers":{}}`
resolved to the COMPLETE host set instead of no servers at all — the
opposite of what the operator configured (GitHub #6283).
`--strict-mcp-config` was being passed correctly; the merge happened
before it, in the daemon, so strict mode constrained an already-widened
set. Introduced by #5277 and present in v0.4.16 through main.
Restore the three-state contract in resolveEffectiveMcpConfig:
null / unset -> inherit the provider's native MCP configuration
{"mcpServers":{}} -> strict empty, no host servers
non-empty object -> strict allowlist, exactly those servers
Two explicit inherit paths keep the additive behaviour reachable without
weakening the default:
- runtime_config.mcp.inherit_runtime = true opts an agent back in.
- The claim response now carries mcp_config_overlay_only so the daemon
can tell an agent-authored config from a per-task Composio overlay.
Without it, enabling an integration on an agent that never configured
MCP would have stripped the host servers it was already inheriting.
Both decode paths fail closed: malformed runtime_config never enables
inheritance, and a failed runtime merge falls back to the agent's own
config.
The web MCP tab and the `agent create/update --mcp-config` help text
described the old additive behaviour, which is how a tightened config
could look correct while exposing every host server; both now state
which mode is in effect.
Note for rollout: the fix lives in the daemon, so self-hosted users must
upgrade the daemon — a server/UI upgrade alone does not apply it.
Co-authored-by: multica-agent <[email protected]>
* fix(mcp): close review gaps in the authoritative mcp_config change
Addresses the four must-fix findings from review of #6292.
1. Deleting the last managed server no longer widens access.
removeManagedMcpServer cleared the config to null, which now means
"inherit the host's MCP servers" — so a delete took the agent from one
allowed server to every server on the host. It now leaves an explicit
`{"mcpServers":{}}`. Restoring inheritance moved to a separate
clearManagedMcpConfig action behind its own confirmation that states the
widening. The delete dialog no longer claims "Runtime servers are not
affected", which was the opposite of the truth.
2. The UI no longer promises a boundary an old daemon does not enforce.
The strict semantics live in the daemon, so a config saved against an
older daemon is not yet in effect. Adds the authoritative-mcp-v1 daemon
capability:
- The daemon advertises it and reports authoritative_mcp on the
runtime-capabilities response.
- The claim path fails closed: a managed, non-inheriting mcp_config
claimed by a daemon without the capability cancels the task and
returns 412 with an actionable message, instead of letting that daemon
merge the host's servers in. runtime_config.mcp.inherit_runtime is the
documented escape hatch, and it is honest — it declares that the
operator accepts the host's servers.
- The MCP tab shows "needs upgrade" rather than "Not exposed" while the
bound runtime lacks the capability.
3. Saving OpenClaw settings no longer drops the inherit opt-in.
parseOpenclawRuntimeConfig discarded unknown keys and the tab persisted
the result as the whole runtime_config, so one unrelated routing save
silently deleted mcp.inherit_runtime. Unknown keys now round-trip
through OpenclawRuntimeConfig.passthrough, excluded from the dirty check
so they cannot make the form look edited.
4. Documents the new semantics in the built-in creating-agents skill and
its source map: the three states, the persisted
runtime_config.mcp.inherit_runtime field, and the claim-time capability
gate.
Also corrects the PR's rollout claim: there is no database migration, but
this does add a persisted JSON field and change the meaning of an existing
one.
Co-authored-by: multica-agent <[email protected]>
* fix(mcp): stop the authoritative-daemon gate from blocking valid claims
CI's backend job failed three handler claim tests with the new 412. Two
distinct problems, both real:
1. The gate fired for a non-object mcp_config. 66 handler fixtures seed
`[]`, which is not a valid MCP config and cannot carry `mcpServers`, so
it expresses no boundary to protect. An old daemon does not widen it
either: mergeRuntimeAndAgentMcpConfig fails to unmarshal a non-object
and falls back to the agent config alone (verified directly). Gating
these blocked tasks with no security benefit, so the gate now requires a
JSON object.
2. The shared daemon test-request helper advertised no capabilities, so
every claim test was accidentally simulating a pre-#6283 daemon. It now
defaults authoritative-mcp-v1 on, matching what every current daemon
sends. Only that capability — skill-bundles / coalesced-comments / rpc
are feature negotiations whose absence tests real legacy behaviour, so
they stay opt-in per test.
Adds claim-level coverage for the gate itself, which is what the unit tests
alone could not catch: an outdated daemon gets 412 with an actionable
message and the task is cancelled; a capability-advertising daemon gets
200; the inherit_runtime opt-in lets an outdated daemon through; and an
unmanaged or non-object config is never gated.
Verified against a real migrated schema this time (throwaway Postgres),
which is how the three failures were reproduced locally and confirmed
fixed: `go test ./internal/handler ./internal/daemon` both ok.
Co-authored-by: multica-agent <[email protected]>
* fix(mcp): surface the daemon-upgrade refusal and stop gating safe providers
Addresses the second review round on #6292.
1. The refusal is now visible wherever the operator looks. The default
claim path is the machine-level BATCH endpoint, which skips build
failures and still answers 200 {"tasks":[]}, so the previous bare
CancelTask showed a task that vanished with no stated reason — turning an
explicit upgrade requirement into an unexplained failure. The claim path
now fails the task with a new classified reason,
mcp_config_daemon_outdated, plus the actionable message. That reaches the
user on all three claim paths and on any daemon version, which a new
response field could not: the audience is by definition a daemon too old
to read one. The per-runtime path keeps its 412.
The reason is deliberately not auto-retryable — the same outdated daemon
would claim the retry and fail it again.
2. The gate no longer cancels safe tasks. It applied to every provider, but
only claude / codebuddy / codex / cursor / opencode / openclaw were ever
merged with host MCP by an old daemon (loadRuntimeMcpServerConfigs).
Qwen was never merged and already had strict semantics, so its tasks were
being failed for a risk that does not exist. Scoped via
providersOldDaemonsMergedRuntimeMcp; an unknown provider does not gate,
because the gate should only fire where the old behaviour is concrete.
3. The new authoritative_mcp flag now goes through the API schema layer.
Both local-skills responses were returning raw network JSON, so the flag
that decides whether the UI may assert an MCP boundary rested on an
unchecked type assertion. Adds RuntimeLocalSkillListRequestSchema with
authoritative_mcp and mcp_supported defaulting to FALSE — the fail-closed
direction — and a MALFORMED_ fallback that cannot express a guarantee.
Claim-level tests now cover all three paths, which is what the previous
helper-only tests missed: per-runtime 412, batch recording the refusal on
the task while still delivering the healthy tasks in the same batch, WS RPC
refusing and accepting, the qwen negative case, the inherit_runtime escape
hatch, and unmanaged / non-object configs.
Verified against a real migrated schema (throwaway Postgres):
go test ./internal/handler ./internal/daemon ./pkg/agent ./pkg/taskfailure
./internal/service all ok.
Co-authored-by: multica-agent <[email protected]>
* fix(mcp): register the new failure reason and wire its copy into the UI
Addresses the third review round on #6292.
1. mcp_config_daemon_outdated was declared but never registered in
taskfailure.allReasons, so metrics.NormalizeFailureReason missed the
known-value map and fell through to free-text Classify() — relabelling a
platform-side refusal as `agent_error.unknown` (verified directly) and
leaving the Prometheus series un-pre-warmed. Registered it, canonical
count 22 → 23 (platform 8 → 9), with the wire value and IsAgentError split
pinned. New test pins the WHOLE canonical set through
NormalizeFailureReason so forgetting the next reason fails a test instead
of quietly mislabelling a metric; NormalizeFailureReason had no coverage
at all before.
2. The upgrade copy was dead. The locale strings landed last round but
neither consumer mapped the reason: chatFailureCopy fell back to generic
failure text with the actionable detail buried in the collapsed raw
error, and task-failure.ts rendered the bare wire value
`mcp_config_daemon_outdated` in the agent activity list and issue
execution log. Both are mapped now, with regression tests, plus the
runtime class pinned in failure-class.test.ts. This directly contradicted
the claim in the claim-path comment that every path reaches the user, so
that is now actually true.
3. providersOldDaemonsMergedRuntimeMcp is documented as what it is: a FROZEN
record of what pre-capability daemons merged, not a mirror of the daemon's
current provider switch. The old "keep the two lists in lockstep" note was
actively harmful advice — runtime MCP discovery for a new provider can only
ship in a daemon that already advertises the capability (never gated), so
adding it here would fail tasks on old daemons that never merged for it,
re-creating the qwen false-positive. Pinned with a test.
Also corrects a stale count in task-failure.ts (7 → 9 platform reasons).
Verified against a real migrated schema (throwaway Postgres): full backend
suite green apart from the pre-existing environmental cmd/multica guard; all
9 TestMcpGate_* integration tests pass.
Co-authored-by: multica-agent <[email protected]>
* docs(taskfailure): correct taxonomy counts and finish the reason registration
Non-blocking nits from the fourth review round on #6292.
- Taxonomy counts now say 23 reasons / 9 platform-side. Registering
mcp_config_daemon_outdated last round updated the assertions but not the
prose. Swept the whole repo rather than only the flagged lines, which
turned up four more that were already stale at 21 and drifted further:
handler/dashboard.go, daemon/poisoned.go, core/types/agent.ts, and the
db/queries/task_usage.sql comment sqlc copies into the generated file.
The generated file's comment was updated by hand to match its source.
Running `sqlc generate` churned 58 lines across 47 unrelated files — the
local sqlc version differs from the one that produced the checked-in
output — so that churn was reverted and only the one intended line kept.
- failure_test.go's `required` list now includes
ReasonMcpConfigDaemonOutdated. Length and label assertions already covered
the reason, but the list is documented as the complete canonical set, so
the omission contradicted its own comment.
- Restored the line break in chat-message-list.test.tsx that a previous edit
of mine collapsed.
Comment, test-fixture and formatting only; no behaviour change.
Co-authored-by: multica-agent <[email protected]>
---------
Co-authored-by: Eve <[email protected]>
Co-authored-by: multica-agent <[email protected]>
|
||
|
|
c271f80999 |
MUL-5370 fix: label stalled skill-bundle downloads, align failure-reason copy with the backend taxonomy (#6001)
* fix(daemon): label stalled skill-bundle downloads and make them retryable A skill bundle that could not be downloaded during task preparation surfaced as the bare string "resolve skill bundles: context deadline exceeded". taskfailure.Classify has no rule for a Go context deadline, so it landed in agent_error.unknown — a bucket that is NOT on the server's retry allowlist. A transient stall therefore became a terminal chat failure carrying a label nobody could act on, and the failure was invisible on the Usage page's Errors breakdown. (MUL-5370) - Add the platform-side reason skill_bundle_unavailable and put it on retryableReasons. Retrying is cheap and safe: the agent process never started, and bundles that did arrive are already cached on disk, so successive attempts converge. - Carry a sentinel error from the resolve loop so the reason is derived structurally rather than by matching the wrapped transport error's text, and name the skill, its declared size and the elapsed wait in the wrap — enough to tell "this bundle is too big for the link" from "the link is dead" without reading daemon logs. - Normalise the wire shape an OLD daemon produces (a non-empty catchall plus the previous "resolve skill bundles:" wrapper) on the server side. Installed daemons upgrade on their own cadence, and FailTask only classifies when the caller supplied nothing, so without this the fix would reach only hosts that happened to update — while the un-upgraded hosts most likely to be hitting the bug kept failing terminally. - Teach Classify about "deadline exceeded" and net/http's "Client.Timeout exceeded while awaiting" so any other Go-side deadline that reaches it as text stops falling into the unknown bucket too. - Backfill historical rows in both agent_task_queue and chat_message. Scoped to agent_error.unknown alone — the old wrapper string postdates the in-flight classifier by three weeks, so no row carrying it can hold the legacy coarse value — which keeps the down migration an exact inverse. Co-authored-by: multica-agent <[email protected]> * fix(chat): give chat its own failure copy for the refined reasons #5991 rebuilt the operator-facing failure labels around an open wire string with a raw-value fallback, but the chat bubble kept its own exact-key lookup against the six coarse values from migration 055. So all 14 agent_error.* values still missed and rendered the generic "Something went wrong and the agent couldn't finish replying" — the classification the backend had already computed was discarded at the last step, and that is the message the MUL-5370 reporter saw. - Add resolveFailureReasonKey in packages/core: exact match, else degrade an `agent_error.*` value to its family, else undefined. A reason newer than the shipped client now lands on the family line instead of the fallback. - Rekey the chat copy map by wire value and route it through the helper. Chat deliberately degrades to friendly copy rather than adopting the operator surfaces' raw-value fallback: it is read by the person who just sent a message, and the raw error is one click away under the collapsible. - Add refined chat copy (en / zh-Hans / ja / ko) only where it can say something the family line can't — a different next step: network, auth, quota, rate limit, context overflow, missing/outdated CLI, skill download. - Give skill_bundle_unavailable a label on the web and mobile surfaces and a class on the Usage page's Errors breakdown (runtime — the operator response is "check the daemon's link to Multica", the provider is not involved). - Mobile's two label maps were still coarse-only for the same reason; rekey them by wire value and fill in the refined taxonomy. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
6dba74c3c1 |
fix(task): auto-retry transient "Connection closed mid-response" like chat (MUL-4910) (#5565)
* fix(task): auto-retry transient provider stream cut like chat (MUL-4910) Claude Code's "API Error: Connection closed mid-response" is a transient network cut. In unattended issue runs it fell through to agent_error.unknown / process_failure — neither in retryableReasons — so the task terminated. Interactive chat only appeared resilient because the CLI's own in-process retry usually recovers first; there is no chat-specific retry in Multica. Both paths share the same finalizeStreamResult -> Classify -> retryableReasons pipeline. - classify: route "connection closed" / "mid-response" to agent_error.provider_network, before the exit-status rule so the "exited with error: exit status N" variant also lands here. - retry: add provider_network to retryableReasons. It is resume-safe, so the retry child inherits the session and continues the truncated conversation instead of restarting. Tests cover the new classification (incl. exit-status variant) and the retry/resume flags. Note: mirror the substrings into the MUL-1949 offline backfill SQL to keep in-flight and historical taxonomies aligned. Co-authored-by: multica-agent <[email protected]> * feat(task): defer provider_network's final retry ~5s — three-tier (MUL-4910) Follow-up to the immediate-retry fix: make the connection-closed retry a three-tier schedule — first run + immediate retry + one retry deferred ~5s — so a blip that survives the immediate retry gets a short cooldown before the final attempt instead of firing back-to-back. Reuses the existing deferred/fire_at primitive (the comment-routing escalation mechanism) rather than adding new infrastructure: CreateRetryTask gains an optional fire_at; when set, the child is inserted 'deferred' and the existing PromoteDueDeferredTasksForRuntime sweeper — already run promote-first on every claim poll — flips it to 'queued' at fire_at. No migration, no claim-query change, no daemon change. - retryAttemptCeiling: raise provider_network's ceiling to 3 (other reasons keep max_attempts=2); applied in both retryEligible and MaybeRetryFailedTask's budget pre-check so the primary and sweeper paths agree. - retryDelayForAttempt: only provider_network's final attempt is deferred (5s); every other retry — including provider_network's first — stays immediate. - FailTask + MaybeRetryFailedTask pass fire_at and skip the queued broadcast/notify for a deferred child (promotion emits them at fire time). Timing: the deferred child fires on the first claim poll at/after fire_at, so >= 5s; on an otherwise-idle runtime it can stretch to the poll interval — the same behaviour deferred escalations already have. Tests: pure schedule/eligibility coverage (TestProviderNetworkRetrySchedule) plus a DB test asserting fire_at controls deferred-vs-queued, attempt=3, and resume-safety. Co-authored-by: multica-agent <[email protected]> * fix(task): persist reason-aware retry budget; respect max_attempts=1 disable (MUL-4910) Addresses the pre-merge review must-fix: retryAttemptCeiling raised provider_network to 3 unconditionally, which (1) overrode the max_attempts=1 "auto-retry disabled" contract and (2) persisted a self-contradictory child (attempt=3, max_attempts=2) that leaks to the task API, so a naive attempt < max_attempts consumer would misjudge the budget. - retryAttemptCeiling now returns taskMaxAttempts unchanged when it is <= 1 (disabled stays disabled) and only ever WIDENS otherwise (max(col, 3) for provider_network) — a higher configured budget is kept. - CreateRetryTask takes an optional max_attempts; FailTask and MaybeRetryFailedTask write the reason-aware effective ceiling into the child so the whole retry chain self-describes (attempt=3, max_attempts=3). NULL inherits the parent column, so non-provider_network reasons are unchanged. Tests: - pure: ceiling widens to 3, keeps a higher budget, and never revives a disabled (max_attempts=1) task; eligibility rejects the disabled case. - DB (end-to-end FailTask): default budget → deferred final child at attempt=3/max_attempts=3; first failure → immediate child; max_attempts=1 → no child. Plus CreateRetryTask persists the passed budget / inherits on NULL. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Rusty Raven
|
7356b56a10 |
fix(taskfailure): anchor HTTP status-code matches to digit boundaries (#5275)
Anchor the 401/402/403/429/529 HTTP status-code matches to digit boundaries so embedded numbers (e.g. "402913 tokens", "15290ms") no longer misclassify process/unknown failures as provider errors and skew failure observability. Mirrors the existing 5xx anchoring (providerHTTP5xxRe). Fixes #5271 MUL-4422 |
||
|
|
10afd1af1b |
feat(server): introduce pkg/taskfailure classifier and switch in-flight failure_reason writes (MUL-2946) (#3693)
Lift MUL-1949's offline backfill failure_reason taxonomy into a shared in-flight classifier so the agent_task_queue.failure_reason column is written with refined values (provider_auth_or_access, context_overflow, provider_capacity_or_rate_limit, …) at write time rather than waiting on SQL backfill to re-classify after the fact. PR1 of the Grafana board plan in MUL-2328 — the upcoming PR2 reuses pkg/taskfailure.AllReasons() to pre-warm the Prometheus failure_reason label set. * server/pkg/taskfailure: new package with the canonical 21 Reason constants (7 platform-side + 14 agent_error.* sub-reasons), AllReasons() returning a defensive copy, IsAgentError() prefix check, and Classify(rawError) Reason mirroring the SQL CASE rules from MUL-1949 (db-boy's analysis). 100% statement coverage. * server/internal/daemon/daemon.go: route the 'agent_error' coarse fallback paths (StartTask error, runTask early-return error, CompleteTask permanent rejection, reportTaskResult default branch) and the executeAndDrain default error case (chained after classifyPoisonedError) through taskfailure.Classify so blocked / timeout / unknown-status results all carry a refined reason on the wire. * server/internal/service/task.go: FailTask classifies errMsg when the daemon-supplied failureReason is empty, eliminating the legacy COALESCE(.., 'agent_error') landing. * server/internal/daemon/poisoned.go: alias FailureReasonIterationLimit and FailureReasonAPIInvalidRequest to the canonical taskfailure constants. agent_fallback_message and codex_semantic_inactivity are pre-existing operational reasons not in the canonical 21 — kept as literals for now and revisited in a follow-up PR. Backfill SQL from MUL-1949 stays as the authoritative offline source of truth; this PR keeps the in-flight classifier in lock-step with the SQL CASE expression so historical and future rows share the same taxonomy. No behavior change for the platform-side reasons (queued_expired, runtime_offline, runtime_recovery, timeout, etc.) which already align with the canonical set. Co-authored-by: Eve <[email protected]> Co-authored-by: multica-agent <[email protected]> |