mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-09 01:46:46 +08:00
zai-beta
41
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 m2keloopandm2keloop
|
d908e4882d |
MUL-6710: fix(runtimes): revoke access when making runtime private (#7580)
* fix(runtimes): revoke access when making runtime private * fix(runtimes): regenerate queries for workflow step --------- Co-authored-by: m2keloop <[email protected]> |
||
|
|
2aa20b0353 |
ZIC-245: fix custom Oh-My-Pi runtime profile identity and discovery (#8576)
* fix(runtimes): preserve custom profile base runtime identity Co-authored-by: multica-agent <[email protected]> * fix(runtimes): pass both url params to the profile identity test TestUpdateRuntimeProfile_RejectsIdentityChange nested two withURLParam calls. That helper installs a fresh chi route context each time and does not copy what is already there, so the outer call for profileId dropped id: the handler answered 400 "workspace_id is required" and never reached the immutability check the test exists to cover. Use withURLParams, which carries existing params forward and is what the rest of this file already uses. Co-authored-by: multica-agent <[email protected]> * fix(runtimes): name a runtime target the same way on every surface The base-runtime picker spelled Oh-My-Pi with an inline ternary, so it was the only place that used the product name. The catalog row, both detail headers, the base-runtime detail row and the locked chip in the form kept rendering the stored id under `capitalize` — a user picked "Oh-My-Pi" and the next screen, and the saved profile, said "Omp". Move the name into runtimeTypeLabel next to the target whitelist and read it from every surface. profileRuntimeType replaces the repeated runtime_type ?? protocol_family fallback. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> Co-authored-by: Bohan-J <[email protected]> |
||
|
|
5132ac99eb |
MUL-7235: show cancellation actor in run history (#8249)
* MUL-7235: show cancellation actor in run history
Co-authored-by: multica-agent <[email protected]>
* fix(issues): simplify failed run recovery UI
Co-authored-by: multica-agent <[email protected]>
* Revert "fix(issues): simplify failed run recovery UI"
This reverts commit
|
||
|
|
2aad9842a9 |
MUL-6891, MUL-7048: fix(agents): preserve presence for hidden private runtimes (#7830)
* fix(agents): preserve presence for hidden private runtimes Co-authored-by: multica-agent <[email protected]> * test: trim hidden-runtime regression fixtures Co-authored-by: multica-agent <[email protected]> * fix(agents): use coarse hidden runtime availability Co-authored-by: multica-agent <[email protected]> * fix(agents): refresh unstable availability Co-authored-by: multica-agent <[email protected]> * fix(agents): omit visible runtime availability Co-authored-by: multica-agent <[email protected]> * fix(agents): poll projected availability on all clients Co-authored-by: multica-agent <[email protected]> * fix(runtimes): stop availability polling once every projection is offline Polling now runs at 30s only while at least one agent's projected runtime_availability is online or unstable. A stable offline projection stops polling; recovery is already covered by daemon:register and reconnect invalidating the agents query. Co-authored-by: multica-agent <[email protected]> * fix(agents): publish heartbeat recovery refreshes * fix(agents): keep projected availability polling Co-authored-by: multica-agent <[email protected]> * fix(agents): bound recovery and projection polling Co-authored-by: multica-agent <[email protected]> * fix(agents): stop polling stable offline projections Co-authored-by: multica-agent <[email protected]> * fix(agents): align projected presence signals Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> Co-authored-by: worker-opencode <[email protected]> Co-authored-by: Sol-Boy <[email protected]> |
||
|
|
571e61128d |
MUL-7002: remove database reads from WebSocket heartbeats (#7976)
* fix(runtime): remove websocket heartbeat lookups Co-authored-by: multica-agent <[email protected]> * fix(runtime): preserve explicit deregistration Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
731af7ccd6 |
MUL-7002: push runtime deletion invalidation to active daemons (#7969)
* fix(runtime): invalidate deleted active runtimes Co-authored-by: multica-agent <[email protected]> * fix(runtime): separate invalidation delivery metrics Co-authored-by: multica-agent <[email protected]> * fix(runtime): complete invalidation metric separation Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
7c1d826a98 |
chore(metrics): remove database-sampled metrics (#7819)
Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
08a1e6e934 |
MUL-6742: fix archived-agent runtime garbage collection (#7641)
* fix(runtime): garbage collect archived-agent runtimes Co-authored-by: multica-agent <[email protected]> * fix(runtime): address GC review feedback Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
8442504201 |
fix(runtimes): hide private runtimes from non-owners (#7435)
* fix(runtimes): enforce private runtime access * fix(runtimes): gate agent model discovery * refactor(runtimes): reuse isRuntimeUsableForUser on runtime detail The detail page re-derived the owner-or-public rule inline instead of calling the shared helper every other surface in this PR uses. The two disagreed on an ownerless runtime: the helper is fail-closed (matching the server's read gate, which reuses canUseRuntimeForAgent), while the inline copy rendered Usage for a public one and could only 404. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: m2keloop <[email protected]> Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
22db0b0a2c |
MUL-5651: preserve tasks through runtime network partitions (#7129)
* fix(runtime): preserve tasks through network partitions Co-authored-by: multica-agent <[email protected]> * fix(runtime): bound reconnect retry waiting Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
b30a3ae55c |
MUL-6164 feat(runtime): take an unusable agent CLI offline and refuse its triggers with the fix (#6966)
* fix(daemon): take a runtime offline when its CLI cannot be executed (MUL-6164) A CLI that cannot exec was classified as a transient probe failure, so the runtime stayed online and every task it claimed died at fork/exec. The user in MUL-6164 hit the same error once per task with no signal that the runtime was the problem — an npm placeholder stub passes exec.LookPath, so nothing else the daemon checks can tell it apart from a working install. ENOEXEC (and its Windows spelling, ERROR_BAD_EXE_FORMAT) is a fact about the bytes on disk, not a hiccup: it will reproduce until someone reinstalls. Give it its own probe verdict and route it down the track below-minimum already uses — claim barrier, seq-stamped hold, deregistration — so the provider goes offline with its reason instead of collecting failed tasks. Recovery needs no new machinery: a demoted provider has no runtime, which puts it back in providersMissingRuntimes, and the first healthy probe clears the hold and lets converge register it again, with no restart. One thing the below-minimum track did not need: a confirmation window. The repair we tell users to run overwrites the bin entry in place, so a probe landing mid-copy sees a truncated file. Demotion therefore requires two sightings a window apart; a genuinely broken install pays one extra probe round, and a spurious verdict would in any case recover on its own. detectBuiltinRuntimes' second return value is now "providers condemned this round, with the evidence against them" rather than "below-minimum versions" — same shape, both kinds, and the value is the reason /health already shows. Co-authored-by: multica-agent <[email protected]> * feat(runtime): refuse triggers for an unusable agent CLI instead of queueing them silently (MUL-6164) Taking the runtime offline stopped the per-task failures, but it replaced them with silence: the next @mention or assignment queued as if the machine were merely asleep, and nothing ran for two hours until queued_expired. Waiting only makes sense when waiting is a plan — a sleeping laptop comes back on its own; a CLI that cannot be executed does not. dispatch/reason.go already draws exactly that line between runtime_offline ("bring the machine back, queued work waits") and agent_runtime_required ("nothing will ever claim this"), and warns that collapsing the two sends users to fix the wrong thing. This adds the third member of that family, runtime_unusable, and routes the new verdict to it end to end: - The daemon reports WHY on deregistration — a stable code plus the repair command read from the package's own manifest, since only that side knows the path. Stored on agent_runtime.metadata, which registration overwrites wholesale, so a repaired runtime drops the explanation structurally. - AgentReadiness now returns a verdict (available / waitable / blocked) instead of a bool plus prose, and the direct-agent trigger paths start consulting it — blocking ONLY on the blocked verdict, so an offline machine still queues. agentReadinessReasonCode, which re-derived the same code from agent state, is gone: one source, no drift. - Every path that refuses without a human watching the response leaves the reason on the issue: an assignment, or another agent's @mention. The interactive surfaces already carry the code back to the composer, which now localizes it in all four languages. Below-minimum keeps today's behaviour deliberately. It is equally futile to wait on, but changing when THAT refuses a trigger is a separate product decision. Co-authored-by: multica-agent <[email protected]> * fix(runtime): keep the unusable verdict through a late register, and repair Windows (MUL-6164 review) Two ways the refusal path failed to reach the user. The verdict could be erased by an older answer. A register whose request was still in flight across the demotion upserts the runtime row afterwards, and that upsert overwrites metadata wholesale — including the reason just stored. The local hold refuses the response correctly, but the cleanup that took the revived row offline again sent no reason, so the server ended up "offline, cause unknown". Every admission path reads that as "the machine will come back" and queues, which is exactly the behaviour this change exists to remove. The demotion record now keeps the structured cause and the revived-row cleanup re-attaches it per row. The regression test drives the real interleave — a register held inside the server handler across the verdict — and fails without the fix. The repair command was POSIX-only. Windows had just gained detection (ERROR_BAD_EXE_FORMAT), so a Windows user was correctly refused and then handed `cd 'C:\...' && node install.cjs`: cmd.exe treats the quotes literally and PowerShell 5.1 has no `&&`. The command is now rendered for the shell of the machine that reported it — Set-Location on two lines for PowerShell, with PowerShell's own quote escaping — and carries that shell so the notice fences it truthfully instead of labelling everything bash. Also corrected the notice's own docs: it is written for every author, not only unattended ones. The chip and toast carry the reason code but not the repair command, so a human needs the durable copy too. The behaviour was already this; the comment claimed otherwise. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
96bf122f2c |
MUL-6107: preserve task history during runtime GC (#6894)
* MUL-6107: preserve task history during runtime GC Co-authored-by: multica-agent <[email protected]> * fix(server): address runtime GC review Co-authored-by: multica-agent <[email protected]> * fix(server): finish runtime GC review nits Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
c4f0d8cc8c |
MUL-5999: make workspace teardown index-driven, bounded and fenced (#6754)
* perf(workspace): make teardown task discovery index-driven
Workspace teardown found a workspace's tasks with a three-way
`OR agent_id IN (…) OR issue_id IN (…) OR runtime_id IN (…)`, three times
per delete, and swept task_token with the same shape. PostgreSQL costs
that as a Seq Scan on agent_task_queue — the largest table in the
database (5.08M rows / 8.3 GB in production) — so deleting even a tiny
workspace paid for the whole table, inside the transaction that holds the
workspace row lock and global advisory lock 4246.
Resolve the task set one owner key at a time instead, in the handler, and
reuse that snapshot for every task-keyed step. Measured on a 1.2M-row
synthetic copy of the production shape (937 agents / 109 issues / 12
runtimes): the OR form and a 937-element `= ANY(array)` form both Seq
Scan; single-key equality is a Bitmap Index Scan. A UNION of parent-key
joins is worse than the OR (three hash joins, each seq scanning the
table), which is why that earlier suggestion was dropped.
All three ownership paths are preserved: nothing enforces that a task's
agent, runtime and issue share a workspace, so dropping a path would turn
a performance fix into a tenant-cleanup bug.
task_token needs no loop. Its workspace_id has been NOT NULL since
migration 108, so the workspace key alone covers this workspace's rows,
and the two extra OR arms only ever matched tokens owned by another
workspace whose task or agent lives here — both NOT NULL FKs with
ON DELETE CASCADE, so the later task/agent deletes remove exactly those.
Two indexes make the new shapes index-eligible:
- 273 agent_task_queue(runtime_id): every existing runtime_id index is
partial, so all-status consumers had none. This also covers the FK's
own ON DELETE CASCADE probe, which runs per deleted agent_runtime row
on the ordinary delete path.
- 274 task_token(workspace_id): the column has never been indexed.
Measured block usage for the same workspace, before → after:
agent_task_queue discovery 25,665 → 14,485 blocks; task_token sweep
4,086 → 3 blocks. The point is not the ratio at this size but that the
cost now scales with the workspace being deleted instead of with the
global table.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): fence, bound and un-cascade the teardown task sweep
Addresses all four blocking findings from the MUL-5999 review.
1. Concurrency (correctness / tenant isolation). The task set is no longer
a lock-free snapshot reused across steps. Teardown now locks every row
through which a task can belong to the workspace — its agents, issues
and runtimes — FOR UPDATE, and holds them for the rest of the
transaction. Inserting an agent_task_queue row takes FOR KEY SHARE on
each referenced owner, which conflicts with FOR UPDATE, so a concurrent
enqueue (or a reassignment onto one of these runtimes) can no longer
slip a task in behind the sweep. Locking the tasks alone could not do
that: a row that does not exist yet cannot be locked. This is the same
technique the handler already uses on the workspace row to keep
CreateChatSession out of the delete window.
The per-batch page query also takes FOR UPDATE, so under READ COMMITTED
PostgreSQL re-checks ownership after each lock: a task a concurrent
commit already moved elsewhere is skipped rather than deleted on a
stale claim, and a task we locked cannot move before we delete it.
Deadlock (40P01) joins lock_timeout (55P03) as a retryable 503, because
the owner locks can be taken in the opposite order by a concurrent task
update.
2. Cleanup ownership (no cascade reliance). task_token keeps all three
explicit paths, split into separate index-driven statements instead of
one OR: workspace_id in DeleteWorkspaceLeafData, task_id in
DeleteTaskBatch, agent_id in DeleteTaskTokensByAgent. Nothing in the
teardown graph now depends on ON DELETE CASCADE to finish the job.
3. Bounded memory. The whole-workspace task array is gone. The sweep pages
through one owner key at a time with LIMIT 1000 and deletes exactly the
page it read, so process memory, the array parameter and each DELETE's
row count are bounded by the batch size rather than by a workspace's
lifetime task count. The task-keyed arms moved out of the leaf-data
statement into DeleteTaskBatch, which runs per page, and the separate
whole-workspace task delete is gone.
4. Migration retryability. The concurrent-index cleanup hooks are now
derived from a version -> index-name map that covers 257, 261 and
273-277, so an interrupted CREATE INDEX CONCURRENTLY cannot be mistaken
for success on retry.
Three more indexes, each an unindexed FK referencing column that the
per-batch statements need and that the FK's own ON DELETE action already
scanned per row on ordinary deletes: task_token(agent_id),
chat_draft_restore(task_id), autopilot_run(task_id).
Plans re-verified on a 1.2M-row synthetic copy of the prod shape with the
child tables populated: page query is a Bitmap Index Scan under LockRows;
task_usage, chat_draft_restore, task_token and autopilot_run arms are all
index scans.
Tests: the ownership-path test now asserts the sweep is self-sufficient
inside the transaction, before any owner row is deleted, so a cascade
cannot stand in for teardown. New tests cover the enqueue / reassignment
fence (blocked writers, unrelated writer unaffected), multi-page sweeps
and the page bound, plus a migration-retry repair test and a check that
every cleanup hook names the index its migration builds.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): keyset-page the teardown sweep and fail closed on leaks
Addresses the second review round on MUL-5999.
1. Bounded scan, not just bounded result. The page query was
`owner = $1 ORDER BY id LIMIT n` against indexes that cannot produce id
order, so the executor had to read the owner's whole task set before
applying LIMIT — quadratic in pages for a busy owner, which is the case
this issue is about. Measured on a 550k-row probe: the old shape spent
11,630 buffers per page (planner chose a full pkey index scan; on other
statistics it chose an explicit Sort), the keyset shape spends ~2,000
and a later page costs the same as the first.
Pages are now `owner = $1 AND id > $cursor ORDER BY id LIMIT n` against
composite indexes: 273 becomes (runtime_id, id), 278 adds (agent_id, id),
279 adds (issue_id, id) and 280 drops the single-column issue_id index it
supersedes so the hot table does not carry both.
2. Owner enumeration is bounded too. The three LockWorkspaceTaskOwner*
statements no longer return ids at all — they are a set-based write fence
and now cost the API process nothing. Owners are walked separately in
keyset pages of 500 (281-283 add the (workspace_id, id) indexes those
pages need), so a workspace with a huge agent or issue set never has its
whole owner list held in memory.
The keyset column is id rather than a natural key on purpose: a cursor
column must be immutable, and renaming an owner mid-teardown would
otherwise let it sort behind the cursor and escape the sweep.
3. The fence is no longer the only guarantee. Blocking a concurrent enqueue
still relies on an INSERT taking FOR KEY SHARE on the owner rows it
references, which is a property of the legacy FKs rather than a protocol
every enqueue path opts into. Two things now backstop it:
- every owner is re-swept until a pass finds nothing, and teardown fails
closed (rolling the whole transaction back) if tasks keep arriving,
instead of committing a partially cleaned tenant;
- a schema-contract test asserts the three owner FKs still exist, so a
migration that removes them turns the missing fence into a red build
rather than a silent hole.
An explicit deleting-state protocol shared with all seven enqueue paths
remains the right long-term fix and is out of scope here.
End-to-end teardown, real handler, identical synthetic data, main vs this
branch (1.2M-row agent_task_queue, 200k task_token, 300k task_usage):
small workspace, 5k tasks: 29,247 ms -> 1,013 ms (4.38M -> 0.27M blocks)
busy workspace, 60k tasks: 243,863 ms -> 1,724 ms (5.76M -> 2.23M blocks)
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): fence task writes in-transaction, independent of the FKs
Addresses the third review round on MUL-5999.
1. The re-scan did not close the window, and the reviewer is right that it
cannot: a pass that comes back empty returns success, and a task
inserted after that final scan but before COMMIT is not caught by any
number of passes. The only thing that closed that window was the
FOR KEY SHARE an INSERT takes on the agent / issue / runtime rows it
references — a side effect of foreign keys that workspace_delete.sql
itself documents as an expand-phase net a later contract removes.
Migration 284 replaces it with an explicit fence: a BEFORE INSERT OR
UPDATE OF (agent_id, issue_id, runtime_id) trigger on agent_task_queue
that takes FOR KEY SHARE on the workspace row behind every owner the row
references, and raises if one is gone. That conflicts with the FOR UPDATE
LockWorkspaceForDelete already holds for the whole teardown transaction,
so a task write against a workspace being torn down blocks until teardown
finishes and then either proceeds (rolled back) or fails closed
(committed).
A trigger rather than a per-call-site check because the fence has to be
in the same transaction as the write: there are seven INSERTs and several
ownership-changing UPDATEs across agent.sql, autopilot.sql, chat.sql and
runtime.sql, several of them single autocommit statements, so a separate
"take the fence" statement from the caller would be its own transaction
and protect nothing. In-transaction by construction, covers every current
path, and cannot be forgotten by a path added later.
TestTaskWriteFence_BlocksOnWorkspaceLockAlone isolates the fence from the
FKs: the holder locks ONLY the workspace row — no agent, issue or runtime
row — so no FK on agent_task_queue has anything to block on, since none of
them references workspace. Inserts through all three ownership paths and a
reassignment all block; an unrelated workspace's insert and a status-only
update still pass. Dropping the trigger fails that test.
Cost, measured with 5,000 bare inserts: 147 ms without the fence,
238 ms with it — about 18 us per inserted row for up to three indexed
lookups and a shared row lock. It does not fire on the hot claim /
dispatch / complete path, which updates status and timestamps rather than
ownership.
The verification passes stay as a self-check, no longer described as the
concurrency guarantee.
2. Migration 280 is withdrawn. Its down path rebuilt the superseded index
with CREATE INDEX CONCURRENTLY, and the runner only runs INVALID-index
cleanup hooks in the up direction, so an interrupted rollback would leave
an invalid index that a later `down` skips — and rolling back 279 too
would then leave no usable issue_id index at all. The redundant
single-column index stays for now; dropping it needs its own migration
with a direction-aware cleanup. This also settles the stale comment in
task_usage.sql, which still names the old index and remains correct.
3. Keyset cursors no longer start from the all-zero uuid, which is itself a
valid uuid and would have made a row with that id invisible forever.
Every walk now has an explicit first-page query with no cursor predicate.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): move the task-write fence into the write statements
Addresses the fourth review round on MUL-5999.
1. The trigger is gone. Migration 284 now defines
lock_task_owner_workspaces(agent_id, issue_id, runtime_id), a predicate
that every statement writing agent_task_queue ownership calls in its own
WHERE clause:
INSERT INTO agent_task_queue (...)
SELECT ... WHERE lock_task_owner_workspaces($1, $3, $2)
Nine statements now carry it: the seven task INSERTs (agent.sql x5,
autopilot.sql, chat.sql), LinkTaskToIssue, and ReassignTasksToRuntime —
the last through a MATERIALIZED CTE so a bulk UPDATE evaluates the
VOLATILE predicate once instead of per row. Clearing runtime_id is
exempt: it cannot move a task into another workspace.
Calling it from the writing statement keeps the lock in the writer's own
transaction, which a separate caller-issued statement could not do for
the autocommit enqueue paths. TestTaskOwnershipWritesCallTheFence is the
trade for dropping the trigger: it scans the query files and fails the
build if a statement writes ownership without the fence.
2. The fence no longer defers to the foreign keys. The trigger version
counted only owner references that still resolved, so a deleted owner
made itself invisible: with all three gone it returned early, and with
one of three gone the count still matched. The predicate now compares
resolved owners against the number of owner arguments supplied and
returns false on any mismatch, so a stale write is rejected on its own
terms. TestLockTaskOwnerWorkspaces_RejectsUnresolvableOwner covers each
shape, including the one-of-three case.
3. Deterministic lock order. The workspace rows are locked with
ORDER BY w.id, so mirrored cross-workspace writes cannot take the same
pair in opposite orders. Honest note on the test: 25 mirrored concurrent
pairs never deadlock either way, because the current plan happens to
return the rows in a stable order — the ORDER BY is what makes that a
guarantee rather than an accident of the plan, and the test is a
regression test for the symptom.
4. The fail-closed test now tests what its name says. The previous one only
inserted a neighbour-owned task and could not have failed. The
replacement runs the actual race: a writer calls the real
CreateAgentTask, blocks on the workspace lock, teardown deletes the
workspace and commits, and the writer must come back with pgx.ErrNoRows
— the fence's own verdict. An FK doing the work instead surfaces as
SQLSTATE 23503, which the assertion rejects explicitly.
Cost: 5,000 predicate calls in 53.7 ms, about 11 us per fenced write, for
up to three indexed lookups and a shared row lock. Status, dispatch and
completion updates do not call it.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): stop the legacy merge when the task fence refuses
Addresses the P1 from the fifth review round on MUL-5999.
ReassignTasksToRuntime returned only a row count, so a fenced refusal was
indistinguishable from "the old runtime had no tasks". mergeLegacyRuntimes
read both as success and went on to DeleteAgentRuntime(old), where
agent_task_queue's ON DELETE CASCADE silently took the history the
reassignment exists to preserve — and once that FK goes away, the same path
would instead leave tasks pointing at a deleted runtime. Either way the
merge was leaning on the cascade, which is what this PR set out to stop.
The query now returns the fence verdict alongside the count:
SELECT (SELECT ok FROM fence) AS fence_ok,
(SELECT count(*) FROM reassigned) AS reassigned_tasks
and the merge is a single helper, mergeLegacyRuntime, that aborts with
errRuntimeMergeFenced before touching anything else when fence_ok is false.
Tasks now move FIRST, because that is the step carrying the fence: nothing
else in the merge — agent reassignment, legacy daemon_id bookkeeping, the
old runtime delete — runs unless the fence agreed. The next daemon
registration re-runs the merge, by which time the workspace is either gone
or its teardown rolled back.
TestMergeLegacyRuntime_KeepsOldRuntimeWhenFenceRefuses covers it: with a
vanished merge target the old runtime and its task both survive and the task
stays on the old runtime, while a merge onto a live runtime still completes.
Removing the abort fails that test — and fails it with SQLSTATE 23503 from
agent_runtime_id_fkey, which is precisely the "an FK is doing the work"
shape the review objected to.
Also replaced the fixed 300 ms sleep in the queued-writer race with a poll
on pg_stat_activity for a backend actually parked on a lock, so the test
cannot pass without the race being set up (it now fails on timeout instead).
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): make the legacy runtime merge a single transaction
Addresses the P1 from the sixth review round on MUL-5999.
The merge ran four autocommit statements, so the fence's workspace lock
ended with the statement that took it. Two ways that left a tenant
inconsistent while daemon registration still reported success:
* a transient failure on the agent reassignment committed the task move
and nothing else, leaving tasks on the new runtime and their agents on
the old one until some later registration happened to retry;
* in the gap after the task commit, a workspace teardown could lock and
delete the target runtime — and its UnbindTasksFromRuntime step clears
runtime_id on exactly the tasks that had just moved. The next merge
looks tasks up by the OLD runtime id, so that history was detached for
good.
mergeLegacyRuntime now opens a transaction and runs the whole merge on one
qtx: fenced task reassignment, agent reassignment, the legacy daemon_id
audit write, and the old-runtime delete, then commits. The fence stays the
first statement, so the workspace lock it takes is held until COMMIT — a
teardown cannot start deleting the target mid-merge — and the lock order
against teardown is unchanged. Any failure returns an error and rolls the
whole thing back.
The audit write is no longer best-effort: inside a transaction a failed
statement poisons the rest, so it either lands with the merge or the merge
is abandoned.
TestMergeLegacyRuntime_RollsBackWhenAStepFails injects the failure in the
database rather than through a code seam — a trigger makes the agent move
raise — and asserts the task, the agent, both runtime rows and the audit
column are all exactly as they were, then that the same merge completes
once the injection is removed. Reverting to autocommit fails it with
"task moved to <new runtime>; the task reassignment was not rolled back".
Also took the non-blocking suggestion: the queued-writer race now has the
writer report its own pg_backend_pid() over a channel and waits for that
specific backend to park on a lock, instead of matching statement text
against every backend on a shared test database.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): fence late writes against the runtime a merge is collapsing
Addresses the P1 from the seventh review round on MUL-5999, which the
reviewer reproduced rather than inferred.
The fence locked the workspace row and nothing else. A task write takes
FOR KEY SHARE there, and so did the legacy runtime merge — and two KEY SHARE
locks do not conflict. So an enqueue against the OLD runtime went straight
through after the merge had scanned its tasks, and DeleteAgentRuntime then
removed that brand-new task through agent_task_queue's ON DELETE CASCADE.
Making the merge one transaction widened the window rather than creating it:
the agent's new runtime_id is invisible until COMMIT, so concurrent writers
keep resolving the old runtime for the whole merge.
Both sides of the contract now lock the owner rows too, in one order:
workspace rows (by id), then agent, issue, runtime.
* lock_task_owner_rows (migration 284, renamed from
lock_task_owner_workspaces) takes FOR KEY SHARE on each owner row after
the workspace rows, and re-checks that they still exist. That is what
conflicts with a delete.
* The merge takes FOR KEY SHARE on the workspace and then FOR UPDATE on
both runtime rows in id order, before the task scan, and holds them to
COMMIT. FOR UPDATE is the level that conflicts with a writer's KEY
SHARE; both runtimes in one ordered statement so two merges running in
opposite directions cannot take the pair in opposite orders. Locking
fewer rows than asked for means one disappeared, and the merge is
abandoned.
A late writer therefore either commits before the merge takes the lock — in
which case the scan that follows sees its task and moves it — or waits, and
then finds the runtime gone and writes no row.
TestRuntimeMerge_LateWriteToOldRuntimeIsFencedNotCascaded drives the merge
statement by statement so it can stop where the window was (after the agent
update, before the old runtime is deleted), enqueues against the old runtime
through the generated query, and requires that the writer is parked on a
lock, that it comes back as pgx.ErrNoRows, and that the merge's own history
ends up on the target. It explicitly rejects both 23503 and a row that
lands and then vanishes.
Both halves are load-bearing: dropping the runtime lock from the fence, or
the FOR UPDATE from the merge, fails that test — the latter with "the late
write was rejected by a foreign key (23503), not by the fence".
Cost: 5,000 predicate calls in 74.4 ms, ~15 us per fenced write (was ~11 us
with the workspace lock alone).
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* docs(workspace): correct the fence comment and drop a trailing newline
Two review nits, no behaviour change.
lockWorkspaceTaskOwners still described migration 284 as a BEFORE trigger on
agent_task_queue. It became a called predicate, lock_task_owner_rows, in
|
||
|
|
b06af2ae17 |
feat(runtime): unbind agents on runtime delete instead of destroying them (#6220)
* feat(runtime): unbind agents on runtime delete instead of destroying them Deleting a runtime archived its agents and then hard-deleted the rows, so the agents and every conversation with them disappeared — while the confirmation dialog said "archive", which a user reasonably reads as recoverable. Retiring a laptop is an ordinary action; losing the agents configured on it is not an ordinary consequence. An agent is now a persistent business object and a runtime is replaceable execution capacity: deleting a runtime unbinds its agents. `runtime_id IS NULL` means unbound — orthogonal to archived — and the agent keeps its instructions, skills, chats, labels, channel installations, autopilots and task history. service.AgentReadiness already refused an agent with no runtime, so the scheduling safety gate needed no change. Two columns become nullable, not one. Without `agent_task_queue.runtime_id`, deleting the runtime still cascades the task history away (and task_message / task_usage / task_token with it), so the agents would survive with no record of anything they did — the same class of loss. A NOT VALID CHECK keeps NULL confined to history: an active task must always have a runtime, so claim / dispatch / delivery-CAS paths can never observe one without. It is written against completed_at rather than a status list so a future non-terminal status fails closed instead of slipping through. Two prerequisites this depends on: - 'deferred' (migration 128) was missing from CancelAgentTasksByRuntimeOrAgent. It went unnoticed because the delete used to cascade those rows away; with the new CHECK it would abort the delete and make the runtime undeletable. - The channel-installation / label / chat-pin / invocation-target / draft-restore cleanups were scoped to "archived agents on this runtime". Archived user agents now survive, so that scope is narrowed to kind='system' — otherwise the fix would produce a subtler loss: agent alive, configuration wiped. Also removes the squad guard that refused (409) when an active squad's leader was an archived agent on the runtime, plus the archived-squad delete that existed only to get past squad.leader_id's RESTRICT FK. The leader is no longer deleted, so nothing needs to be given up to retire a machine. Autopilots are no longer paused either: their assignee survives, and a rebind restores them without the owner having to remember to re-enable. Reason codes: an unbound agent reports agent_runtime_required, not runtime_offline. The copy for runtime_offline tells users to reconnect a machine; an unbound agent has no machine to reconnect, and the fix is to bind a runtime. Chat's bare 409 string gains the same code so the composer can offer that action. API: agents gain runtime_bound. runtime_id stays a string (empty when unbound) so installed clients keep parsing and no gated two-release rollout is needed. The confirmed-delete endpoint is /unbind-agents-and-delete; /archive-agents-and-delete still routes to it, and the compared expected_active_agent_ids set is unchanged — widening it would 409 every older client forever. Co-authored-by: multica-agent <[email protected]> * fix: make runtime unbinding recoverable Co-authored-by: multica-agent <[email protected]> * fix: address runtime unbind review nits Co-authored-by: multica-agent <[email protected]> * fix: resolve runtime unbind review blockers Co-authored-by: multica-agent <[email protected]> * fix(migrations): renumber runtime unbind after main merge Co-authored-by: multica-agent <[email protected]> * test(daemon): avoid late-request lease flake Co-authored-by: multica-agent <[email protected]> * test(autopilots): bind validation fixture runtime Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Eve <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
c3dd9ec845 |
Machine-level batch task claim endpoint (MUL-4257) (#5193)
* feat(daemon-claim): machine-level batch task claim endpoint (MUL-4257) Collapse the per-runtime /tasks/claim poll fan-out into a single machine-level batch claim to cut /api/daemon claim request volume. Server: - agent.sql: = ANY(runtime_ids) batch variants of the claim queries (ListQueuedClaimCandidatesByRuntimes, PromoteDueDeferredTasksForRuntimes, ReclaimStaleDispatchedTasksForRuntimes); runtime.sql: GetAgentRuntimes(= ANY) so a whole machine's runtimes are resolved/promoted/reclaimed/listed in a constant number of queries instead of N. - service.ClaimTasksForRuntimes: claim up to max_tasks across a runtime set, preserving per-(issue,agent) serialization, the concurrency cap, the empty-claim cache short-circuit, and every dispatch side effect. Batch promote replays the per-row side effects (task:queued + empty-cache Bump). - handler.ClaimTasksByRuntime (canonical POST /api/daemon/tasks/claim, with a transitional /claim alias): validates daemon_id (required; must match the mdt_ token) and rejects runtimes bound to a different daemon (group-ownership check mirroring the WS path); resolves+authorizes each runtime_id; claims; and finalizes each task through the SAME FinalizeTaskClaim as the per-runtime endpoint (atomic token + delivered_comment_ids receipt), requeueing the exact claim and omitting it on failure. buildClaimedTaskResponse is extracted from the per-runtime handler and returns the delivered-comment ids plus a structured *claimBuildFailure so both paths share identical payload building and failure semantics (workspace-isolation, chat-input load/empty). - max_tasks: negative -> 400, zero -> empty (never coerce to 1), positive capped at 32. runtime_ids parsed with non-panicking util.ParseUUID. Daemon: - Client.ClaimTasks posts daemon_id + runtime set + free-slot count to the canonical path under a short request-scoped timeout, bounding the head-of-line coupling the per-runtime pollers avoid (MUL-1744). Tests: service batch drain / max_tasks cap / deferred-promote receipt / finalize-failure rollback+requeue; handler routing + token, cross-workspace skip, cross-daemon skip, daemon_id required, owner-missing cancel, max_tasks=0/negative, invalid-uuid skip, comment delivery receipt, stale-reclaim replacement receipt; client posts/parses (daemon_id + canonical path). Follow-up: cut the daemon pollLoop over to a single batched poller (flips the MUL-1744 isolation contract; needs its concurrency tests redesigned). Co-authored-by: multica-agent <[email protected]> * feat(daemon-ws): generic WS request/response transport for daemon RPC (MUL-4257) Add a generic daemon->server request/response layer over the existing WS control connection, the transport for WS-first claim (HTTP fallback): - protocol: daemon:rpc_request / daemon:rpc_response envelopes with a correlation request_id + method + body, and an rpc-v1 capability gate. - daemonws.Hub: SetRPCHandler + goroutine-dispatched handleRPCFrame (bounded by a per-connection in-flight cap) that echoes the request_id; missing handler / saturation return non-2xx so the daemon falls back to HTTP. Read limit raised to 64KB for rpc requests carrying a runtime set. - hub tests: round-trip, handler-error->non-2xx, no-handler->503. Co-authored-by: multica-agent <[email protected]> * feat(daemon-ws): WS-first task claim over the generic RPC transport (MUL-4257) Bind claim to the WS request/response layer, with HTTP fallback: - server: handler.DaemonRPCHandler adapts a daemon:rpc_request (method tasks.claim) to the existing HTTP ClaimTasksByRuntime via a synthetic in-process request carrying the WS connection's identity (daemon_id + workspace + capabilities), so all auth / payload-building / finalization is reused unchanged. Wired via daemonHub.SetRPCHandler. ClientIdentity now captures X-Client-Capabilities so capability gating matches the HTTP path. - daemon: wsRPCClient correlates responses by request_id over the shared WS connection; attached to the live connection's write channel (guarded so a Call racing teardown never sends on a closed channel) and detached on disconnect. rpc_response frames are routed in the read loop. Daemon.ClaimTasksWSFirst issues tasks.claim over WS and falls back to the HTTP claim endpoint on any transport failure (no conn / buffer full / timeout) — wired into the poller at the poller cutover. - tests: handler tasks.claim RPC end-to-end (claims + dispatches) + unknown method 404; daemon wsRPCClient round-trip / timeout / unavailable / server-error / detach-fails-pending (all under -race). Co-authored-by: multica-agent <[email protected]> * feat(daemon): cut claim poller over to machine-level ClaimTasksWSFirst (MUL-4257) Replace the per-runtime HTTP poll loop with a single batch poller: each cycle acquires all free execution slots (slot-before-claim) and issues ONE ClaimTasksWSFirst across every runtime the daemon hosts (WS-first, HTTP fallback), dispatching each returned task to its runtime. Wakeups (targeted / catch-up / runtime-set change) collapse to one nudge. Removes runRuntimePoller + runtimePollOffset. The WS handshake now advertises the same capabilities as HTTP (+ rpc-v1) so WS-built claim payloads keep skill-ref / coalesced-comment gating. Trades per-runtime isolation (MUL-1744) for one request, bounded by the short per-request WS timeout / client timeout. Tests: batch poller claims across runtimes + skips-at-capacity + pollLoop shutdown drain (replacing the per-runtime poller tests); heartbeat isolation + runtime-set watcher kept. Co-authored-by: multica-agent <[email protected]> * fix(daemon-ws): WS RPC disconnect-race panic + batch stale-comment-plan repair (MUL-4257) Two PR #5193 review blockers: 1) WS RPC send-on-closed-channel race, both ends: - server: give each connection a cancelable ctx (cancelled on readPump teardown) and run the RPC handler under it, so a slow claim stops on disconnect; guard c.send with sendMu/sendClosed (trySend) so a late RPC response goroutine never writes to the closed channel. Heartbeat ack routed through the same guard. - daemon: wsRPCClient.deliver now sends under the mutex, serialized with attach(nil)'s close+delete, so a delivered response can't hit a channel the detach path just closed. - regressions (-race): daemon deliver-vs-detach; server disconnect-during-handler-response. 2) batch claim now runs the stale-comment-plan repair: extracted the per-runtime handler's repair (trigger deleted, only coalesced survive -> cancel + replay survivors) into shared repairStaleCommentPlanIfNeeded, called by both claim paths. Prevents the batch path (now the default poller) from finalizing+dispatching a task with no comment input and silently dropping the surviving user comment. Regression: batch omits the stale task, cancels it, and rebuilds the survivor into a new trigger plan. Co-authored-by: multica-agent <[email protected]> * fix(daemon-ws): server-side RPC deadline + legacy claim fallback (MUL-4257) Two review blockers: 1) WS RPC timeout/fallback (GPT-Boy): the daemon's WS wait didn't cancel server-side claim, so a slow WS claim could commit after the daemon fell back to HTTP, leaking dispatched tasks and breaking the free-slot bound. Fix: RPC envelope carries TimeoutMs; the server bounds the handler ctx by it (so ClaimTasksByRuntime's tx is cancelled/rolled back at the deadline), and the daemon waits budget + grace so a claim that committed before the deadline still reports back. A committed-then-unreported claim degrades to the same stale-reclaim safety net as HTTP, never a double effective claim. Regression: server-side TimeoutMs cancels the handler. 2) Backward compat (Terra-Boy): a new daemon against a server without the batch route (/api/daemon/tasks/claim 404) couldn't claim. Fix: ClaimTasksWSFirst falls back to the legacy per-runtime ClaimTask loop on a batch 404 and caches 'batch unsupported' (reset on WS reconnect to re-probe after a server upgrade). Regression: server exposing only the legacy route. Co-authored-by: multica-agent <[email protected]> * fix(daemon-ws): no double-claim on WS teardown/detach (MUL-4257) Sol-Boy review blocker: on reconnect, teardown failed the pending RPC (→ HTTP fallback) but then flushed the queued tasks.claim frame to the still-alive socket, so the server committed the WS claim on top of the HTTP one — double claim, WS batch orphaned to stale reclaim, breaking the free-slot bound. - Teardown now closes the connection FIRST, so runWSWriter discards the queued RPC frame (write error path) instead of delivering it. - A detach while a claim's frame is already in flight now returns a distinct errWSRPCUncertain; ClaimTasksWSFirst does NOT HTTP-fall-back on uncertain (the WS claim may have committed) — it skips the cycle and lets reclaim / the next poll recover. Genuine 'not sent' / timeout still fall back (safe: the server-side deadline guarantees no uncommitted claim by budget+grace). - Regression: detach during an in-flight WS claim asserts zero HTTP claims (at most one path claims); plus the existing detach/deliver-race and server-timeout tests. Co-authored-by: multica-agent <[email protected]> * fix(daemon-ws): cancelable RPC frames close the backpressure double-claim (MUL-4257) Sol-Boy review blocker: the client's response budget starts at enqueue, but the socket write is async (10s write deadline). A backpressured writer could hold a tasks.claim in the local queue past the client timeout — the daemon HTTP-fell-back, then the writer woke and delivered the stale WS frame, so the server committed it too: same free slots claimed twice. No detach occurs, so the prior errWSRPCUncertain fix did not cover it. - WS frames are now cancelable (wsOutbound{sent,canceled} under a mutex). The writer calls beginWrite() before WriteMessage and skips cancelled frames. - On give-up (timeout / detach / ctx), Call cancels the queued frame: if it was still pending the cancel wins and the frame is guaranteed never delivered (errWSRPCUnavailable → safe HTTP fallback); if the writer already began sending it the cancel loses and the outcome is errWSRPCUncertain (no fallback). The decision is atomic, so at most one transport claims. Tests: wsOutbound cancel-before-write vs write-before-cancel; Call timeout cancels an unsent frame (writer then drops it) vs uncertain when already sent; plus the updated detach and existing timeout/race tests. Co-authored-by: multica-agent <[email protected]> * fix(batch-claim): return partial success instead of dropping committed claims (MUL-4257) Sol-Boy review blocker: ClaimTasksForRuntimes reclaims (step 2) and claims per agent (step 6) in independent transactions, but a step-4 candidate-SELECT error or a mid-loop ClaimTask error did 'return nil, err' — discarding tasks already committed as dispatched. The handler 500s; the daemon sees a definite (non- uncertain) 500 and HTTP-falls-back, claiming a SECOND batch into the same free slots while the first batch waits for stale reclaim — the double-claim this PR removes. - Both error paths now prefer partial success: if any task has already committed (claimed non-empty), return it (nil error) so the handler finalizes and returns 200; the errored candidates stay queued for the next poll. The remaining error is logged. Only a genuinely empty result still returns the error (safe: no committed claim to lose, HTTP fallback just re-fails). Regression (internal/service, DB-backed, fault-injected): - PartialSuccessOnSecondAgentClaimFailure: fail the 2nd ClaimTask's Begin → the first agent's committed task is returned, not dropped. - PartialSuccessOnCandidateQueryFailureAfterReclaim: a stale dispatched task is reclaimed, then the candidate SELECT fails → the reclaimed task is returned. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Eve <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Jiayuan Zhang
|
1427e8abd3 | feat(agents): add conversational creation studio (#5296) | ||
|
|
fd58e13bec |
feat(runtimes): custom runtime names + searchable machine-grouped picker (MUL-4217) (#5070)
* feat(runtimes): custom runtime names + searchable machine-grouped picker
MUL-4217. Runtime names were daemon-generated ("Claude (host)") and
uneditable, so picking one at agent-create time was painful once a
workspace had many machines.
Phase 1 — create-agent RuntimePicker: add a search box (>6 runtimes) and
group options by machine (Local/Remote/Cloud, online-first, current
machine first) reusing buildRuntimeMachines/filterRuntimeMachines. Rows
show the provider under a machine header instead of a flat repeated list.
Phase 2 — custom names: new nullable agent_runtime.custom_name column,
never written by the registration/heartbeat upsert so the daemon can't
clobber it; display is coalesce(custom_name, name) via runtimeDisplayName.
PATCH /api/runtimes/:id gains custom_name (+ apply_to_machine to name every
runtime sharing a daemon_id in one action, owner-scoped for non-admins).
Rename UI on the runtime detail page; `multica runtime rename` CLI command.
Verified: go build/vet, sqlc, handler tests (incl. new custom-name single
+ machine-fanout), 1650 views + 764 core TS tests, typecheck, locale parity.
Co-authored-by: multica-agent <[email protected]>
* fix(runtimes): address review — persist machine name on new registrations, keep custom_name in register response
Elon's review on #5070 (MUL-4217):
1. Machine name looked "lost" when a new provider registered on an
already-named machine — the new row landed with custom_name=null and
broke sharedCustomName. Now a fresh runtime inherits the machine's shared
custom name at register time (ListDaemonCustomNames + sharedDaemonCustomName),
so the machine title stays stable as providers come and go.
2. DaemonRegister rebuilt the response row by hand and dropped custom_name,
so register returned custom_name:null — inconsistent with list/get/update.
Both branches now carry CustomName.
Also: tighten the updateRuntime patch type to custom_name?: string (drop the
misleading `| null`, since the server treats null as "unchanged", not "clear").
Tests: register response preserves custom_name; new runtime inherits machine name.
Co-authored-by: multica-agent <[email protected]>
* fix(runtimes): inherit machine name for failed-profile registrations too
Elon's re-review of #5070 (MUL-4217): the machine-name inheritance added
last round only covered the normal req.Runtimes path. The req.FailedProfiles
branch also upserts a daemon_id-scoped agent_runtime row (offline, profile
registration error), which shows up in the runtime list / machine grouping —
so on a named machine a failed custom-profile row landed with custom_name=NULL
and dragged the machine title back to the hostname.
Extract the inheritance into h.inheritMachineCustomName and call it from both
the normal runtime path and the failed-profile path. Add a test: named daemon
+ failed profile upsert -> the failed row's persisted custom_name is inherited.
Co-authored-by: multica-agent <[email protected]>
---------
Co-authored-by: J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
|
||
|
LinYushen
|
c5eb778532 |
Revert "fix: keep runtime provider arbiter during profile rollout (#4251)" (#4258)
This reverts commit
|
||
|
|
a08281a1b2 |
fix: keep runtime provider arbiter during profile rollout (#4251)
Co-authored-by: Eve <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
LinYushenandmultica-agent
|
52e76e7b23 |
MUL-3284: server API + daemon (custom runtime PR2) (#4149)
* MUL-3284: add runtime_profile schema (custom runtime PR1) Schema-only foundation for custom runtimes. Additive migration 120: - New workspace-level `runtime_profile` table: the shared, team-visible definition of a custom runtime (e.g. an in-house Codex wrapper). protocol_family is CHECK-constrained to the exact backend list in agent.New() (server/pkg/agent/agent.go). The only args column is `fixed_args` (args every agent on the runtime must inherit); there is deliberately no generic per-agent args field — those stay on agent.custom_args. - `agent_runtime.profile_id` (nullable, FK -> runtime_profile ON DELETE CASCADE): NULL = built-in runtime, non-NULL = a registered instance of a custom profile. - Partial unique index agent_runtime_workspace_daemon_profile_key on (workspace_id, daemon_id, profile_id) WHERE profile_id IS NOT NULL. The legacy UNIQUE (workspace_id, daemon_id, provider) constraint is left INTACT so the existing registration upsert (ON CONFLICT (workspace_id, daemon_id, provider) in runtime.sql) keeps resolving its arbiter and the server stays green. Converting that key to a partial (WHERE profile_id IS NULL) index and making the upsert profile-aware is PR2's registration work, not this migration. Verified up + down against Postgres 17: full `migrate up` applies 120; schema shows the table, column, partial index and intact legacy constraint; functional checks pass (partial index blocks dup (ws,daemon,profile), allows same profile on another daemon; CHECK and display_name uniqueness reject bad input; legacy ON CONFLICT still resolves; profile delete cascades to instances); down/up round-trip is clean. Co-authored-by: multica-agent <[email protected]> * MUL-3284: drop DB FKs/cascade from runtime_profile migration (review fix) Per review (house rule: no new database foreign keys / cascades; relational integrity lives in the application layer): - runtime_profile.workspace_id: drop REFERENCES workspace ON DELETE CASCADE -> plain UUID NOT NULL. - runtime_profile.created_by: drop REFERENCES "user" ON DELETE SET NULL -> plain UUID. - agent_runtime.profile_id: drop REFERENCES runtime_profile ON DELETE CASCADE -> plain UUID. CHECK constraints, UNIQUE (workspace_id, display_name), the workspace index, and the partial unique index agent_runtime_workspace_daemon_profile_key are unchanged. The legacy UNIQUE (workspace_id, daemon_id, provider) constraint remains untouched. Behavioral consequence: the database no longer auto-removes a profile's agent_runtime instance rows on profile delete. That cleanup moves into PR2's profile-delete path. Up-migration comments document this; down-migration comment no longer references FKs/cascade. Re-verified on Postgres 17: migrate up applies 120; no FK constraints exist on the new columns; partial index still blocks dup (ws,daemon,profile_id); CHECK and display_name uniqueness still reject bad input; deleting a profile now leaves the runtime row orphaned (proving cascade is gone); down/up round-trip clean with the legacy constraint intact. Co-authored-by: multica-agent <[email protected]> * MUL-3284 PR2 (server): runtime_profile CRUD + profile-aware registration Server/DB half of the custom-runtime feature. - Migration 121: convert the legacy UNIQUE (workspace_id, daemon_id, provider) constraint on agent_runtime into a partial unique index scoped to built-in rows (WHERE profile_id IS NULL). With 120's partial index on profile_id this lets one daemon host the built-in provider AND custom profiles of the same protocol family without collision. - Queries: runtime_profile CRUD; ListEnabledRuntimeProfilesForWorkspace (daemon-facing); CountAgentsByProfile + DeleteAgentRuntimesByProfile for the app-layer cascade; profile-aware UpsertAgentRuntimeWithProfile; the built-in UpsertAgentRuntime ON CONFLICT now spells out WHERE profile_id IS NULL so it targets the right partial index. sqlc regenerated. - agent.SupportedTypes / IsSupportedType: single-source protocol_family whitelist, in lockstep with agent.New and the migration 120 CHECK. - Handlers + routes: runtime_profile CRUD (member-read, admin-write) with protocol_family whitelist validation, display_name uniqueness (409), and fixed_args validation (no generic per-agent args — iron rule); a daemon-token endpoint GET /api/daemon/workspaces/{id}/runtime-profiles; DeleteRuntimeProfile does the app-layer cascade (delete instance rows then profile, in one tx) and refuses (409) while active agents are bound. - DaemonRegister accepts an optional per-runtime profile_id: validates the profile belongs to the workspace and is enabled, registers via the profile-aware upsert, and skips legacy hostname merge for custom rows. AgentRuntimeResponse now carries profile_id. Verified on Postgres 17: migrate up through 121; built-in + custom codex coexist on one daemon; both upsert arbiters are idempotent; delete-by-profile cascade removes only the custom instance; migrate down reverses 121 then 120 and replays clean. go build ./... and go vet pass; handler test package compiles. Daemon-side wiring (fetch profiles, PATH-resolve command_name, register with profile_id, exec uses command_name) lands in a follow-up commit on this branch. Co-authored-by: multica-agent <[email protected]> * MUL-3284 PR2 (daemon): pull profiles, PATH-resolve, register, exec command Daemon-side half of custom runtime profiles, against the server contract on this branch. - client.go: GetRuntimeProfiles(workspaceID) -> GET /api/daemon/workspaces/{id}/runtime-profiles (mirrors GetWorkspaceRepos); RuntimeProfile / RuntimeProfilesResponse types. - types.go: Runtime gains profile_id (parsed from the register response so runtimeIndex carries it). - daemon.go: * appendProfileRuntimes — called inside registerRuntimesForWorkspace before the empty-runtimes guard. Best-effort fetch (older server 404s are logged and swallowed; never fails registration). Per enabled profile: resolve command_name via PATH (exec.LookPath, behind a `lookPath` test hook), skip+log when absent, best-effort version probe, record the resolved absolute path keyed by profile_id, and append a registration entry {name, type=protocol_family, version, status:online, profile_id}. A custom-only host (no built-in agents) still registers. * profileCommandPaths map (guarded by d.mu) + recordProfileCommandPath / customCommandPathForRuntime helpers. * runTask: looks up the claimed task's RuntimeID -> profile command path and overrides the executable path, synthesizing an AgentEntry so a custom runtime runs even when the host has no built-in agent of the same provider. provider (=protocol_family) is unchanged so agent.New still selects the right backend. - Tests: GetRuntimeProfiles request shape; profile runtime appended + path recorded (custom-only host); profile skipped when command not on PATH; profiles-fetch-404 is best-effort; customCommandPathForRuntime bookkeeping. - agent: lockstep test pinning SupportedTypes to agent.New and the migration 120 protocol_family CHECK. Iron rule honored: profile carries no generic per-agent args. fixed_args are parsed and carried but intentionally NOT wired into the launch command yet (optional/best-effort; explicit TODO(MUL-3284) in appendProfileRuntimes). Verified: go build ./... clean; go vet ./internal/daemon/... clean; go test ./internal/daemon/... pass (existing + 5 new); full go test ./internal/handler/ suite passes against a migrated Postgres 17; agent lockstep test passes. Co-authored-by: multica-agent <[email protected]> * MUL-3284 PR2: profile delete runs full archived-agent cascade (fix 500) Review fix. DeleteRuntimeProfile previously guarded only on ACTIVE agents, but agent.runtime_id is ON DELETE RESTRICT — a profile whose runtimes had only ARCHIVED agents passed the guard, then DeleteAgentRuntimesByProfile hit the FK and the handler 500'd. Now it mirrors the mature runtime-delete cascade (DeleteAgentRuntime): in one transaction it enumerates the profile's runtime rows, refuses (409) any with active agents or active squads led by archived agents, then for each runtime pauses autopilots pinned to its archived agents, drops archived squads led by them, and hard-deletes the archived agents before removing the runtime rows and the profile. No code path can now fall through to a raw FK error. - queries: ListAgentRuntimeIDsByProfile (sqlc regen). Reuses the existing per-runtime teardown queries (CountActiveSquadsWithArchivedLeadersByRuntime, ListArchivedAgentIDsByRuntime, PauseAutopilotsByAgentAssignees, DeleteSquadsByArchivedAgentsOnRuntime, DeleteArchivedAgentsByRuntime). - tests: TestDeleteRuntimeProfile_ArchivedAgentCascade (archived-only profile deletes cleanly: 204, runtime + archived agent + profile gone) and TestDeleteRuntimeProfile_ActiveAgentBlocks (active agent → 409, survives). Verified against Postgres 17: both new tests pass; full handler suite, daemon tests, and agent lockstep test pass; go vet clean. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> |
||
|
Thanh MinhandKiro
|
8abdc77961 |
MUL-2489 fix(runtime): delete archived squads before runtime teardown (#2955)
* fix(runtime): delete squads referencing archived agents before runtime teardown The DeleteAgentRuntime handler was failing with 500 'failed to clean up archived agents' because squad.leader_id has an ON DELETE RESTRICT FK on agent(id). When an archived agent was still referenced as a squad leader (even on an archived squad), the DELETE FROM agent query was blocked. Fix: add DeleteSquadsByArchivedAgentsOnRuntime query that removes squads whose leader_id points to an archived agent on the target runtime, and call it before DeleteArchivedAgentsByRuntime in the handler. Closes TMI-85 * test(runtime): cover squad cleanup before archived-agent deletion Adds four tests around the DeleteSquadsByArchivedAgentsOnRuntime fix: * TestDeleteSquadsByArchivedAgentsOnRuntime_Query — query-level: deletes squads whose leader is an archived agent on the target runtime, leaves squads with active leaders or archived leaders on a different runtime alone, and is safe to call when nothing matches. Covers the archived- squad case that originally hid the FK blocker from `multica squad list`. * TestDeleteAgentRuntime_RemovesSquadsLedByArchivedAgents — handler end-to-end regression for TMI-85. Reverting the handler change makes this fail with the exact 500 'failed to clean up archived agents' the user reported. * TestDeleteAgentRuntime_NoSquadsRegression — happy path for runtimes whose archived agents were never squad leaders, ensuring the new step is a no-op there. * TestDeleteAgentRuntime_StillBlockedByActiveAgents — preserves the 409 CountActiveAgentsByRuntime guard so the active-agent contract isn't silently regressed by the new cleanup ordering. Refs TMI-85 * chore: remove internal issue tracker references from test comments * fix(runtime): keep active squads during runtime teardown * fix(runtime): block runtime delete on active archived-leader squads * fix(runtime): make runtime delete 409 path a no-op --------- Co-authored-by: Kiro <[email protected]> |
||
|
|
341ce7bfa5 |
feat: support local working directory for projects (MUL-2618 v1) (#3283)
* feat(project): add local_directory project_resource type (MUL-2662)
Adds a second project_resource type alongside github_repo so a project
can be pinned to an existing directory on a specific daemon (the v1 of
the local-working-directory flow tracked in MUL-2618). The ref schema is
{ local_path, daemon_id, label? }; local_path must be absolute and
daemon_id is required. The same (daemon_id, local_path) pair is allowed
on multiple projects by design — no UNIQUE constraint is added.
Implementation reuses the existing project_resource API surface: the new
type is wired through the validator switch with no migration, no new
events, and no daemon-handler changes (daemon already passes through
arbitrary resource types via ProjectResources). The CLI gains
--local-path / --daemon-id / --ref-label shortcuts so
`multica project resource add --type local_directory` mirrors the
existing `--type github_repo --url ...` ergonomics; the generic --ref
flag still works for both types.
Tests cover the full CRUD lifecycle, the same-path-across-projects
allowance, the same-path-same-project conflict, the validator rejections
(missing/blank/relative path, missing daemon_id, wrong payload type),
and the cross-platform isAbsoluteLocalPath helper.
Co-authored-by: multica-agent <[email protected]>
* feat(project): add update endpoint + label-shadow guard for project_resource (MUL-2662)
Addresses the Elon review on PR #3263:
- Add PUT /api/projects/{id}/resources/{resourceId} with sqlc query,
matching handler, CLI `project resource update`, and a new
EventProjectResourceUpdated WS event. resource_type stays immutable;
ref/label/position are all individually optional.
- Catch same-project (daemon_id, local_path) collisions where only the
embedded label differs — the row-level UNIQUE only matches the full
ref JSON, so a label typo would otherwise let the same working
directory bind twice.
- Tests cover the update lifecycle (label-only / ref / clear / 404 /
invalid path) and the label-shadow conflict on both create and
update; the in-place rename still succeeds because the conflict
scan ignores the row being edited.
Incidental: regenerating sqlc picked up a missing skills_local scan in
UpdateAgentCustomEnv that drifted in from #3200.
Co-authored-by: multica-agent <[email protected]>
* fix(project): close bundled-create label-shadow gap + merge resource_ref on CLI update (MUL-2662)
Two follow-ups from MUL-2662 review round 2:
- CreateProject inline resources path now dedupes local_directory entries on
(daemon_id, local_path) before opening the transaction. The DB-level
UNIQUE(project_id, resource_type, resource_ref) constraint only fires on a
full JSON match, so two rows with the same target but different `label`
would otherwise slip past. Standalone POST/PUT already cover this via
findLocalDirectoryConflict; bundled create was the missing surface.
- `multica project resource update` now seeds resource_ref from the existing
row before applying per-type shortcut flags, so `--default-branch-hint x`
on its own no longer constructs a payload missing `url` (which the server
400s on). Local_directory partial edits get the same merge behavior.
Co-authored-by: multica-agent <[email protected]>
* feat(desktop): local_directory project_resource UI (MUL-2665) (#3273)
* feat(desktop): local_directory project_resource UI (MUL-2665)
First UI surface for the local-working-directory flow tracked in MUL-2618.
Lets users on the desktop pin a project to an existing folder on this
machine; web stays read-only since the per-daemon check can't be done in
the browser.
What's new for the renderer:
- ProjectResourcesSection grows a desktop-only "Add local directory"
button next to the existing GitHub-repo popover. Clicking it opens
Electron's native folder picker, validates the path through a new
IPC pair (existence + r/w), and submits a project_resource of
resource_type=local_directory with daemon_id pulled live from
daemonAPI.getStatus.
- LocalDirectoryRow renders the rename pencil + path tooltip, and
greys out when ref.daemon_id != this machine's daemon_id (with a
"only available on the machine that registered this directory"
tooltip). Delete stays enabled so users can drop stale registrations
from any device.
- LocalDirectoryHint sits above the issue-detail comment composer and
shows "Agent will work in-place at {label} ({path})" when the issue's
project has a local_directory matching this daemon. Hidden on web.
- TaskStatusPill picks up a new "waiting_for_directory_release" stage
that the daemon will publish when it dequeues a task but can't
acquire the path lock. The render is in place now so the daemon
sibling subtask can wire the status string without an additional UI
PR.
Plumbing:
- @multica/core/types gains LocalDirectoryResourceRef +
UpdateProjectResourceRequest, and the api client gets the matching
PUT method backed by the server endpoint that landed in
|
||
|
LinYushenandmultica-agent
|
bf8a346cf0 |
feat(runtimes): cascade-archive agents on runtime delete (MUL-2667) (#3266)
* feat(runtimes): cascade-archive agents on runtime delete (MUL-2667) Replace the bare 409 "cannot delete runtime: it has active agents" with a structured response carrying the blocking agent list, and wire a cascade endpoint that archives those agents, cancels their tasks, pauses dangling autopilots and deletes the runtime in a single transaction. The unified DeleteRuntimeDialog opens directly in cascade mode when the runtime has bound agents, pivots from light to cascade if the strict DELETE refuses with runtime_has_active_agents, and re-prompts when the cascade refuses with runtime_delete_plan_changed (live agent set drifted while the dialog was open). The online-local self-healing rule is preserved at the affordance level (kebab hidden, Diagnostics button disabled with tooltip) and re-checked at confirm time as defence in depth. Co-authored-by: multica-agent <[email protected]> * fix(runtimes): close cascade race + i18n delete dialog (PR #3266 review) - Acquire FOR UPDATE on the runtime row at the top of the cascade tx so FK-validated agent INSERTs/UPDATEs that would point at this runtime block until commit, and lock each currently-active agent row via ListActiveAgentsByRuntimeForUpdate so a concurrent archive/move of an existing active row also blocks. - Switch the bulk archive from runtime-keyed (ArchiveAgentsByRuntime) to ID-keyed (ArchiveAgentsByIDs), narrowed to the user-confirmed expected_active_agent_ids set. Combined with the runtime row lock, this guarantees no agent outside the confirmed plan can be silently archived between plan-compare and archive even at read-committed. - Wire delete-runtime-dialog.tsx to runtimes locale via useT(); add detail.delete_dialog.{light,cascade} keys (EN with _one/_other plurals, zh-Hans _other) covering titles, descriptions, warning, notices, checkbox, buttons, table headers, presence labels, and toasts. Resolves the i18next/no-literal-string CI failure. - Locale parity test passes (51 tests). All 4 dialog test cases pass unmodified (EN copy preserves original wording). Full views vitest: 91 files / 792 tests green; full server go test: green. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> |
||
|
YYClaw
|
614dfae884 |
MUL-2488 feat(timezone): Scheduling / Viewing two-layer timezone architecture (#2968)
* docs(timezone): add scheduling/viewing timezone architecture RFC * feat(db): replace daily rollups with task_usage_hourly, add user.timezone Migrations 100-104: add "user".timezone (Viewing tz), build the UTC hourly task_usage_hourly rollup with its pipeline, drop the legacy task_usage_daily / task_usage_dashboard_daily pipelines, and drop the agent_runtime.timezone column. Report queries now slice day boundaries at read time by the caller-supplied @tz instead of materialising in a fixed tz. Regenerate sqlc. * feat(server): add task_usage_hourly backfill command Replace the two legacy backfill commands (daily / dashboard_daily) with a single backfill_task_usage_hourly that loads historical task_usage into the new UTC hourly rollup, sliced per workspace. * refactor(server): resolve viewing timezone in report handlers Report handlers resolve the Viewing tz per request (?tz query param, then user.timezone, then UTC) and pass it to the hourly-rollup queries. Drop the UseDailyRollup feature flags and the old raw-scan/daily-rollup dual paths, remove the /api/usage endpoints, and stop the daemon from reporting and the runtime handler from accepting host timezone. * refactor(core): switch report queries to viewing timezone API client and dashboard/runtime queries send ?tz with each report request, the user schema/types carry the new timezone field, and the runtime timezone field/mutation is removed. * feat(views): add viewing timezone preference and UI Add the useViewingTimezone hook and a Timezone setting in Preferences; report charts and the dashboard week boundary follow the viewer tz. Remove the runtime detail timezone editor and its locale strings. * fix(test): update fixtures and stabilize tests for timezone refactor The timezone architecture refactor changed several types without updating dependent test code: - RuntimeDevice no longer has a timezone field — drop it from the create-agent-dialog runtime fixture. - User now requires a timezone field — add it to the apps/web mockUser fixture. - The PreferencesTab timezone tests asserted on the async save handler (PATCH then store update) with a bare expect, racing the mutation's settle callback, and timed out querying the Select's ~600-option IANA list on a loaded CI runner. Wrap the assertions in waitFor and extend the timeout for those three tests. * docs(timezone): document self-host migration order and trigger invariant Add a SELF-HOST UPGRADE ORDER runbook to the backfill command's package comment: applying migrations 100-104 in a single migrate-up drops the legacy daily rollups before the hourly backfill runs, leaving dashboards empty until cron catches up. Add an INVARIANT comment on trg_atq_dirty_hourly noting that agent_id must be added to the trigger's OF list if it ever becomes mutable, otherwise dirty buckets for the old agent_id are silently missed. * style(runtimes): drop trailing blank line in runtime-detail |
||
|
|
fc8528d64d |
feat(autopilot): support assigning to a squad (MUL-2429) (#2888)
* feat(autopilot): support assigning autopilot to a squad (MUL-2429) Path A (Squad-as-Leader) from the RFC: when an autopilot's assignee is a squad, dispatch resolves to squad.leader_id and executes against the leader's runtime — semantics match a human manually assigning the issue to that squad, no fan-out. Backend scope only; frontend picker change is a follow-up PR. Changes: - 096_autopilot_squad_assignee migration: drop agent FK on autopilot.assignee_id, add assignee_type column (default 'agent'), add autopilot_run.squad_id attribution column. - service.AgentReadiness: single source of truth for archived / runtime-bound / runtime-online checks. Shared by autopilot admission gate, run_only dispatch, and isSquadLeaderReady. - service.resolveAutopilotLeader: translates assignee_type/id to the agent that actually runs the work. - dispatchCreateIssue: stamps issue with assignee_type='squad' for squad autopilots and enqueues via EnqueueTaskForSquadLeader. - dispatchRunOnly: belt-and-braces readiness re-check after resolving squad → leader so a leader that went offline between admission and dispatch produces a clean failure instead of a doomed task. - handler.CreateAutopilot / UpdateAutopilot: accept assignee_type with squad/agent existence + leader-archived validation. Backward-compatible default of "agent" preserves the contract for older clients. - Analytics: AutopilotRunStarted/Completed/Failed events carry assignee_type and squad_id; PostHog can now group autopilot runs by squad without joining back to the autopilot row. Co-authored-by: multica-agent <[email protected]> * fix(autopilot): reject archived squads, route post-admission skips, cleanup dangling-agent autopilots (MUL-2429) Addresses three review findings on PR #2888: 1. Archived squad handling: validateAutopilotAssignee now rejects squads with archived_at set; resolveAutopilotLeader returns errSquadArchived so the admission gate fails closed; DeleteSquad now mirrors the issue transfer for autopilot rows (TransferSquadAutopilotsToLeader) so surviving autopilots flip to assignee_type='agent' (leader) instead of dangling at the archived squad. 2. dispatchRunOnly post-admission readiness: introduces errDispatchSkipped sentinel, recognised by DispatchAutopilot via handleDispatchSkip so the run is recorded as `skipped` (not `failed`). Manual triggers no longer 500 when the leader's runtime goes offline between admission and task creation. New TestManualTriggerDoesNotErrorOnPostAdmissionSkip locks the behaviour in. 3. Dangling agent assignee after migration 096 dropped the FK: shouldSkipDispatch now distinguishes pgx.ErrNoRows / errSquadArchived (hard skip — retrying won't help) from transient DB errors (fail-open). DeleteAgentRuntime pauses autopilots that target agents about to be hard-deleted (ListArchivedAgentIDsByRuntime + PauseAutopilotsByAgentAssignees) so the breakage surfaces as a paused row in the UI instead of a quiet skip-burning loop. Unit tests cover the sentinel unwrap contract and errSquadArchived errors.Is behaviour. Integration test TestAutopilotDispatchSkipsWhenRuntimeOffline re-verified against a fresh DB with migration 096 applied. Co-authored-by: multica-agent <[email protected]> * fix(autopilot): bump last_run_at on post-admission skip (MUL-2429) Match recordSkippedRun (pre-flight skip) and the success path so the scheduler / "last seen" UI both reflect that this tick evaluated the trigger, even when the post-admission readiness gate caught a late regression. Addresses Emacs review caveat #1 on PR #2888. Co-authored-by: multica-agent <[email protected]> * feat(autopilot): mixed agent/squad assignee picker in dialog (MUL-2429) End-to-end UI for assigning an autopilot to a squad. Closes the PR #2888 backend gap: the squad-as-assignee feature was already wired in Go (Path A, RFC §4) but the desktop dialog never offered the choice. - core/types/autopilot: add `AutopilotAssigneeType`, surface `assignee_type` on `Autopilot` + Create/Update request payloads. - views/autopilots/pickers/agent-picker: switch to a polymorphic AssigneeSelection (`{type, id}`); render agents and squads as two grouped sections with shared pinyin search. - views/autopilots/autopilot-dialog: maintain `assigneeType` state, send it on create/update, render the trigger avatar / hover dot with `assignee.type`. - views/autopilots/autopilots-page + autopilot-detail-page: render the assignee row using `autopilot.assignee_type` so squad-typed autopilots show the squad avatar + name, not a broken agent lookup. - locales: add `agents_group` / `squads_group` / `select_assignee` keys (en + zh-Hans), keep legacy `select_agent` for callers that still reference it. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Lambda <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Bohan Jiangandmultica-agent
|
63d215e1c3 |
feat(runtime): visibility (public/private) gate on CreateAgent / UpdateAgent (#2419)
* feat(runtime): visibility (public/private) gate on CreateAgent / UpdateAgent Closes the hole where a plain workspace member could pick another member's runtime in the Create Agent dialog and bind an agent to it — the backend wasn't checking runtime ownership, so the agent ran on someone else's hardware / tokens. Reported on GH #1804. Schema - Migration 083 adds agent_runtime.visibility ('private' default, 'public') with a CHECK constraint. Existing rows default to private — same ownership semantics as before, no behavior change for legacy data. Backend - canUseRuntimeForAgent predicate: allow when caller is workspace owner/admin, the runtime owner, or the runtime is public. - CreateAgent and UpdateAgent both gate on it: UpdateAgent matters because a plain member could otherwise create on their own runtime, then re-bind to a private one. - PATCH /api/runtimes/:id accepts { visibility } — owner/admin only, validated against the same private/public allow-list. Frontend - Create-agent dialog renders other-owned private runtimes disabled with a Lock badge + tooltip explaining who to ask. - Inspector runtime-picker disables the same set so re-binding fails the same way at the UI layer. - Runtime detail diagnostics gains a Visibility editor (owner/admin) or read-only chip (everyone else). - Runtime list shows a private/public chip next to the name. Tests - Go: canUseRuntimeForAgent truth table; CreateAgent / UpdateAgent end-to-end gate tests (admin / runtime owner / plain member); PATCH visibility owner / admin / member / invalid-value coverage. - Vitest: create-agent dialog disabled state on private/public runtimes, default-runtime selection skips locked rows; runtime detail visibility editor → mutation, read-only fallback. Migrating runtimes: existing rows default to private to preserve the "owner only" status quo. Owners switch to public via the detail page diagnostics card. Co-authored-by: multica-agent <[email protected]> * fix(runtime): apply timezone+visibility atomically; don't seed locked template runtime Two issues surfaced in review of MUL-2062: 1. PATCH /api/runtimes/:id ran the timezone branch first, which: - returned early on a tz no-op, silently dropping a concurrent `visibility` patch in the same body; - committed the timezone mutation (+ usage rollup rebuild) before validating visibility, so an invalid visibility left the row half-updated. Validate every field first, then run the mutations in order. The no-op short-circuit now only triggers when nothing else is requested. 2. The Create Agent dialog in duplicate mode unconditionally seeded `template.runtime_id` as the selected runtime, even when that runtime is now private and owned by someone else — the user saw a selected row they couldn't submit (Create → backend 403). Fall back to the first usable runtime when the template's runtime is locked, and gate the Create button on `selectedRuntimeLocked` as defense in depth. Tests: - Go: TestUpdateAgentRuntime_CombinedPatchAppliesBoth (tz no-op + visibility flip), TestUpdateAgentRuntime_InvalidVisibilityDoesNotMutateTimezone (atomic-fail invariant). - Vitest: duplicate template pointing at a locked runtime now seeds the first usable one; Create button stays disabled when no usable alternative exists. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> |
||
|
Bohan Jiangandmultica-agent
|
f5c2994aed |
feat(workspace): revoke a member's runtimes when they leave or are removed (#2401)
* feat(workspace): revoke a member's runtimes when they leave or are removed Previously, leaving or being removed from a workspace only deleted the member row — every runtime the departed user owned in that workspace remained in the DB, kept its daemon_token valid, and stayed reachable to the workspace's other members. The departed user lost access but their machine kept doing work. This change converges the runtime state in the same transaction as the member-row deletion: agents pinned to those runtimes are archived, in-flight tasks are cancelled (so the daemon's per-task status poller interrupts the running agent gracefully), the runtimes are forced offline, and the daemon_token rows are deleted. After commit the DaemonTokenCache is invalidated and agent:archived / daemon:register events fire so connected clients reconcile immediately. Server-side state convergence is the production safety net; the daemon_token revoke takes effect once the mdt_ flow is live (today most daemons fall back to PAT/JWT, and the member-row deletion is what stops those requests via requireWorkspaceMember). Daemon-side handling (recognising the resulting 401/404 and tearing down the local pairing for that workspace) lands in a follow-up. Co-authored-by: multica-agent <[email protected]> * fix(workspace): also cancel tasks for archived agents on member revoke CancelAgentTasksByRuntime only matched tasks whose runtime_id was in the revoked set, missing a real path: agent.runtime_id can be reassigned via UpdateAgent, but agent_task_queue.runtime_id keeps the value from when the task was queued. So an agent currently bound to the leaving member's runtime gets archived correctly, but its older tasks still pinned to a prior runtime stay 'queued' — and ClaimAgentTask does not gate on agent.archived_at, so those orphaned tasks remain claimable by the prior runtime. Replace CancelAgentTasksByRuntime with CancelAgentTasksByRuntimeOrAgent, which OR-matches runtime_ids and the archived agent IDs in one UPDATE. Pass the archived agent IDs through from revokeAndRemoveMember. Adds TestDeleteMember_CancelsTasksFromAgentReassignment as a regression guard: same agent, two runtimes, the older task on the surviving runtime must end up cancelled while the surviving runtime stays online. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> |
||
|
|
d6349c16ec |
feat(runtime): per-runtime timezone for token-usage aggregation (MUL-1950) (#2394)
* feat: per-runtime timezone for token usage aggregation The runtime token-usage charts (daily and hourly tabs on the runtime-detail page) bucketed every event by the Postgres session timezone, which is UTC in production. For an operator in UTC+8 that meant a Tuesday afternoon's tasks landed in Tuesday early-morning's bar — the chart was always one off. Fix: store an IANA timezone on agent_runtime and aggregate under it. * migrations 081 / 082 add agent_runtime.timezone (TEXT NOT NULL DEFAULT 'UTC') and rebuild the rollup pipeline (window function and both trigger functions) to compute bucket_date with AT TIME ZONE rt.timezone instead of bare DATE(). * No historical backfill — task_usage_daily rows already on disk keep their UTC bucket_date; only future writes / re-touches recompute under the new tz. (Product call from MUL-1950: 'guarantee future correctness'.) * runtime_usage.sql gains a @tz parameter on ListRuntimeUsage and GetRuntimeUsageByHour and threads tz through GetRuntimeTaskHourly Activity. ListRuntimeUsageDaily reads bucket_date as-is since the rollup already wrote it in tz. * parseSinceParamInTZ replaces the raw N×24h cutoff with start-of- day-N in the runtime's tz so 'last 7 days' lines up with bucket boundaries. * Daemon registration sends the host's IANA tz (TZ env, then time.Local), and UpsertAgentRuntime preserves any user override via a CASE-on-existing-value pattern so a daemon reconnect can't silently revert the operator's setting. * New PATCH /api/runtimes/:id endpoint (UpdateAgentRuntime) lets the runtime detail page edit the tz; the editor seeds with the browser tz on first interaction. Refs: MUL-1950 Co-authored-by: Copilot <[email protected]> Co-authored-by: multica-agent <[email protected]> * fix: harden runtime timezone rollups Co-authored-by: multica-agent <[email protected]> * fix: address runtime timezone review nits Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Eve <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: multica-agent <[email protected]> Co-authored-by: Eve <[email protected]> |
||
|
|
ce00e05169 |
Add canonical PostHog core metrics events (#2302)
* Add canonical PostHog core metrics events Co-authored-by: multica-agent <[email protected]> * Address analytics review feedback Co-authored-by: multica-agent <[email protected]> * Tighten analytics review follow-ups Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Devv <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
LinYushen
|
cc527c34be |
perf(heartbeat): batch runtime last_seen_at writes (#2213)
Batches runtime heartbeat last_seen_at updates while preserving the 60s flush / 150s sweeper stale-window invariant. Also drains pending heartbeat writes during graceful shutdown. |
||
|
Bohan Jiang
|
09f04847d3 | feat(server): redis-backed runtime liveness with DB fallback (#2121) | ||
|
LinYushenandCopilot
|
0b1333fb00 |
feat(server): orphan-task recovery + auto-retry + manual rerun (MUL-1128) (#1476)
* feat(server): orphan-task recovery + auto-retry + manual rerun (MUL-1128)
When the daemon process crashed mid-task the issue was stuck at
in_progress for up to 2.5h: the in-flight task timeout was the only
mechanism that ever moved the row, and the runtime heartbeat sweeper
only fires after the runtime stays offline for 45s — a quick restart
beats both windows.
This change implements the A+B plan from the issue thread:
A. lifecycle hygiene
- migration 055 adds attempt / max_attempts / parent_task_id /
failure_reason / last_heartbeat_at to agent_task_queue
- new daemon-auth endpoint POST /runtimes/{id}/recover-orphans:
daemon calls it on every register so the server fails any
dispatched/running tasks the previous process left behind
- new daemon-auth endpoint POST /tasks/{id}/session: persists the
agent's session_id + work_dir mid-flight so a crash doesn't
lose the resume pointer (claude+codex emit MessageStatus with
SessionID; daemon forwards on the first one it sees)
- FailAgentTask / FailStaleTasks / FailTasksForOfflineRuntimes
now set failure_reason ('agent_error' / 'timeout' /
'runtime_offline')
B. auto-retry with resume context
- TaskService.MaybeRetryFailedTask spawns a fresh queued attempt
carrying parent's session_id/work_dir when the failure reason
is infrastructure-shaped (timeout, runtime_offline,
runtime_recovery) and attempt < max_attempts; skips autopilot
- wired into the runtime sweeper paths and TaskService.FailTask
so the user transparently sees a new in_progress run instead of
a stuck row
- new user-auth POST /api/issues/{id}/rerun + multica issue rerun
CLI for the manual escape hatch
Co-authored-by: Copilot <[email protected]>
* fix(server): address PR review for orphan-task recovery (MUL-1128)
Three review-must-fix items on top of the A+B implementation:
1. recover-orphans now funnels through TaskService.HandleFailedTasks,
the same shared post-failure pipeline used by the runtime sweeper.
This guarantees task:failed events are emitted, agent status is
reconciled, and issues stuck in_progress with no remaining active
task are reset to todo even when no auto-retry is created
(max_attempts exhausted, autopilot, non-retryable reason).
2. RerunIssue now uses CancelAgentTasksByIssueAndAgent, scoped to the
issue's current assignee. The previous implementation called
CancelAgentTasksByIssue, which would collateral-cancel parallel
@-mention agents on the same issue.
3. GetLastTaskSession now considers both completed and failed tasks
(mirroring GetLastChatTaskSession), ordering by the most recent
timestamp. With UpdateAgentTaskSession pinning session_id/work_dir
mid-flight, an auto-retry or manual rerun of a daemon-crash failure
now actually resumes the prior conversation context instead of
starting fresh — matching the stated B-branch behaviour.
go build / go vet pass; the existing service and agent test suites pass.
runtime_sweeper / handler integration tests require a local DB with the
055 migration (and the pre-existing 050 first_executed_at column).
Co-authored-by: Copilot <[email protected]>
---------
Co-authored-by: Copilot <[email protected]>
|
||
|
devv-eveandJiang Bohan
|
637bdc8eb3 |
feat(analytics): full PostHog pipeline + 6 funnel events (MUL-1122) (#1367)
* feat(analytics): add PostHog client with async batch shipping Introduces server/internal/analytics, the shipping layer for the product funnel defined in docs/analytics.md. Capture is non-blocking — events are enqueued into a bounded channel and a background worker batches them to PostHog's /batch/ endpoint. A broken backend drops events rather than blocking request handlers. Local dev and self-hosted instances run a noop client until the operator sets POSTHOG_API_KEY. This is PR 1 of MUL-1122; signup and workspace_created emission land in the follow-up commit so this change is independently reviewable. * feat(server): emit signup and workspace_created analytics events Wires analytics.Client through handler.New and main, then emits the first two funnel events: - signup fires from findOrCreateUser (which now reports isNew), covering both the verification-code and Google OAuth entry points — a single emission site guarantees Google signups aren't missed. - workspace_created fires after the CreateWorkspace transaction commits, with is_first_workspace computed from a post-commit ListWorkspaces count so we can distinguish fresh-user activation from returning-user expansion. Tests use analytics.NoopClient so nothing ships from test runs. PR 1 of MUL-1122; runtime_registered and issue_executed follow in later PRs per the plan. * refactor(analytics): drop is_first_workspace from workspace_created Stamping "is this the user's first workspace?" at emit time races under concurrent CreateWorkspace requests: two transactions committing close together can both read a post-commit count greater than one and both emit false. Fixing it at the SQL layer requires a schema change we don't want in PR 1. PostHog answers the same question exactly from the event stream (funnel on "first time user does X" / cohort on $initial_event), so removing the property loses no information and makes the emit side race-free. * docs(analytics): document self-host safety defaults Spell out why self-hosted instances never ship events upstream by default (empty POSTHOG_API_KEY → noop client) and explain how operators can point at their own PostHog project without any code change. * feat(analytics): emit runtime_registered, issue_executed, team_invite_* Three server-side funnel events, all gated on first-time state transitions so retries and re-runs don't inflate the WAW buckets: - runtime_registered fires from DaemonRegister when UpsertAgentRuntime reports (xmax = 0) — i.e. the row was inserted, not updated. Heartbeats and re-registrations stay silent. - issue_executed fires from CompleteTask after an atomic UPDATE issue SET first_executed_at = now() WHERE id = $1 AND first_executed_at IS NULL flips the column for the first time. Retries, re-assignments, and comment-triggered follow-up tasks hit the WHERE clause and no-op. Carries nth_issue_for_workspace so the ≥1/≥2/≥5/≥10 buckets filter without extra queries. - team_invite_sent fires from CreateInvitation and team_invite_accepted from AcceptInvitation, closing the expansion funnel. Adds a 050 migration for issue.first_executed_at plus a partial index so the workspace-scoped executed-count query doesn't scan the never-executed tail. * feat(config): surface PostHog key via /api/config Extends AppConfig with posthog_key / posthog_host sourced from env on every request (so operators can rotate the key via secret refresh without a restart). Reading the key off the server — rather than baking it into the frontend bundle via NEXT_PUBLIC_* — means self-hosted instances inherit the blank key automatically and never ship events upstream. * feat(analytics): wire posthog-js identify + UTM capture on the client Adds @multica/core/analytics — a thin wrapper around posthog-js that owns attribution capture and identity merge. Posthog-js config comes from /api/config (not NEXT_PUBLIC_*), so self-hosted instances whose server returns an empty key automatically run the SDK inert. captureSignupSource stamps a multica_signup_source cookie with UTM params and the referrer's origin (never the full referrer — that can leak OAuth code/state in the callback URL). The backend signup event reads this cookie on new-user creation. Identity flows: - auth-initializer fires identify() right after getMe() resolves, on both cookie and token paths. A getConfig/getMe race is handled by buffering a pending identify inside the analytics module and flushing it once initAnalytics finishes. - auth store calls identify() on verifyCode / loginWithGoogle / loginWithToken and resetAnalytics() on logout so the next login merges cleanly without bleeding events. * docs(analytics): describe runtime_registered, issue_executed, invite events Fills in the schema for the remaining funnel events. Captures the design commentary that belongs next to the contract rather than in a PR description — in particular why issue_executed uses the atomic first_executed_at flip instead of counting task-terminal events, and why runtime_registered relies on xmax = 0 rather than a query-then-write. * fix(analytics): drop non-atomic nth_issue_for_workspace from issue_executed Computing the workspace's Nth-issue ordinal at emit time is not atomic under concurrent first-completions — two transactions can both run MarkIssueFirstExecuted, then both run CountExecutedIssuesInWorkspace, and both observe count=1 before either has committed, so both events go out stamped as n=1. Serialising it would mean a per-workspace advisory lock or a SERIALIZABLE-isolated tx; PostHog answers the same question exactly at query time via row_number() partitioned by workspace_id, so the emit-time property adds risk without adding information. Removes the property from analytics.IssueExecuted, deletes the unused CountExecutedIssuesInWorkspace query, and regenerates sqlc. The partial index stays — any future workspace-scoped executed-issue query will want it. * fix(analytics): wire $pageview and harden signup_source cookie payload Two frontend fixes from the PR review: - PageviewTracker, mounted under WebProviders, fires capturePageview on every Next.js App Router path / query-string change. Without this the capturePageview helper in @multica/core/analytics was never called and the acquisition funnel's / → signup step was empty. - captureSignupSource now caps each UTM / referrer value at 96 chars *before* JSON.stringify, and drops the whole cookie when the serialised payload still exceeds 512 chars. Previously the overall slice(0, 256) could leave a half-JSON string on the wire that neither the backend nor PostHog could parse. Both capturePageview and identify now buffer a single pending call when fired before initAnalytics resolves — otherwise the initial "/" pageview and same-turn login identify race the /api/config fetch and get dropped. resetAnalytics clears both buffers so a logout→login cycle stays clean. * fix(analytics): URL-decode signup_source cookie on read Go does not URL-decode Cookie.Value automatically, so the frontend's JSON-then-encodeURIComponent payload was landing in PostHog as percent-encoded garbage (%7B%22utm_source...). Unescape on read so the backend receives the original JSON string the frontend intended, and drop values that fail to decode or exceed the server-side cap — sending truncated garbage is worse than sending nothing. Oversized-cookie guard matches the frontend's SIGNUP_SOURCE_MAX_LEN. * docs(analytics): reflect nth-issue drop, $pageview wiring, cookie encoding Pulls the schema doc back in line with the code: issue_executed no longer advertises nth_issue_for_workspace (with a note about why PostHog derives it at query time instead), the frontend $pageview section names the actual PageviewTracker component that fires it, and the signup_source section documents the per-value cap / overall drop rule and the encode-on-write / decode-on-read contract. --------- Co-authored-by: Jiang Bohan <[email protected]> |
||
|
Bohan Jiang
|
a73336dcf8 |
feat(daemon): persistent UUID identity + legacy-id merge at register-time (#1220)
* feat(daemon): persistent UUID identity + legacy-id merge at register-time daemon_id is now a stable UUID persisted to `<profile-dir>/daemon.id` on first start, replacing the hostname-derived id that drifted whenever `.local` appeared/disappeared, a system was renamed, or a profile switched — each of which used to mint a fresh `agent_runtime` row and strand agents on the old one. To migrate existing installs without operator intervention, the daemon reports every legacy id it may have registered under previously (`host`, `host` with `.local` stripped, and `host[-profile]` variants for both). At register-time the server looks up each candidate row scoped to (workspace, provider), re-points its agents and tasks onto the new UUID-keyed row, records which legacy id was subsumed in the new `legacy_daemon_id` column for audit, and deletes the stale row. Result: users running `xxx.local`-keyed runtimes today transparently land on the new UUID row on next daemon restart. The hostname-prefix `MigrateAgentsToRuntime` / `daemon_id LIKE '...-%'` compatibility shim is no longer needed and has been removed along with the handler call that invoked it. * fix(daemon): handle bidirectional .local drift and case drift in legacy merge Review on #1220 flagged two gaps in the legacy-id migration candidate set: 1. Reverse .local: LegacyDaemonIDs only added the stripped variant when the current hostname ended in `.local`. The opposite direction — DB has `foo.local`, current host is `foo` — was missed, so runtimes registered under the `.local` variant stayed orphaned after upgrade. Now both variants (`foo` and `foo.local`) are always emitted, regardless of what `os.Hostname()` currently returns, plus their `-<profile>` suffix forms. 2. Case drift: os.Hostname() has been observed returning different casings on the same machine across mDNS/reboot state. A case-sensitive `=` comparison stranded rows like `Jiayuans-MacBook-Pro.local` when the daemon later reported `jiayuans-macbook-pro.local`. FindLegacyRuntimeByDaemonID now uses `LOWER(daemon_id) = LOWER(@daemon_id)` on both sides, so casing differences merge rather than orphan. The (workspace_id, provider) prefix still bounds the scan to a tiny set of rows so the non-indexed LOWER() comparison has negligible cost. Tests: TestLegacyDaemonIDs gets the mixed-case + reverse-direction cases; daemon_test.go adds TestDaemonRegister_MergesLegacyDaemonIDRuntime_ReverseDotLocal and TestDaemonRegister_MergesLegacyDaemonIDRuntime_CaseDrift. * fix(daemon): consolidate every case-duplicate legacy runtime, not just the first Follow-up review on #1220: after switching to `LOWER(daemon_id) = LOWER(@daemon_id)`, the single-row lookup still only merged one legacy row per candidate. If a machine already had two rows in the DB that differed only in casing (e.g. `Jiayuans-MacBook-Pro.local` AND `jiayuans-macbook-pro.local` coexisting because earlier hostname drift already minted a duplicate), only one of them got consolidated and the other stayed orphaned — violating the "no duplicate runtime per machine after backfill" acceptance. - FindLegacyRuntimeByDaemonID → FindLegacyRuntimesByDaemonID (:many) - mergeLegacyRuntimes iterates every returned row and dedupes across overlapping legacy candidates so `foo` and `foo.local` both resolving to the same stored row don't double-process Test: TestDaemonRegister_MergesAllCaseDuplicateLegacyRuntimes seeds two case-duplicate rows with one agent each and confirms both rows are deleted and both agents end up on the new UUID-keyed row. |
||
|
Jiayuan Zhang
|
ff5f6ac2ee |
fix(daemon): prevent duplicate runtime registration on profile switch (#906)
* fix(daemon): prevent duplicate runtime registration on profile switch The daemon_id included a profile name suffix (e.g. "hostname-staging"), so switching profiles created a new daemon_id that bypassed the UPSERT dedup constraint, leaving orphaned runtime records in the database. Three changes: - Remove profile suffix from daemon_id — use stable hostname only. The unique constraint (workspace_id, daemon_id, provider) already prevents collisions within the same workspace. - Auto-migrate agents from old offline runtimes to the newly registered runtime during DaemonRegister (same workspace/provider/owner). - Add TTL-based GC in the runtime sweeper to delete offline runtimes with no active agents after 7 days. Closes MUL-695 * fix(daemon): address code review issues on PR #906 1. Move gcRuntimes() to the main sweep loop — previously it was inside sweepStaleRuntimes() after an early return, so it only ran when new runtimes were marked stale. Now it runs every sweep cycle independently. 2. Fix DeleteStaleOfflineRuntimes to exclude runtimes with ANY agent reference (not just active ones). The FK agent.runtime_id is ON DELETE RESTRICT, so archived agents also block deletion. 3. Scope MigrateAgentsToRuntime to the same machine by matching daemon_id LIKE '<current_daemon_id>-%'. This prevents cross-machine agent migration when the same user has multiple devices. |
||
|
Bohan Jiang
|
6209e2f3ae |
fix(server): allow deleting runtimes when all bound agents are archived (#589)
Previously, runtimes could never be deleted once an agent was created because agents can only be archived (not deleted) and the count check included archived agents. Now the check only counts active agents, and archived agents are cleaned up before runtime deletion. |
||
|
LinYushenandClaude Opus 4.6
|
ff27a249cc |
feat(runtime): add owner tracking, filtering, and delete (#535)
Add owner_id to agent_runtime table to track who registered each runtime. Backend: new delete endpoint with role-based permissions (owner/admin can delete any, members only their own), list filtering by owner (?owner=me), and agent dependency check before deletion. Frontend: Mine/All filter toggle in runtime list, owner display in list items and detail view, delete button with AlertDialog confirmation. Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> |
||
|
Jiayuan
|
b112d1f1ae |
feat(tasks): add coalescing queue and task lifecycle guards
- Coalescing queue: use HasPendingTaskForIssue (queued/dispatched only) instead of HasActiveTaskForIssue so comments during a running task enqueue exactly one follow-up task that picks up all new comments. - Stale task cleanup: runtime sweeper now fails orphaned tasks when their runtime goes offline (daemon crash/network partition). - Cancel-aware daemon: handleTask checks task status after execution and discards results if the task was cancelled mid-run (e.g. reassign). - Terminal issue guard: ClaimTaskForRuntime auto-cancels pending tasks for done/cancelled issues instead of executing them. - Race condition safety net: unique partial index ensures at most one pending task per issue at the DB level. |
||
|
JiayuanandClaude Opus 4.6
|
b3bbf92a1d |
fix(runtime): add server-side sweeper to detect stale runtimes
The only path to marking a runtime offline was the daemon's deregister call on graceful shutdown. If the daemon crashed, was killed, or lost network, the status stayed "online" forever. Add a background goroutine that sweeps every 30s and marks runtimes offline after 45s without a heartbeat (3 missed intervals). Co-Authored-By: Claude Opus 4.6 <[email protected]> |
||
|
JiayuanandClaude Opus 4.6
|
38d595d81d |
feat(cli): restructure CLI commands for better UX
- Add top-level `multica login` that combines auth + workspace auto-discovery - Restructure daemon into subcommands: start, stop, status, logs - Add background daemon mode with PID management - Add daemon deregistration on shutdown (new API endpoint + SQL query) - Remove unused commands: runtime list, status, agent get/delete/stop - Make `config` show config directly instead of requiring `config show` - Update README to reflect new CLI structure Co-Authored-By: Claude Opus 4.6 <[email protected]> |
||
|
Jiayuan Zhang
|
cdfa63af15 | feat(runtime): add local codex daemon pairing |