mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-08 01:16:45 +08:00
c4f0d8cc8c
* perf(workspace): make teardown task discovery index-driven
Workspace teardown found a workspace's tasks with a three-way
`OR agent_id IN (…) OR issue_id IN (…) OR runtime_id IN (…)`, three times
per delete, and swept task_token with the same shape. PostgreSQL costs
that as a Seq Scan on agent_task_queue — the largest table in the
database (5.08M rows / 8.3 GB in production) — so deleting even a tiny
workspace paid for the whole table, inside the transaction that holds the
workspace row lock and global advisory lock 4246.
Resolve the task set one owner key at a time instead, in the handler, and
reuse that snapshot for every task-keyed step. Measured on a 1.2M-row
synthetic copy of the production shape (937 agents / 109 issues / 12
runtimes): the OR form and a 937-element `= ANY(array)` form both Seq
Scan; single-key equality is a Bitmap Index Scan. A UNION of parent-key
joins is worse than the OR (three hash joins, each seq scanning the
table), which is why that earlier suggestion was dropped.
All three ownership paths are preserved: nothing enforces that a task's
agent, runtime and issue share a workspace, so dropping a path would turn
a performance fix into a tenant-cleanup bug.
task_token needs no loop. Its workspace_id has been NOT NULL since
migration 108, so the workspace key alone covers this workspace's rows,
and the two extra OR arms only ever matched tokens owned by another
workspace whose task or agent lives here — both NOT NULL FKs with
ON DELETE CASCADE, so the later task/agent deletes remove exactly those.
Two indexes make the new shapes index-eligible:
- 273 agent_task_queue(runtime_id): every existing runtime_id index is
partial, so all-status consumers had none. This also covers the FK's
own ON DELETE CASCADE probe, which runs per deleted agent_runtime row
on the ordinary delete path.
- 274 task_token(workspace_id): the column has never been indexed.
Measured block usage for the same workspace, before → after:
agent_task_queue discovery 25,665 → 14,485 blocks; task_token sweep
4,086 → 3 blocks. The point is not the ratio at this size but that the
cost now scales with the workspace being deleted instead of with the
global table.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): fence, bound and un-cascade the teardown task sweep
Addresses all four blocking findings from the MUL-5999 review.
1. Concurrency (correctness / tenant isolation). The task set is no longer
a lock-free snapshot reused across steps. Teardown now locks every row
through which a task can belong to the workspace — its agents, issues
and runtimes — FOR UPDATE, and holds them for the rest of the
transaction. Inserting an agent_task_queue row takes FOR KEY SHARE on
each referenced owner, which conflicts with FOR UPDATE, so a concurrent
enqueue (or a reassignment onto one of these runtimes) can no longer
slip a task in behind the sweep. Locking the tasks alone could not do
that: a row that does not exist yet cannot be locked. This is the same
technique the handler already uses on the workspace row to keep
CreateChatSession out of the delete window.
The per-batch page query also takes FOR UPDATE, so under READ COMMITTED
PostgreSQL re-checks ownership after each lock: a task a concurrent
commit already moved elsewhere is skipped rather than deleted on a
stale claim, and a task we locked cannot move before we delete it.
Deadlock (40P01) joins lock_timeout (55P03) as a retryable 503, because
the owner locks can be taken in the opposite order by a concurrent task
update.
2. Cleanup ownership (no cascade reliance). task_token keeps all three
explicit paths, split into separate index-driven statements instead of
one OR: workspace_id in DeleteWorkspaceLeafData, task_id in
DeleteTaskBatch, agent_id in DeleteTaskTokensByAgent. Nothing in the
teardown graph now depends on ON DELETE CASCADE to finish the job.
3. Bounded memory. The whole-workspace task array is gone. The sweep pages
through one owner key at a time with LIMIT 1000 and deletes exactly the
page it read, so process memory, the array parameter and each DELETE's
row count are bounded by the batch size rather than by a workspace's
lifetime task count. The task-keyed arms moved out of the leaf-data
statement into DeleteTaskBatch, which runs per page, and the separate
whole-workspace task delete is gone.
4. Migration retryability. The concurrent-index cleanup hooks are now
derived from a version -> index-name map that covers 257, 261 and
273-277, so an interrupted CREATE INDEX CONCURRENTLY cannot be mistaken
for success on retry.
Three more indexes, each an unindexed FK referencing column that the
per-batch statements need and that the FK's own ON DELETE action already
scanned per row on ordinary deletes: task_token(agent_id),
chat_draft_restore(task_id), autopilot_run(task_id).
Plans re-verified on a 1.2M-row synthetic copy of the prod shape with the
child tables populated: page query is a Bitmap Index Scan under LockRows;
task_usage, chat_draft_restore, task_token and autopilot_run arms are all
index scans.
Tests: the ownership-path test now asserts the sweep is self-sufficient
inside the transaction, before any owner row is deleted, so a cascade
cannot stand in for teardown. New tests cover the enqueue / reassignment
fence (blocked writers, unrelated writer unaffected), multi-page sweeps
and the page bound, plus a migration-retry repair test and a check that
every cleanup hook names the index its migration builds.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): keyset-page the teardown sweep and fail closed on leaks
Addresses the second review round on MUL-5999.
1. Bounded scan, not just bounded result. The page query was
`owner = $1 ORDER BY id LIMIT n` against indexes that cannot produce id
order, so the executor had to read the owner's whole task set before
applying LIMIT — quadratic in pages for a busy owner, which is the case
this issue is about. Measured on a 550k-row probe: the old shape spent
11,630 buffers per page (planner chose a full pkey index scan; on other
statistics it chose an explicit Sort), the keyset shape spends ~2,000
and a later page costs the same as the first.
Pages are now `owner = $1 AND id > $cursor ORDER BY id LIMIT n` against
composite indexes: 273 becomes (runtime_id, id), 278 adds (agent_id, id),
279 adds (issue_id, id) and 280 drops the single-column issue_id index it
supersedes so the hot table does not carry both.
2. Owner enumeration is bounded too. The three LockWorkspaceTaskOwner*
statements no longer return ids at all — they are a set-based write fence
and now cost the API process nothing. Owners are walked separately in
keyset pages of 500 (281-283 add the (workspace_id, id) indexes those
pages need), so a workspace with a huge agent or issue set never has its
whole owner list held in memory.
The keyset column is id rather than a natural key on purpose: a cursor
column must be immutable, and renaming an owner mid-teardown would
otherwise let it sort behind the cursor and escape the sweep.
3. The fence is no longer the only guarantee. Blocking a concurrent enqueue
still relies on an INSERT taking FOR KEY SHARE on the owner rows it
references, which is a property of the legacy FKs rather than a protocol
every enqueue path opts into. Two things now backstop it:
- every owner is re-swept until a pass finds nothing, and teardown fails
closed (rolling the whole transaction back) if tasks keep arriving,
instead of committing a partially cleaned tenant;
- a schema-contract test asserts the three owner FKs still exist, so a
migration that removes them turns the missing fence into a red build
rather than a silent hole.
An explicit deleting-state protocol shared with all seven enqueue paths
remains the right long-term fix and is out of scope here.
End-to-end teardown, real handler, identical synthetic data, main vs this
branch (1.2M-row agent_task_queue, 200k task_token, 300k task_usage):
small workspace, 5k tasks: 29,247 ms -> 1,013 ms (4.38M -> 0.27M blocks)
busy workspace, 60k tasks: 243,863 ms -> 1,724 ms (5.76M -> 2.23M blocks)
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): fence task writes in-transaction, independent of the FKs
Addresses the third review round on MUL-5999.
1. The re-scan did not close the window, and the reviewer is right that it
cannot: a pass that comes back empty returns success, and a task
inserted after that final scan but before COMMIT is not caught by any
number of passes. The only thing that closed that window was the
FOR KEY SHARE an INSERT takes on the agent / issue / runtime rows it
references — a side effect of foreign keys that workspace_delete.sql
itself documents as an expand-phase net a later contract removes.
Migration 284 replaces it with an explicit fence: a BEFORE INSERT OR
UPDATE OF (agent_id, issue_id, runtime_id) trigger on agent_task_queue
that takes FOR KEY SHARE on the workspace row behind every owner the row
references, and raises if one is gone. That conflicts with the FOR UPDATE
LockWorkspaceForDelete already holds for the whole teardown transaction,
so a task write against a workspace being torn down blocks until teardown
finishes and then either proceeds (rolled back) or fails closed
(committed).
A trigger rather than a per-call-site check because the fence has to be
in the same transaction as the write: there are seven INSERTs and several
ownership-changing UPDATEs across agent.sql, autopilot.sql, chat.sql and
runtime.sql, several of them single autocommit statements, so a separate
"take the fence" statement from the caller would be its own transaction
and protect nothing. In-transaction by construction, covers every current
path, and cannot be forgotten by a path added later.
TestTaskWriteFence_BlocksOnWorkspaceLockAlone isolates the fence from the
FKs: the holder locks ONLY the workspace row — no agent, issue or runtime
row — so no FK on agent_task_queue has anything to block on, since none of
them references workspace. Inserts through all three ownership paths and a
reassignment all block; an unrelated workspace's insert and a status-only
update still pass. Dropping the trigger fails that test.
Cost, measured with 5,000 bare inserts: 147 ms without the fence,
238 ms with it — about 18 us per inserted row for up to three indexed
lookups and a shared row lock. It does not fire on the hot claim /
dispatch / complete path, which updates status and timestamps rather than
ownership.
The verification passes stay as a self-check, no longer described as the
concurrency guarantee.
2. Migration 280 is withdrawn. Its down path rebuilt the superseded index
with CREATE INDEX CONCURRENTLY, and the runner only runs INVALID-index
cleanup hooks in the up direction, so an interrupted rollback would leave
an invalid index that a later `down` skips — and rolling back 279 too
would then leave no usable issue_id index at all. The redundant
single-column index stays for now; dropping it needs its own migration
with a direction-aware cleanup. This also settles the stale comment in
task_usage.sql, which still names the old index and remains correct.
3. Keyset cursors no longer start from the all-zero uuid, which is itself a
valid uuid and would have made a row with that id invisible forever.
Every walk now has an explicit first-page query with no cursor predicate.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): move the task-write fence into the write statements
Addresses the fourth review round on MUL-5999.
1. The trigger is gone. Migration 284 now defines
lock_task_owner_workspaces(agent_id, issue_id, runtime_id), a predicate
that every statement writing agent_task_queue ownership calls in its own
WHERE clause:
INSERT INTO agent_task_queue (...)
SELECT ... WHERE lock_task_owner_workspaces($1, $3, $2)
Nine statements now carry it: the seven task INSERTs (agent.sql x5,
autopilot.sql, chat.sql), LinkTaskToIssue, and ReassignTasksToRuntime —
the last through a MATERIALIZED CTE so a bulk UPDATE evaluates the
VOLATILE predicate once instead of per row. Clearing runtime_id is
exempt: it cannot move a task into another workspace.
Calling it from the writing statement keeps the lock in the writer's own
transaction, which a separate caller-issued statement could not do for
the autocommit enqueue paths. TestTaskOwnershipWritesCallTheFence is the
trade for dropping the trigger: it scans the query files and fails the
build if a statement writes ownership without the fence.
2. The fence no longer defers to the foreign keys. The trigger version
counted only owner references that still resolved, so a deleted owner
made itself invisible: with all three gone it returned early, and with
one of three gone the count still matched. The predicate now compares
resolved owners against the number of owner arguments supplied and
returns false on any mismatch, so a stale write is rejected on its own
terms. TestLockTaskOwnerWorkspaces_RejectsUnresolvableOwner covers each
shape, including the one-of-three case.
3. Deterministic lock order. The workspace rows are locked with
ORDER BY w.id, so mirrored cross-workspace writes cannot take the same
pair in opposite orders. Honest note on the test: 25 mirrored concurrent
pairs never deadlock either way, because the current plan happens to
return the rows in a stable order — the ORDER BY is what makes that a
guarantee rather than an accident of the plan, and the test is a
regression test for the symptom.
4. The fail-closed test now tests what its name says. The previous one only
inserted a neighbour-owned task and could not have failed. The
replacement runs the actual race: a writer calls the real
CreateAgentTask, blocks on the workspace lock, teardown deletes the
workspace and commits, and the writer must come back with pgx.ErrNoRows
— the fence's own verdict. An FK doing the work instead surfaces as
SQLSTATE 23503, which the assertion rejects explicitly.
Cost: 5,000 predicate calls in 53.7 ms, about 11 us per fenced write, for
up to three indexed lookups and a shared row lock. Status, dispatch and
completion updates do not call it.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): stop the legacy merge when the task fence refuses
Addresses the P1 from the fifth review round on MUL-5999.
ReassignTasksToRuntime returned only a row count, so a fenced refusal was
indistinguishable from "the old runtime had no tasks". mergeLegacyRuntimes
read both as success and went on to DeleteAgentRuntime(old), where
agent_task_queue's ON DELETE CASCADE silently took the history the
reassignment exists to preserve — and once that FK goes away, the same path
would instead leave tasks pointing at a deleted runtime. Either way the
merge was leaning on the cascade, which is what this PR set out to stop.
The query now returns the fence verdict alongside the count:
SELECT (SELECT ok FROM fence) AS fence_ok,
(SELECT count(*) FROM reassigned) AS reassigned_tasks
and the merge is a single helper, mergeLegacyRuntime, that aborts with
errRuntimeMergeFenced before touching anything else when fence_ok is false.
Tasks now move FIRST, because that is the step carrying the fence: nothing
else in the merge — agent reassignment, legacy daemon_id bookkeeping, the
old runtime delete — runs unless the fence agreed. The next daemon
registration re-runs the merge, by which time the workspace is either gone
or its teardown rolled back.
TestMergeLegacyRuntime_KeepsOldRuntimeWhenFenceRefuses covers it: with a
vanished merge target the old runtime and its task both survive and the task
stays on the old runtime, while a merge onto a live runtime still completes.
Removing the abort fails that test — and fails it with SQLSTATE 23503 from
agent_runtime_id_fkey, which is precisely the "an FK is doing the work"
shape the review objected to.
Also replaced the fixed 300 ms sleep in the queued-writer race with a poll
on pg_stat_activity for a backend actually parked on a lock, so the test
cannot pass without the race being set up (it now fails on timeout instead).
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): make the legacy runtime merge a single transaction
Addresses the P1 from the sixth review round on MUL-5999.
The merge ran four autocommit statements, so the fence's workspace lock
ended with the statement that took it. Two ways that left a tenant
inconsistent while daemon registration still reported success:
* a transient failure on the agent reassignment committed the task move
and nothing else, leaving tasks on the new runtime and their agents on
the old one until some later registration happened to retry;
* in the gap after the task commit, a workspace teardown could lock and
delete the target runtime — and its UnbindTasksFromRuntime step clears
runtime_id on exactly the tasks that had just moved. The next merge
looks tasks up by the OLD runtime id, so that history was detached for
good.
mergeLegacyRuntime now opens a transaction and runs the whole merge on one
qtx: fenced task reassignment, agent reassignment, the legacy daemon_id
audit write, and the old-runtime delete, then commits. The fence stays the
first statement, so the workspace lock it takes is held until COMMIT — a
teardown cannot start deleting the target mid-merge — and the lock order
against teardown is unchanged. Any failure returns an error and rolls the
whole thing back.
The audit write is no longer best-effort: inside a transaction a failed
statement poisons the rest, so it either lands with the merge or the merge
is abandoned.
TestMergeLegacyRuntime_RollsBackWhenAStepFails injects the failure in the
database rather than through a code seam — a trigger makes the agent move
raise — and asserts the task, the agent, both runtime rows and the audit
column are all exactly as they were, then that the same merge completes
once the injection is removed. Reverting to autocommit fails it with
"task moved to <new runtime>; the task reassignment was not rolled back".
Also took the non-blocking suggestion: the queued-writer race now has the
writer report its own pg_backend_pid() over a channel and waits for that
specific backend to park on a lock, instead of matching statement text
against every backend on a shared test database.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): fence late writes against the runtime a merge is collapsing
Addresses the P1 from the seventh review round on MUL-5999, which the
reviewer reproduced rather than inferred.
The fence locked the workspace row and nothing else. A task write takes
FOR KEY SHARE there, and so did the legacy runtime merge — and two KEY SHARE
locks do not conflict. So an enqueue against the OLD runtime went straight
through after the merge had scanned its tasks, and DeleteAgentRuntime then
removed that brand-new task through agent_task_queue's ON DELETE CASCADE.
Making the merge one transaction widened the window rather than creating it:
the agent's new runtime_id is invisible until COMMIT, so concurrent writers
keep resolving the old runtime for the whole merge.
Both sides of the contract now lock the owner rows too, in one order:
workspace rows (by id), then agent, issue, runtime.
* lock_task_owner_rows (migration 284, renamed from
lock_task_owner_workspaces) takes FOR KEY SHARE on each owner row after
the workspace rows, and re-checks that they still exist. That is what
conflicts with a delete.
* The merge takes FOR KEY SHARE on the workspace and then FOR UPDATE on
both runtime rows in id order, before the task scan, and holds them to
COMMIT. FOR UPDATE is the level that conflicts with a writer's KEY
SHARE; both runtimes in one ordered statement so two merges running in
opposite directions cannot take the pair in opposite orders. Locking
fewer rows than asked for means one disappeared, and the merge is
abandoned.
A late writer therefore either commits before the merge takes the lock — in
which case the scan that follows sees its task and moves it — or waits, and
then finds the runtime gone and writes no row.
TestRuntimeMerge_LateWriteToOldRuntimeIsFencedNotCascaded drives the merge
statement by statement so it can stop where the window was (after the agent
update, before the old runtime is deleted), enqueues against the old runtime
through the generated query, and requires that the writer is parked on a
lock, that it comes back as pgx.ErrNoRows, and that the merge's own history
ends up on the target. It explicitly rejects both 23503 and a row that
lands and then vanishes.
Both halves are load-bearing: dropping the runtime lock from the fence, or
the FOR UPDATE from the merge, fails that test — the latter with "the late
write was rejected by a foreign key (23503), not by the fence".
Cost: 5,000 predicate calls in 74.4 ms, ~15 us per fenced write (was ~11 us
with the workspace lock alone).
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* docs(workspace): correct the fence comment and drop a trailing newline
Two review nits, no behaviour change.
lockWorkspaceTaskOwners still described migration 284 as a BEFORE trigger on
agent_task_queue. It became a called predicate, lock_task_owner_rows, in
9fc1d79, and this comment is the place someone reads to understand how the
fence works — so it now says what it is: a predicate every ownership-writing
statement calls in its own WHERE clause, taking FOR KEY SHARE on the
workspace rows and then the owner rows.
Also removed the blank line at EOF in the migration, which git diff --check
was flagging.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
---------
Co-authored-by: Eve <[email protected]>
Co-authored-by: multica-agent <[email protected]>
29 lines
1.6 KiB
SQL
29 lines
1.6 KiB
SQL
-- Single statement: CREATE INDEX CONCURRENTLY cannot run inside a transaction
|
|
-- or share a multi-command migration file.
|
|
--
|
|
-- agent_task_queue has never carried a plain runtime_id index. Every existing
|
|
-- one is partial — idx_agent_task_queue_runtime_pending (004),
|
|
-- idx_agent_task_queue_claim_candidates (067),
|
|
-- idx_agent_task_queue_dispatched_prepare (125) — and a partial index is only
|
|
-- usable when the planner can prove the query predicate implies the index
|
|
-- predicate. Two all-status consumers therefore had no index at all:
|
|
--
|
|
-- 1. Workspace teardown, which has to find every task belonging to a
|
|
-- workspace's runtimes regardless of status (MUL-5999).
|
|
-- 2. The runtime_id foreign key itself. PostgreSQL does not index the
|
|
-- referencing side of a FK, so `DELETE FROM agent_runtime` runs its
|
|
-- ON DELETE CASCADE probe (`WHERE runtime_id = $1`) once per deleted
|
|
-- runtime row — with no usable index that is a full scan of the largest
|
|
-- table in the database per runtime, on the ordinary delete path too.
|
|
--
|
|
-- Cheap to carry: runtime_id is set on essentially every row, so this is a
|
|
-- one-entry-per-row btree with no maintenance beyond what the FK column
|
|
-- already implies.
|
|
--
|
|
-- The trailing id column is what makes teardown's keyset paging free: the sweep
|
|
-- pages with `runtime_id = $1 AND id > $cursor ORDER BY id`, and without id in
|
|
-- the index every page would have to read the runtime's whole task set and sort
|
|
-- it before applying LIMIT.
|
|
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_agent_task_queue_runtime_id
|
|
ON agent_task_queue (runtime_id, id);
|