mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-08 09:26:50 +08:00
admin-workspace-create2
154
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
da1671ca20 |
MUL-7420: Project workflows — columns and handoffs on the workflow (#8814)
* feat(workflows): workspace workflows with step handoffs (MUL-7420)
A workflow is a named, workspace-level selection of statuses from the
shared status catalog, in board order, with an optional handoff on each
step. Projects opt in through project.workflow_id; a project without one
keeps the whole catalog (the implicit Default workflow), so existing
workspaces see no change.
- issue_workflow table + project.workflow_id, CRUD API under
/api/issue-workflows, POST /api/projects/{id}/workflow with a dry-run
status mapping plan. Only issues on statuses the target workflow does
not list are remapped, and remapping never hands issues off.
- Issue writes keep the status inside the project's workflow. Entering a
step with a handler through a status change assigns the issue to that
handler and starts its run with the step's brief as the handoff note;
an explicit assignee in the same write wins. Creating an issue starts
at the workflow's starting step.
- Task claims carry the issue's project workflow, and the brief renders
a Project Workflow section so agents move along this project's
statuses instead of the built-in ones.
- Statuses a workflow lists cannot be archived.
- Adoption is gated by the project_workflows_v1 flag; existing
workflows keep working and projects can always switch back to Default.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): workflow settings, project picker and workflow-aware boards (MUL-7420)
- Settings → Workflows (behind project_workflows_v1): the Default
workflow plus named workflows, each shown as its step chain with
handlers and the projects using it; an editor to pick statuses from the
library, order them, mark the starting step, and set each step's
handoff, instructions and next/back statuses.
- Project Workflow property: switching dry-runs first and asks where
issues on unlisted statuses go; nothing else moves and nothing is
handed off.
- A project board's columns are its workflow's steps, and each column
head names who entering it hands the issue to.
- Status pickers offer only the issue's workflow steps and say who each
hands off to; boards refuse drops onto statuses the dragged card's
workflow does not list.
- New issues in a workflow project start on its starting step, with a
hint naming who they will be handed to.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(workflows): name built-in steps once in the brief; roomier picker hints (MUL-7420)
Built-in statuses have no stored name, so the handoff brief printed
`todo` todo. Print the key alone when the name adds nothing.
The status picker widens for workflow projects and shows the handoff as
an arrow and name, so step names no longer truncate to one letter.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): handoff previews and a single timeline entry per handoff (MUL-7420)
- GET /api/issues/{id}/workflow-handoff?status= says what moving an issue
would do: who it goes to, which runs of its current agent are still
active, and the brief the handler receives. Backs the confirmation.
- POST /api/issue-workflows/preview-brief renders a step's brief from an
unsaved workflow, for the editor's preview.
- Mapping plans list the statuses that stay, so the switch dialog can say
which issues are untouched.
- A handoff is recorded on the status_changed activity (workflow, handler,
whether a run started, the brief) instead of a separate assignee row.
- The brief now reads like the one shown in the design: the workflow, the
current step, its instructions, the statuses, and the commands to hand
off.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): rebuild the workflow UI to the reviewed design (MUL-7420)
The first cut used generic controls where the design had purpose-built
ones. This brings every surface up to the mockup:
- Settings → Workflows is one page: the workflows (step chain with
handler avatars, handoff count, projects using it; Default first) and,
below, the shared status library with each status's workflow usage.
- The workflow editor is a full page instead of a dialog: draggable step
cards summarising each handoff, and a panel for the selected step —
swap its status, pick the handler with a segmented control and a
searchable target, handoff instructions, next/back statuses, a preview
of the brief the handler receives, duplicate and delete.
- The project Workflow picker lists each workflow's statuses, handoffs and
projects, links to settings, and always confirms the switch with what
moves and what stays.
- Moving an issue into a handoff step (status picker, table, board drop)
asks first: who it goes from and to, a default-on option to stop the
previous agent's run, and the brief.
- Status pickers name the workflow and project, show each step's handler
with an avatar or that the assignee stays, and explain themselves.
- Boards: the handler chip in project column heads, hatched columns a
dragged card cannot enter with the reason, and a drop hint naming who
the card will go to.
- The timeline records a handoff under its status change, with the brief
the run received; the issue sidebar shows the workflow it follows.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(workflows): word a queued previous run as queued in the handoff confirm
The confirm dialog said the previous agent was "still running (for N
minutes)" even when its task had only been queued. Queued runs now read
as queued, and the checkbox cancels them instead of stopping them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): refuse a run's status change once the issue left its step
A run decides where the issue goes next from the step it was queued for.
When someone else moved the issue while the run worked, applying that
decision undid their move: in the live Grok run, a PM parked NW-1 as
blocked and the still-running agent moved it straight back to code
review.
Tasks now record the issue's status when they are queued
(agent_task_queue.workflow_step, written by the task INSERTs; automatic
retries keep their parent's step), and a run's own status changes move
it along. In a workflow project, an agent status change from a run whose
step no longer matches the issue is refused with 409
workflow_step_moved, naming where the issue is now and who moved it, so
the agent stops instead of retrying.
A handoff away from a squad now also stops the runs made on the squad's
behalf, and stopped runs record who stopped them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(workflows): keep every status entry point inside the issue's workflow
The issue context menu listed every catalog status and applied handoff
steps without the confirmation the status picker and board show, so a
right-click could pick a status the workflow refuses or start an agent
unasked. It now offers the workflow's steps and opens the handoff
confirmation through a shared modal.
One-click status actions no longer assume built-in statuses a workflow
may not list: "mark done" lands on the workflow's done step (hidden when
it has none), un-marking a duplicate returns to the workflow's start,
and "mark duplicate" is offered only when the workflow lists cancelled.
The batch toolbar offers a single project's workflow statuses when the
selection sits in one project.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(workflows): keep issue wakeups consistent with workflow steps
Three places where project workflows and issue wakeups met:
- A wakeup run is queued when it fires, so it took the issue's status at
that moment as its workflow step. A wakeup an agent registered then
bypassed the stale-step refusal: parked by a PM, the issue could be
moved on by the agent's "check CI, then hand to review" wakeup. It now
keeps the registering run's step; a member's wakeup still starts from
the issue's status.
- Retries carried handoff_note only for wakeup runs, so a retried
workflow handoff lost its brief. Rows with a workflow step keep it.
- Switching or editing a workflow remaps statuses in bulk without the
close rule, so an issue remapped into a done status kept its wakeups
and pending wakeup runs. The remap now ends them like any status write.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(wakeups): don't also wake the agent a write hands the issue to
A workflow handoff or an assignment already starts the receiving agent's
run. If that agent also subscribed an event wakeup to the issue's status
or assignee, the same write queued a second run for one fact, and
workflows made such writes frequent.
The write now names the agent it hands the issue to in the
transaction-local multica.handed_to_agent (the new agent assignee, a
squad's leader, or a step's handler even when it already owned the
issue), and wakeup capture skips that agent's subscriptions for the
write's issue.* events. Other agents' wakeups, and task, comment and
reaction events, are captured as before.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): tell agents not to watch workflow issues with wakeups
The handoff brief and the runtime's project workflow section now say
that each step's handler gets a run when the issue enters the step, so
wakeups on a workflow issue are for waiting on time or on something
outside it, not on its status, assignee or runs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(workflows): refresh project usage after project changes
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): MUL-7420 split columns from per-project handoffs
Workflows now only define a board's columns between a fixed head
(backlog, todo) and tail (blocked, done, cancelled). Handoffs move to
project.workflow_handoffs, so projects can share columns and still hand
off to different people. Default becomes a real, editable row that every
project without a named workflow follows. Adds project handoff, add-column
(project fork or shared) and project brief preview endpoints; same-agent
adjacent columns re-run the agent.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(core): MUL-7420 project handoffs and Default workflow client model
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(views): MUL-7420 column-header handoffs and project workflow menu
Board column headers set the project's handoff for that column; the
project toolbar gets a workflow menu (full flow, switch, templates) and an
add-column action. Views outside a project show Default's columns, with
other statuses reachable through a status filter. Adds the issue flow card,
create-project workflow choice, a columns-only workflow editor and a
lighter status picker.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(views): MUL-7420 wrap create-project workflow label in a menu group
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): MUL-7420 keep workflow structure in settings; no squad handoffs
Projects no longer add columns or create workflows: the add-column
endpoint is removed and creating a workflow is admin-only. Handoffs go
to one agent, a member or the project lead; squads route work themselves,
so a handoff to one is rejected, as is the issue-creator handler.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* refactor(core): MUL-7420 drop the project add-column client
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(views): MUL-7420 simpler handoff picker and workflow editor form
The handoff form is one searchable "Hand to" picker (no handoff, the
project lead, an agent or a member); the next status defaults to the next
column and next/back overrides sit under "More options". The workflow
editor gets labelled name and description fields and shows the fixed
statuses as one start and one end row. Templates move from the project
menu to Settings → Workflows → New, and the project board loses its
add-column action.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* refactor(server): MUL-7420 keep handoffs on the workflow, make backlog and blocked optional
Handoffs move back onto workflow steps: a workflow is the whole pipeline,
and a project only picks one. The project lead stays a relative handler, so
projects can still share a workflow. Handoff edits are admin-only, through
the workflow write or PUT /api/issue-workflows/{id}/handoffs/{status}; the
brief preview moves to the workflow too. Drops the unreleased
project.workflow_handoffs migration.
Todo, done and cancelled are required; backlog and blocked are optional and
keep their fixed places. Without blocked, the brief points agents at the
step's back status.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
Co-authored-by: Cursor <[email protected]>
* feat(views): MUL-7420 edit handoffs on the workflow, optional backlog and blocked
The workflow editor sets each column's handoff in its side panel, and
Backlog and Blocked switch on and off. The board's column popover and the
flow panel edit the workflow's handoff, say how many projects it reaches,
and are read-only for non-admins. Drops the per-project handoff client
code.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(wakeups): MUL-7420 keep platform rules on writes that hand off nobody
Migration 579 skips an issue.* event for the agent a write hands the issue
to. Its filter compared agent_id with the transaction's handed_to_agent
setting using IS DISTINCT FROM, which is false when both are NULL. Main's
platform rules (child_done) have no agent_id, so every issue.* event that
handed the issue to nobody was dropped for them: a parent leaving backlog
never woke its agent for sub-issues that closed meanwhile
(TestChildDoneHeldWhileParentInBacklog).
Rules without an agent are never the handoff target, so they always
capture. Extends TestHandoffDoesNotAlsoWakeTheHandler with a platform rule
on a handoff and on a plain status change; the latter fails on the old
filter.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(views): MUL-7420 three-way handoff choice; quieter move confirm
The handoff form picks one of three directions, as in the mockup: no
handoff, hand to an agent (starts a run), or hand to a member, project lead
included (assignee only). What the step does, where it goes next and the
back status only reach an agent's brief, so they show and save for agent
handoffs alone.
The board's move confirmation keeps the agent's brief collapsed. Create
project shows the workflow picker even when Default is the only workflow,
with a note on where to make more. The hidden-columns panel says a board's
columns come from its workflow, with a link to manage workflows for admins.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(workflows): confirm handoff on project move, per-project handoff copy, two-pane editor (MUL-7420)
- A single issue moved into a workflow project from issue detail confirms
the handoff of the column it lands on; the write opts in with
enter_workflow_step. Batch, API and CLI moves stay administrative.
- The handoff preview takes project_id to preview such a move.
- Saving a shared workflow's handoff from a board can keep the change to
that project: the server copies the workflow for it and switches the
project in one transaction (copy_for_project_id).
- The column popover says which workflow it edits and who else uses it.
- The workflow editor is a step list with a detail panel for the
selected step.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* style: gofmt workflow tests
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(views): MUL-7420 show next/back always; edit workflow name and description in place
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* feat(views): MUL-7420 trim copy on the workflows settings page
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix: MUL-7420 keep pre-workflow behavior for workspaces that never adopt workflows
The run step guard now applies only to workflows that hand off, and the
standalone Statuses settings page keeps its original header.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(views): MUL-7420 use a solid placeholder tone on the workflow header
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
* fix(workflows): keep the runtime brief cache-stable; fix project board and batch status gaps (MUL-7420)
- Move the project workflow out of the runtime brief into the per-turn
prompt so CLAUDE.md stays byte-identical across steps and issues.
- An explicit status filter overrides the project workflow's columns; the
hidden-columns panel lists statuses outside the workflow.
- Batch status offers only statuses every selected workflow allows; a
partial batch rolls back, refetches and warns.
- setProjectWorkflow returns void instead of an unparsed body.
Co-authored-by: Cursor <[email protected]>
Co-authored-by: multica-agent <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: multica-agent <[email protected]>
Co-authored-by: Lambda <[email protected]>
Co-authored-by: Cursor <[email protected]>
|
||
|
Jiayuan Zhangandmultica-agent
|
231dedb552 |
MUL-8034: chore: clean historical comments across the codebase (#9274)
* chore: clean historical comments (MUL-8034) Co-authored-by: multica-agent <[email protected]> * docs: preserve pending Composio expiry contract Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> |
||
|
|
815fe37b1f |
MUL-7685: fix(agents): paginate task history and aggregate duration (#8804)
* fix(agents): paginate task history and aggregate duration * test(agents): cover pagination boundaries and query lifecycle * fix(migrations): register agent history index retry cleanup * fix(agents): show totals only for complete task history * fix(agents): average duration over the full rolling window The server bounds duration totals by completed_at > now() - 30 days, which can span 31 calendar days. deriveAgentActivity summed duration only after trimming buckets to the chart's 30 local-day slots, so runs on the oldest partial day were dropped from the average. Sum duration for every returned bucket before slotting. Co-authored-by: multica-agent <[email protected]> * fix(migrations): renumber agent history index to 552 main now carries 551_pr_merge_status, and migration numbers from 129 onward must be unique (TestMigrationNumericPrefixesAreUnique). Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
dbfc014bdd |
MUL-7610: fix(autopilot): avoid loading task history during webhook recovery (#8710)
* fix(autopilot): avoid loading task history during webhook recovery * refactor(autopilot): move webhook recovery task check into agent.sql Rename HasTasksByIssue to HasTaskForIssue and place it beside the other per-issue task existence queries. Group test imports. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
93ae552ba9 |
feat: add messages to active Claude Code and Codex runs (MUL-7565) (#8682)
* feat: add live run supplements (MUL-7565) Co-authored-by: multica-agent <[email protected]> * fix: address MUL-7565 review feedback Co-authored-by: multica-agent <[email protected]> * fix: resolve MUL-7565 supplement panel and receipt review Co-authored-by: multica-agent <[email protected]> * refactor: remove MUL-7565 supplement ordinals Co-authored-by: multica-agent <[email protected]> * fix: close supplement composer after run remount * fix: exclude task supplements from completion reconciliation * fix: build a persistent binary for local daemon launches * feat: support Claude task supplements through SDK hooks * refactor: simplify task supplement delivery * test: remove redundant task supplement coverage * fix: remove redundant supplement delivery notices * refactor: extract local daemon launcher fix to its own PR * fix: preserve ordinary comment size compatibility * fix: accept empty task start acknowledgements * fix: bound task start response decoding and retries --------- Co-authored-by: Forge-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> Co-authored-by: yushen <[email protected]> |
||
|
hutiefang76
|
1879bc35e3 |
MUL-7351: fix(daemon): fence and retry task start acknowledgements (LANX-3) (#8400)
* fix(daemon): recover quick-create writes and task start acknowledgements * fix(daemon): fence task start retries by claim generation * fix(daemon): align start claims with main and canonical UTC |
||
|
Multica Eveandmultica-agent
|
8c9f865e3e |
MUL-7480: Revert comment steering feature (#8648)
* Revert "MUL-7480: Link delivered steers to final replies (#8623)" This reverts commit |
||
|
|
c375f83c2e |
MUL-7418: add issue event and time wakeups (#8467)
* feat: add issue event and time wakeups (MUL-7418) Co-authored-by: multica-agent <[email protected]> * feat(wakeups): capture issue collaboration event lifecycle Co-authored-by: multica-agent <[email protected]> * fix(wakeups): preserve reply source when reopening threads Co-authored-by: multica-agent <[email protected]> * feat(wakeups): summarize sidebar and issue activity cues Co-authored-by: multica-agent <[email protected]> * fix(wakeups): restore disabled configurations with a two-way toggle Co-authored-by: multica-agent <[email protected]> * feat(wakeups): manage workspace wakeups in Autopilot Co-authored-by: multica-agent <[email protected]> * chore(wakeups): reserve migrations after current main Co-authored-by: multica-agent <[email protected]> * test(wakeups): align inventory fixtures with shared contracts Co-authored-by: multica-agent <[email protected]> * fix(wakeups): distinguish trigger conditions and execution results Co-authored-by: multica-agent <[email protected]> * fix(wakeups): suppress registration feedback and drain bounded evidence Co-authored-by: multica-agent <[email protected]> * fix(wakeups): align sidebar actions with titles Co-authored-by: multica-agent <[email protected]> * fix(wakeups): bound database work and unify shared issue visibility Co-authored-by: multica-agent <[email protected]> * feat(wakeups): filter mutation events by member or agent actor Co-authored-by: multica-agent <[email protected]> * refactor(wakeups): merge structured evidence and narrow receipt locks Co-authored-by: multica-agent <[email protected]> * feat(wakeups): let people edit wakeup instructions Co-authored-by: multica-agent <[email protected]> * fix(wakeups): align inventory controls and compact prompt editing Co-authored-by: multica-agent <[email protected]> * fix(mobile): handle deferred agent run status Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Lambda <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
8ce795b007 |
MUL-7480: Steer active agent turns from comments (#8605)
* feat: steer active agent turns from comments (MUL-7480) Co-authored-by: multica-agent <[email protected]> * fix: make comment steering hint driven Co-authored-by: multica-agent <[email protected]> * fix: harden active comment steering races Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Forge-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Bohan Jiang
|
d7009df8cc |
Revert "MUL-7300 fix(issues): preserve Quick Create original input (#8519)" (#8546)
This reverts commit
|
||
|
|
a6472044c3 |
MUL-7300 fix(issues): preserve Quick Create original input (#8519)
* fix(issues): preserve quick-create original input Co-authored-by: multica-agent <[email protected]> * fix(issues): validate quick-create origins Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
78a5f69ae1 |
MUL-6813: fix: surface private runtime owner mismatch as explicit failure (#7805)
* fix: fail private runtime owner mismatches (PUCK-89) Co-authored-by: multica-agent <[email protected]> * fix: keep runtime mismatch repair off idle claim path * test: cover mixed runtime claim outcomes (PUCK-89) Co-authored-by: multica-agent <[email protected]> * fix: settle runtime owner mismatches during claim * fix: reject ownerless agents on private runtime claims * chore: restore claim version sampling layout (PUCK-89) Co-authored-by: multica-agent <[email protected]> * fix: re-authorize delivery at the finalize boundary, dedicated runtime_access_denied client copy, fixture convention (PUCK-89) Blocker 1 — final delivery gate: FinalizeTaskClaim now runs a caller- supplied authorize closure inside its transaction. The singular and batch claim paths share finalizeClaimDelivery, which re-locks the runtime row (FOR UPDATE), re-reads the agent, and re-verifies the private-runtime owner fence against CURRENT ownership before the task token commits. A concurrent re-registration that would change owner_id blocks until the gate commits, closing the stale-snapshot TOCTOU. Mismatched tasks settle through the existing failClaimedTaskBeforeLaunch -> FailTask path; singular keeps 200 {"task":null}, batch skips the task and keeps returning valid ones. Regressions cover both paths. Blocker 2 — client copy: runtime_access_denied gets dedicated actionable copy (make runtime public / rebind-copy agent) on the issues blocked-trigger mapping, both chat send-failure toasts, the chat failure-reason map, the mobile dispatch-reason helper, and the autopilot run-now toast. Generic fallbacks unchanged; locale keys added for en/ko/ja/zh-Hans. Blocker 3 — fixture convention: runtime_access_denied_test.go now uses testutil.Call; daemon_runtime_access_test.go seeds its chat task via the dbfx.Task fixture. Test semantics unchanged. Co-authored-by: multica-agent <[email protected]> * fix: bind final task token to locked runtime owner Co-authored-by: multica-agent <[email protected]> * fix: keep delivered user context on current owner Co-authored-by: multica-agent <[email protected]> * test: use testutil.Call in settlement-failure claim regression Convert the last manual httptest.NewRecorder flow in TestFinalizeClaimDelivery_SettlementFailureIsUnsettled to the repository's required testutil.Call helper (hard convention from review) and drop the now-unused net/http/httptest import. No production code changes. Co-authored-by: multica-agent <[email protected]> * PUCK-132: settle queued private-runtime owner mismatches as runtime_access_denied Align persisted task failure semantics with admission semantics: queued private-runtime ownership mismatches now settle as runtime_access_denied, letting clients reach the dedicated recovery copy instead of the generic invalid_task_identity copy. Actual task/agent identity violations (agent rebound, agent deleted, response identity mismatch, error_agent_runtime_changed) keep invalid_task_identity. - add taskfailure.ReasonRuntimeAccessDenied (permanent, non-retryable ownership authorization failure; agent process never launched) - swap the reason in the two owner-mismatch settlement paths in handler/daemon.go (response-assembly recheck + delivery-gate authz-error default branch, including ownerless-runtime denial) - regression A (queued issue task), B (queued chat task + assistant failure message via existing FailTask path), C (agent rebind keeps invalid_task_identity) Co-authored-by: multica-agent <[email protected]> * PUCK-132: map persisted runtime access failures in clients * fix(i18n): add missing fr translations for runtime_access_denied keys Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> Co-authored-by: worker-opencode <[email protected]> Co-authored-by: puck-181 <puck-181@local> |
||
|
Bohan Jiang
|
b425073cae |
MUL-7409 fix(runtime): stop steering stale-instance cleanup at the shared profile (#8483)
Deleting a profile-backed runtime instance is refused while its profile exists, and the refusal said "delete its runtime profile instead". For the case that produces it most often — a retired machine's leftover row in a profile other, healthy machines still use — that points at a workspace-wide delete which takes those machines' runtimes too, and which a bound agent refuses anyway with a second message naming no agent and no machine. Retention GC already deletes these rows after 7 days offline, ignoring profile_id, so the refusals now describe that instead of recommending the destructive route: - Instance refusal is status-aware, and only promises automatic cleanup when it matches the gate gcRuntime actually applies: the runtime's own undrained tasks plus those owned by every user agent bound to it, archived included. - Profile refusal names the blocking agents and the machine each sits on, with a per-class remedy — ordinary agents can be reassigned or archived; Mika cannot be archived but can be rebound; a Builder session is reachable only by its creator. Those clauses come from counts over the whole blocker set, so a class outside the 20-row sample is still reported. - Response bounded at 20 entries with an exact total, capping rows read inside the locked delete transaction. - CLI surfaces the server's sentence instead of dumping the raw JSON body. When a delete is allowed or refused is unchanged; the guard predicates and the instance-refusal error code are untouched. The remaining capability gap — retiring an offline instance on demand — is tracked separately. Fixes MUL-7409 |
||
|
|
2b42c8e3f0 |
perf(server): slim daemon task status lookup (#8484)
Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
f39debf8d1 |
MUL-7344: skip the issue read and the comment scan when the server already answered them (#8377)
* feat(daemon): read the comment delta with one --since, not a roots scan (MUL-7344) A resumed follow-up turn whose claim carried a server-computed issue-wide comment delta was still told to run the workflow's two-step scan: a `--roots-only --summary` pass to discover which threads moved, then a `--thread ... --since ...` expansion per thread. That rediscovers, in two or more calls, an answer the server already has. `--since <anchor>` without `--thread` IS that answer. ListCommentsSinceForIssue returns every comment on the issue created after the anchor, in every thread and at every nesting level, with no folding, on the same `created_at > since` predicate CountNewCommentsSince counted with. It is a strict superset of what the roots scan could surface: roots-only `last_activity_at` is MAX(created_at) over a thread, so any comment able to move a thread's activity is already in the `--since` result. So BuildNewCommentsHint now hands over exactly one wide read and says it answers step 2's scan — the same rule BuildResumedCommentsHint already applies to a server-computed EMPTY delta. The optional second command stays a THREAD read on `--tail 30` for the reply itself; it is never the wide read, and it never combines `--thread` with `--since`, which drops the thread root. Brief step 2 and the multica-platform skill's bounded-reads rule both gain the matching static sentence, so an agent is not left choosing between the hint and two surfaces that contradict it. Cold start, dropped resume and unknown-delta paths are untouched, byte for byte: only an affirmative server report may stand in for the scan. Co-authored-by: multica-agent <[email protected]> * feat(claim): compare the issue against the last run so a resumed turn can skip the read (MUL-7344) A follow-up turn that resumes its provider session already holds the issue it read minutes ago, and was still told, unconditionally, to run `multica issue get` before doing anything. On an issue that did not move, that read returns bytes the session already has. The claim now answers the question instead. Each claim records the comparable state of the issue it built its payload from — status, assignee, priority, and sha256 of title and description — in a new `agent_task_queue.issue_snapshot` JSONB column, and compares against the snapshot the SAME agent's previous run on the SAME issue recorded. The response carries `issue_state_delta_known`, `issue_changed_fields`, and the current `issue_status` / assignee. Why a per-claim snapshot and not a `content_revision` on `issue`: a revision column has to be bumped by every writer that touches one of those five fields, and the first one missed reports "unchanged" forever with no symptom. A snapshot taken at claim time has exactly one writer. It also rides the anchor the comment delta already uses (agent_id, issue_id, started_at DESC), so both deltas on a claim mean "since your last run" in the same sense. The daemon renders one of three things, gated on a REAL resume: - unchanged → reported as workflow step 1's answer; the read survives only as "if resumed memory is not enough". - changed → the changed field NAMES plus the read, so the agent knows what to look for instead of diffing blind. - otherwise → the pre-existing instruction, byte for byte. `issue_state_delta_known` carries the same contract as `new_comments_delta_known` (MUL-6984), and for the same reason: a cold start, a missing prior snapshot, a read error, a foreign shape version and an old server all produce an empty changed-field list, identical to a genuine "nothing changed". Only the flag separates them, and only the affirmative one may stand in for the read. The compared set is named in full in the prompt rather than summarised as "the issue": labels, parent, due date, stage, project and metadata are NOT compared, and an agent told "unchanged" has to be able to see that. The snapshot write is NOT gated on the task being comment-backed — an assignment run that recorded nothing would leave the next run with no baseline. It shares the receipt's CAS, so a stale handler cannot write over a newer reclaim or after execution starts, and a snapshot that fails to encode is simply not recorded: the next run reports "not compared" and reads the issue. Co-authored-by: multica-agent <[email protected]> * fix(claim): renumber the migration past main and fold in review nits (MUL-7344) CI caught a real collision: main landed 471_comment_deleted_at through 474 while this branch was open, so 471_agent_task_issue_snapshot shared a numeric prefix and TestMigrationNumericPrefixesAreUnique failed. Renumbered to 475 after merging main; no other migration references it. Review nits from the PR review, all non-blocking, all cheap to take now: 1. Brief step 2's closing sentence told the agent that a resumed run's scan surfaces moved threads via `last_activity_at`. That is now true only on the path where the delta is UNKNOWN — when the server hands over a `--since` delta there is no scan to read it from. Rewritten to attach the `last_activity_at` guidance to the scan itself, wherever the scan happens. Done in this PR rather than deferred because this PR already moves brief bytes, so it costs no extra prompt-cache invalidation. 2. The two per-claim anchor reads are now one. GetLastTaskStartedAtForIssueAndAgent and GetLastTaskIssueSnapshotForIssueAndAgent selected the same row with the same WHERE and ORDER, differing only in projection; they are replaced by GetLastRunAnchorForIssueAndAgent returning started_at and issue_snapshot together. A comment-triggered claim is back to ONE anchor read, as before this branch. The saved round trip is the smaller half. The real gain is that "since your last run" is now a single resolved fact on a claim instead of two independently-resolved ones that could disagree. The shared read also shares a failure mode, and both consumers already degrade the same safe way: comment scan required, issue comparison unknown. 3. The hint's `%d new comment(s)` excludes the injected trigger and the agent's own comments; the `--since` read it hands over has no such filter, so the row count is legitimately higher. Narrowing the read to match would mean a second count-shaped query for no gain, so the text now states the mismatch and its direction outright instead of leaving the agent to reconcile two numbers. New test pins the merge: one claim, one prior run, both deltas resolved from that single anchor row. Co-authored-by: multica-agent <[email protected]> * fix(migrations): renumber the issue-snapshot migration to 478 (MUL-7344) main landed 475_issue_status_key_not_reserved through 477 while the previous renumber was in flight, so 475_agent_task_issue_snapshot collided in turn. Merged main again and moved to 478. TestMigrationNumericPrefixesAreUnique passes, and the full ledger applies cleanly from scratch through 478. Co-authored-by: multica-agent <[email protected]> * fix(claim): measure both deltas from the run whose session is resumed (MUL-7344) Review found a real correctness bug, not a performance one: the claim could tell a resumed agent "the issue is unchanged" while the session it handed back predated the change. Both deltas were anchored on the most recently STARTED run for (agent, issue). The session the daemon resumes is chosen by a different rule, and the two are not the same row: - GetLastTaskSession skips poisoned and retired sessions, so a newer run that died on an unprocessable conversation is passed over for an OLDER healthy one. - A manual rerun resumes the exact source task the operator clicked. So: an old session remembers description A, the user edits it to B, a newer run records B and then dies unresumable. The next follow-up resumes the OLD session but compares B against B, reports "unchanged", and the agent keeps working from a context that still says A. PriorSessionResumeUnavailable does not catch this — it flags a missing Codex rollout, which is only one of the ways the resumed session can be older than the newest run. The fix binds both deltas to the run actually being resumed. GetLastTaskSession now returns that row's started_at and issue_snapshot alongside its session, the rerun path takes them from its source task, and the delta computation moves after resume resolution so it can only ever read the adopted row. Where no session is adopted — no prior session, a session on another runtime, a forced-fresh retry, a rerun whose source failure poisoned the conversation — no delta is reported and the daemon performs the reads it always has. That makes the standalone anchor query unnecessary: the anchor is now a projection on a row the claim already fetched, so a claim reads no extra row at all (GetLastRunAnchorForIssueAndAgent is removed). The precondition is now visible in the fixtures: a prior run has to be resumable for a delta to exist, so the comment-delta tests seed a session_id. The regression test covers both paths and was verified to fail against the previous anchor — both sub-cases report an empty changed-field list, which is exactly the wrong "unchanged". Also renumbers the migration 478 → 491; main landed 478_issue_status_category_expand through 490 while the branch was open. Co-authored-by: multica-agent <[email protected]> * fix(claim): a never-started anchor row may date neither delta (MUL-7344) Final review found the last corner of the resumed-run bug. Binding the anchor to the resumed SESSION was necessary but not sufficient: GetLastTaskSession returns that session's latest TERMINAL row, and that row is not always a run that ran. A retry child can inherit the session, be claimed — which writes its snapshot — and then fail before the agent ever starts: runtime offline during prepare, cancelled while dispatched. Its snapshot describes the issue as of ITS claim, while the session's memory stops at the last run that actually executed. If the description was edited in between, the current issue matches that snapshot and the claim reports "unchanged" to an agent whose context predates the edit. The started_at gate now covers both deltas rather than only the comment one. The comment delta was already immune because it read StartedAt directly; the issue delta read only the snapshot, so it had no such guard. Hoisting the check makes the row's requirement explicit — a row that never ran dates nothing — and the inner duplicate is dropped so no reader has to wonder what each guards. Regression test reproduces the exact fixture and was verified to fail without the gate: known=true with an empty changed-field list, i.e. the false "unchanged". Also from review: - Adopt the no-adopted-session table (missing session, session on another runtime, force-fresh retry, poisoned rerun source). Each seeds a usable snapshot AND an unread comment, so a regression has something to report and the assertion cannot pass on an empty fixture. - The resumed-run regression test asserted a comment anchor that never executed: both sub-cases created only the trigger comment, which the count excludes, so NewCommentsSince was empty and the branch was dead. A comment between the two runs now makes it run and pins known/count/since. - Rewrite the migration header, which still described the old "(agent_id, issue_id, started_at DESC)" anchor and an index rationale for a column that is never a lookup key. Co-authored-by: multica-agent <[email protected]> * refactor(claim): compare title, description and status only (MUL-7344) The snapshot compared five fields. The comparison answers exactly one question — must the agent run `issue get` again? — so a field earns a place only if changing it alters what the agent does AND its current value is not already in the per-turn message. Assignee fails the second test: the claim ships the current assignee outright, and "is this mine now" is answered by that value, not by whether it moved. Priority fails the first: it is reachable only by reading the issue, but it does not change what the agent does. Both are dropped. Status stays even though it is also shipped as a current value, so comparing it is strictly redundant for "what is it now". It earns its place as a signal rather than a value: "changed: status" is the clearest evidence that somebody intervened between the runs — a push back from in_review to todo means the delivery was rejected — and it costs nothing to carry. Narrowing the set means a narrower claim, so the prompt now says "compared title, description and status". An issue whose only change is assignee or priority is reported as unchanged, which is correct and now pinned by a test that also asserts the current assignee still ships — without that second assertion the fixture's UPDATE could silently not apply and the test would pass on nothing. Snapshot version goes to 2, because the rule is that the version tracks the compared SET. No deployment has written a v1 row, but the gate is what makes narrowing safe in general: a v1 row read by v2 code is "not compared", which degrades to the unconditional read. Test fixtures now build snapshots at issueSnapshotVersion rather than a literal 1, so a future bump cannot quietly turn them into not-compared tests while they keep passing. The foreign-version case stays explicit at +1. Also renumbers the migration 491 -> 495; main landed 491 through 494 while the branch was open. Fifth renumber. Co-authored-by: multica-agent <[email protected]> * refactor(claim): keep the snapshot at v1 (MUL-7344) The previous commit bumped issueSnapshotVersion to 2 when it narrowed the compared set. That was the version rule applied past its purpose. The rule — "bump when the compared SET changes" — exists to stop a stored snapshot from being compared against a differently-shaped one. It protects DEPLOYED rows. This column is introduced by this same unmerged PR: the migration exists only on this branch, main has neither the column nor the writer, and no environment has ever stored a row with the wider set. There is nothing for the bump to protect, and shipping v2 as the first version the column has ever held would leave a reader hunting for a v1 that never existed. Back to 1. The `v` field and its unknown-version gate stay exactly as they were — those are what make a real narrowing safe later. The comment now says which version this is and why the review-time narrowing did not earn a bump, so the rule still has a stated precedent without a burned number. No behaviour change: the compared set is still title, description and status, and fixtures build at issueSnapshotVersion rather than a literal, so they followed the constant back down. Co-authored-by: multica-agent <[email protected]> * fix(claim): only a completed run may date a delta (MUL-7344) Final review found a window one step later than the never-started case. A snapshot is written at CLAIM time. The daemon then writes started_at (TaskService.StartTask) and only afterwards launches the provider. So a run can carry both a snapshot and a started_at while its prompt never reached the session: the daemon exits during prepare, the runtime drops, orphan recovery marks the row failed. Anchoring there compares the current issue against a snapshot describing an edit that session never saw — reporting "unchanged" across a real change, and dating the comment delta late enough to hide every comment older than that row's start. Reproduced on the real CreateRetryTask -> claim -> StartTask -> RecoverOrphanedTasksForRuntime -> follow-up claim path. Nothing on the row separates "failed after the provider ran" from "failed before it ran". session_id is inherited by CreateRetryTask at insert, so it does not mean the session was fed; started_at precedes the launch, so it does not mean the turn began. The only terminal status that PROVES delivery is 'completed' — a run cannot finish a turn it never started. So failed and cancelled sources now date neither delta. Their SESSION is still resumed exactly as before, and GetLastTaskSession still accepts them on purpose; those runs simply perform the context reads they always did. That is the cheap side of the trade: a follow-up after a failure or a stop is already recovering from something, while the case the optimization exists for — an ordinary follow-up after a healthy run — is untouched. Both conditions now live in newResumedRunAnchor rather than at the use site, so "which run may date a delta" is answered once, and GetLastTaskSession carries the status needed to answer it. Regression test is the reviewer's repro, verified to fail without the rule: known=true with an empty changed list AND a computed zero comment count, i.e. both a false "unchanged" and a false "no new comments". Also: - Renumber the migration 495 -> 499; main landed 495 through 498. Sixth renumber. - The migration's content_revision rationale named the old five-field set while arguing about hypothetical issue-row writers; it now refers to "any compared field", which is what the argument actually depends on. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
d95c20aab2 |
feat: complete issue status category upgrade (MUL-7365 PR3) (#8466)
* feat: converge issue status categories on upgrade (MUL-7365) Co-authored-by: multica-agent <[email protected]> * refactor(status): remove redundant stored-category normalization (MUL-7365) Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Lambda <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
7ebe0bf58d |
MUL-7213: feat(server): Triage has no executor (#8373)
* feat(server): start no run for an issue in triage (MUL-7213) An issue waiting in Triage produces no execution run, and the triage run itself leaves no trace an execution run can inherit. The rule cannot be enforced at the call sites. Five paths enqueue without looking at status at all (comment and mention dispatch, quick action, manual rerun, auto-retry), and dispatchIssueRun discards the enqueue error entirely, so a missed call site would fail silently. So the check sits at the queue door — immediately before every INSERT of an issue-linked agent_task_queue row — and the upstream short-circuits exist only so that previews and dispatch reasons stay honest about what the door will do. guardIssueNotInTriage covers all five insert sites: both enqueue funnels, FailTask's in-transaction retry, the orphan sweeper's retry, and the delegated-failure recovery dispatch. Two future paths are meant to be exempt and both are Triage's own — the triage run, and accept, which leaves Triage in the same transaction that enqueues the execution run. Upstream, WillEnqueueRun, maybeEnqueueOnAssign, the two shouldEnqueue*OnAssign gates and the comment trigger router all refuse Triage; rerun and quick action refuse with 403 issue_in_triage, a new dispatch reason code. WillEnqueueRun also treats leaving Triage as a creation rather than a status change: accept is when the issue becomes work, and the backlog-to-elsewhere rule would refuse it outright. Two narrower rules follow. retryEligible excludes triage runs for the reason it excludes autopilot runs: re-triaging reads the entry as it stands now, so replaying a failed attempt is never the right recovery. And GetLastTaskSession excludes them from all three CTEs, so the first execution run after accept does not resume the conversation in which the agent was still deciding whether the entry was worth keeping. Refusals are deliberately quiet where a caller cannot act on them. The FailTask guard never returns — that transaction also carries the parent's failed status, and losing it would leave the task stuck in 'running'. Verified: internal/handler, internal/service and cmd/server pass; every new test was confirmed to fail with the change neutralized. Co-authored-by: multica-agent <[email protected]> * fix(server): close two triage session bypasses around rerun (MUL-7213) Review found that session isolation had two holes the query-level exclusion could not reach, and that the two quietest queue-door paths had no direct coverage. A manual rerun resolves its session from the task the user named, via rerun_of_task_id, not from GetLastTaskSession — so after accept, naming the historical triage task reran the triager and resumed the triage conversation, walking straight around the exclusion. RerunIssue now refuses a triage source: redoing triage is re-triage, a different action on a different status. rerunSourceMatchesTaskScope refuses it too, which is what still holds for a rerun row an older server wrote mid-deploy. GetLatestTaskRolloutMissing reports only whether GetLastTaskSession fell back, so the two have to exclude the same rows. It did not, so a terminal triage run with session_rollout_missing told the first execution run after accept that it had lost the previous turn's context — about a turn it was never entitled to. Tests now cover the two doors that refuse silently: FailTask's in-transaction retry, which must leave the parent failed and add no child, and dispatchDelegatedFailureRecovery, which builds its coordinator task with a direct CreateAgentTask call. Both assert against a positive control so a zero cannot come from an inert fixture; the recovery test also shows the obligation is deferred rather than dropped — the same pending comment wakes the coordinator once the issue is accepted. Verified: internal/handler, internal/service and cmd/server pass; all five new tests were confirmed to fail with the fixes neutralized. Co-authored-by: multica-agent <[email protected]> * refactor(server): let a named agent talk on a Triage issue (MUL-7213) The rule shipped as "an issue in Triage starts no run", which is a mechanical restatement of the product red line rather than the line itself, and it was too wide: @-mentioning an agent to discuss a Triage entry was refused along with everything else. What Triage actually lacks is an EXECUTOR. Nothing may derive one from the issue — the assignee, a prefilled owner, a squad leader, an automatic retry — because that is exactly what the triager is still deciding, and because prefilling an owner is how a run starts today, the triager would otherwise dispatch the very owner it was only proposing. A person naming an agent by hand is a different act: no triager action produces it and the cost is one the person chose. That is the same shape as backlog, which parks assignment but lets comments through, one notch tighter because in Triage the assignee is only a proposal. So the queue door now judges the run's origin rather than the funnel it came through. OriginDerived is the zero value, so a caller that says nothing still gets the strict answer and a new enqueue path cannot leak in by omission; the two mixed wrappers take it as a parameter, which made the compiler enumerate every call site. Comment routing loses its blanket short-circuit and instead refuses the two routes that derive an executor — the assignee fallback and the squad leader fallback. Explicit mentions, replies to an agent, and thread continuation route as they do anywhere else. Rerun now follows its source: a discussion run may be repeated, naming no source means "run the assignee again" and is refused, and the triage run itself keeps its own reason. Assignment, quick action, auto-retry and all three session isolations are unchanged. The delegated-failure door is gone because main's #8406 gate subsumes it: Triage is its own lifecycle category, so it is neither unstarted nor started and both that gate and the pending-recovery query already exclude it. A test pins that. Verified: internal/handler, internal/service and cmd/server pass; both halves of the new rule were confirmed to fail when neutralized in place. Co-authored-by: multica-agent <[email protected]> * refactor(server): hold Triage in its own column, not in status (MUL-7213) `status` answers "how far along is this work". Triage answers "is this work at all". Holding the second question inside the first field makes every reader of `status` re-answer it, which is where the scattered `<> 'triage'` checks came from — main carried 9 of them and this branch had grown to 23. A column of its own asks it once. issue.triage_state is NULL for an ordinary issue and `pending` for one waiting in Triage. Adding a nullable column with no default is a catalog-only change, so no table rewrite and no backfill; the queue reads a partial index sized to itself rather than to the issue table. What follows from the move: - issuestatus goes back to 7 keys and 4 categories. Triage is no longer a category, a behavior, or something ExpandCategories can name, so CategoryForBehavior, BehaviorsForCategory, ParseCategory and ExpandCategories lose their special cases, and the dead board-order arm and status label go with them. - WillEnqueueRun loses `acceptedFromTriage`. Accept clears a column and then takes the ordinary create / assign path, so there is no "was in triage" transition left for the predicate to recognise. - The write lock disappears rather than narrowing. Status, project and parent on a Triage entry are the triager's proposal and accept is what confirms them, so ordinary writes reach them; the marker itself is not a field of the issue API, so there is nothing left to lock. - The duplicate guards carry `triage_state IS NULL` instead of a status key, and the delegated-failure recovery predicate gains the same, in both its SQL and its Go mirror. What does NOT follow, deliberately: the reserved NAME stays. Migration 475's CHECK is deployed and 476's renames are irreversible, so `triage` remains unavailable as a custom status key and Resolve keeps refusing it — one reserved word rather than five special cases. Read-path exclusion is a separate sub-issue and is not attempted here. Also from review: the quick-action refusal is now documented as a first-phase product choice rather than a consequence of the rule (its target is configured, not derived), and ErrIssueInTriage no longer claims that nothing runs. Verified: handler, service, issuestatus, cmd/migrate and cmd/server pass on a database migrated from 001; every column read was confirmed to turn its own tests red when neutralized. cmd/migrate's concurrent-index convention test caught a missing cleanup hook for the new index. Co-authored-by: multica-agent <[email protected]> * fix(server): hold the parent of a Triage entry (MUL-7213) Removing the write lock went one field too far. Status, project, assignee and the rest on a Triage entry are the triager's proposal and accept confirms them, so those writes belong. The parent is not a proposal: - the routing panel has no parent field. "Merge into an existing issue" links rather than re-parents, so a Triage entry hanging off a parent is a half-state the product never asks for; - a child in Triage carries a proposed status, so it is never terminal. stageBarrierClosed would wait on it forever — holding the whole sibling set, or one stage's frontier — and ChildIssueProgress would count it in the denominator. Both reach it through parent_issue_id, so refusing the write is what keeps them out of reach. That is behavior, not display, and it is why this one is worth a lock rather than a read-path filter. UpdateIssue and batch refuse a request that carries parent_issue_id for an issue in Triage, with or without a value: null clears, which is equally a parent decision the triager does not get to make. The batch asks one count rather than listing statuses, since "any" is all it needs. Verified: handler, service, issuestatus, issueguard, cmd/migrate and cmd/server pass; both new tests were confirmed to fail with the guard neutralized. Co-authored-by: multica-agent <[email protected]> * fix(server): drop the triage case from the handoff predicate (MUL-7213) MUL-7379 (#8435) landed on main while this branch was open and added issueStatusIsHandoff with a `triage` case, resting on Triage resolving to a category of its own without a catalog read. Once Triage stops being a status that resolution is gone, so the call errors on a value no issue can hold any more. The case is removed rather than repaired: a Triage entry carries whatever status the triager has proposed and is answered on that. Suppressing notifications for Triage entries is a different rule and belongs to MUL-7219 — the status model was giving it incidentally, and that was never the mechanism the design asks for. Verified on a database migrated from 001: cmd/server, internal/handler, internal/service, internal/issuestatus and cmd/migrate pass. Co-authored-by: multica-agent <[email protected]> * chore(db): renumber the triage migrations past the telemetry batch (MUL-7213) MUL-7236 (#8252) took 479-482 on main while this branch was open, so both triage migrations collided. 483 adds issue.triage_state, 484 builds its partial index; the concurrent-index cleanup hook moves with it. Verified on a database migrated from 001 with the new numbering. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
f639d35f44 |
fix(agents): MUL-7383 exclude cancelled runs from success metrics (#8424)
* fix(agents): exclude cancelled runs from success metrics * refactor(agents): require activity outcome counts from the back-end The previous commit carried a fallback for a new client talking to an older back-end: optional `completed_count` / `cancelled_count`, null propagation through the bucket series, and a "—" success rate when the counts were absent. Multica only supports the other direction — an older daemon or client against a newer back-end — so that branch is an unrequested legacy adapter. Outcome counts are now required at the schema boundary, so the bucket series and window summary carry plain numbers. A response without them is contract drift and `parseWithFallback` degrades the whole list to `[]`, the same as every other list endpoint. Unchanged: a window holding nothing but cancellations still has no success rate and renders "—" rather than claiming 100%, and the sparkline still draws cancellations neutral. Both are the point of the fix, not compatibility. Copy for the unavailable state now says the period has no completed or failed run, which is the only way it is reachable. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
dddbb03363 |
fix(tasks): prevent delegated failure recovery starvation (#8406)
Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
9d18186e6e |
fix(status): MUL-7365 expand category compatibility before backfill (#8407)
* fix(status): stage category expansion and preserve installed client groups (MUL-7365) Co-authored-by: multica-agent <[email protected]> * fix(status): validate mixed storage with GC and recovery compatibility Co-authored-by: multica-agent <[email protected]> * docs(status): align phase one rollout with merged main dependency Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Lambda <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
1b3137a00d |
fix(server): MUL-7364 decide issue GC and recovery prefilter on lifecycle category (#8405)
* fix(daemon): decide issue GC on lifecycle category, not the status key (MUL-7364) MUL-7240 collapsed issue_status.category to four values and changed Effective() to leave nonterminal custom statuses as their stored key. The two gc-check endpoints normalize through Effective(), so raw custom keys started reaching every installed daemon, where isKnownIssueStatus() still only recognizes the 7 built-ins. The GCCompletedTaskTTL full-cleanup path requires a recognized status, so it silently switched off for any workspace using a custom status: environment skeletons accumulate for as long as the parent issue sits there. GCArtifactTTL is unaffected, so the symptom is slow disk growth rather than a total stall. Fix both layers: - Both endpoints resolve the lifecycle category and re-encode it into the legacy seven-value `status` enum (WireCategory), which restores the v0.4.43 behavior for every daemon already installed. - Both endpoints also return the four-value `category`, and the daemon decides on it when present, falling back to `status` for servers that predate this change. A status with no category (unresolvable key, Triage) is returned raw with no category so every daemon fails closed. That removes the daemon's dependency on the 7 built-in keys instead of leaving a comment to guard it -- a comment is what failed to guard it this time, since it claimed the endpoints already answered with a category. Co-authored-by: multica-agent <[email protected]> * fix(server): prefilter unresumable recovery sources by key and category (MUL-7364) ListPendingDelegatedFailureRecoveries excluded source issues via COALESCE(issue_status.category, issue.status) NOT IN ('done', 'cancelled', 'backlog') -- reading the category column but comparing it against the old seven-value key vocabulary. Since MUL-7240 a seeded catalog (every workspace, per migration 469) makes COALESCE return the category, so 'cancelled' arrives as 'closed' and 'backlog' as 'unstarted' and only 'done' was still excluded. Nothing runs wrongly -- loadDelegatedFailureRecoveryTarget re-checks with Effective() -- but those rows are discarded in Go WITHOUT stamping recovery_settled_at, which is the very thing the query's own comment says keeps the scan from growing with history. They are rescanned every tick and spend part of maxPerTick forever, eventually crowding out real recovery signals. Split the predicate by vocabulary instead of widening the list: the key test covers built-ins and does not depend on the catalog being seeded, the category test covers custom terminal statuses. 'backlog' stays a key test because parking is built-in behavior a custom unstarted status does not inherit. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Bohan Jiang
|
a9e82c7973 |
MUL-7282: fix(comments): delete only the selected comment and keep its replies (#8323)
Deleting a comment removed every reply under it, through the parent_id ON DELETE CASCADE. Now delete removes only that comment: one with replies becomes a tombstone (deleted_at set, body/attachments/reactions/resolution cleared) so the replies keep their parent, and one without replies is removed along with any tombstone ancestor left empty.
Tombstones are read-only and are never agent input. Every write to a comment's children takes its owners first (issue -> comment -> child), and a comment commits together with the attachments it was posted with. Clients only promise that replies are kept when the server declares comment_delete_keep_replies_supported, and then delete through DELETE /api/comments/{id}/keep-replies, which older servers do not route.
|
||
|
Bohan Jiang
|
676245af63 |
MUL-7034: fix(server): keep the parent workdir when an automatic retry must start fresh (#8291)
A task that failed with codex_semantic_inactivity is retried with a fresh provider session, which is right: the conversation is what got stuck. But the retry also lost the parent's workdir, so it started in an empty directory and could not see anything the failed attempt had done. Manual Retry already keeps the workdir for the same failure (MUL-4869); only the automatic path dropped it. Server: CreateRetryTask stops nulling work_dir (session_id is still nulled and force_fresh_session still set). The claim handler never read the child's own work_dir, so on both the issue and chat branches it now hands it back as prior_work_dir for an automatic retry (retry_of_task_id set) that must start fresh, with prior_session_resume_unavailable. A force_fresh task without retry lineage, such as the Lark fresh-session command, still inherits nothing. Daemon: a fresh session in a reused directory has no memory of the work there, and the brief's `multica repo checkout` resets an existing checkout, deleting uncommitted changes. The daemon now appends a per-turn "Files from an earlier run" block whenever it actually reused a prior workdir without resuming its session, including the fresh retry after a rejected resume. It is keyed to what the daemon did, so a GC'd directory gets no warning, and it never enters the byte-stable brief. The server offers the parent's workdir only to daemons advertising reused-workdir-notice-v1; an older daemon keeps getting a fresh directory. Closes #7998 |
||
|
importcpp
|
1500abe357 |
MUL-7272: fix(comments): preserve deferred worker reply handoffs (#8288)
A worker's plain reply that arrived after the squad leader's run was claimed, but before it started, was never delivered. Creation deferred it to completion reconcile, and reconcile only replays explicit agent mentions, so the leader never received another run for it. Record such replies in the claimed run's planned inputs. On completion, replay the assigned-squad-leader route only when the reply was recorded there, after rechecking current routing and permissions. Timestamp-only agent replies, self-replies, notes and suppressed replies still earn no follow-up. RegisterPlannedCommentForActiveTask now also rechecks status on the UPDATE target, so a registration that waited on a completing row misses and enqueues a successor instead of attaching an obligation to a completed run. |
||
|
|
5132ac99eb |
MUL-7235: show cancellation actor in run history (#8249)
* MUL-7235: show cancellation actor in run history
Co-authored-by: multica-agent <[email protected]>
* fix(issues): simplify failed run recovery UI
Co-authored-by: multica-agent <[email protected]>
* Revert "fix(issues): simplify failed run recovery UI"
This reverts commit
|
||
|
|
b5a7ee1e0e |
fix(comments): remove delayed assignee fallbacks (#8174)
* fix(comments): remove delayed assignee fallbacks * fix(comments): retire fallback retry descendants Co-authored-by: multica-agent <[email protected]> * fix(comments): preserve fallbacks with merged comments Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Lambda <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Jiayuan Zhang
|
028f3b4437 |
feat(issues): show agent execution inline in comment threads (#8131)
* feat(issues): show agent runs inline with comment replies * fix(issues): unify run comment projection and preserve disclosure focus * refactor(issues): replace nested run cards with inline status rows * fix(issues): align thread roots with reply content * style(issues): remove thread toggle background * style(issues): group reaction picker with comment actions * feat(issues): show readable issue references in run summaries * style(issues): move completed run logs into comment headers * style(issues): balance comment block vertical spacing * style(issues): shimmer live agent progress text * style(issues): unify comment action hover feedback * style(issues): reveal comment actions on block hover * fix(issues): move published reply activity into the header * fix(issues): settle comment UI as soon as replies are published * fix(issues): reveal comment log buttons on hover * fix(issues): defer run placement until trigger comments arrive * fix(issues): isolate agent comment queues by thread * fix(issues): display pending runs in execution order * fix(ci): align integration tests with thread-scoped queues * fix(issues): fit run actions to narrow comment columns |
||
|
|
cc09c3c12a |
MUL-7008: Relax healthy WebSocket claim polling to 3 minutes (#7983)
* MUL-7008: relax websocket claim fallback polling Co-authored-by: multica-agent <[email protected]> * fix(daemon): harden websocket claim polling Co-authored-by: multica-agent <[email protected]> * fix(daemon): harden deferred claim scheduling Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
0eff894b59 |
fix(server): preserve legacy handoff notes (#7989)
Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
d276991c60 |
MUL-6743 refactor(issues): remove the assignment handoff note (#7626)
* refactor(issues): remove the assignment handoff note (MUL-6743) Usage data puts the handoff note at 0.2% of assignments, so the box occupied the confirm dialog's centre for the other 99.8% and bought them nothing. Removing it takes the whole feature with it: the note input and its runtime version gate, the `handoff_note` write field on PATCH/batch issue updates, the task column write, and the daemon's opening-prompt + issue_context.md handoff sections. The confirm dialog itself stays — it still names who is about to run and still offers "Don't start yet" (suppress_run). The `*WithHandoff` enqueue methods only ever varied by that note, so they become `EnqueueTaskForIssueByActor` / `EnqueueTaskForSquadLeaderByActor` — the actor-carrying variants autopilot already called with an empty note. `preview-trigger` drops `handoff_supported`, which nothing read after the gate moved client-side. `agent_task_queue.handoff_note` is deliberately left in place: it holds historical rows and this is the expand half of an expand-contract pair. Nothing reads or writes it now, so a follow-up migration can drop it once no deployed binary selects it. Co-authored-by: multica-agent <[email protected]> * docs(issues): reshoot the assignment confirm dialog (MUL-6743) The asset still showed the handoff note box the surrounding prose no longer describes. Recaptured from the app at the same 600x450 frame; the confirm button now carries the send-chord keycaps it has shipped with since MUL-5694. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Lambda <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
879d0de916 |
refactor(agent tasks): drop the Active sibling runs prompt block (MUL-6984) (#7937)
Every issue turn carried a claim-time snapshot of the same agent's other in-flight tasks, scoped only by (agent, workspace) and capped at five. The scope had no relatedness filter, so the list answered "what else is this agent running right now", not "what else touches the code you are about to change". Measured over one day of runs on this workspace: the block was injected 78 times across 30 runs, listing 96 distinct (current issue, sibling) pairs, and not one pair shared a PR, a branch, or an issue family. `run-messages` was invoked zero times off it. Meanwhile the day's one genuine overlap — a PR review pushing to server/internal/metrics/pricing.go and a concurrent change to the model catalog next to it — was never paired, because overlapping in time is not the same as overlapping in code. The duplicate-run defect this shipped with (#6947) is fixed by shouldSuppressActiveSelfAssignment, which stops a second run on the same (issue, agent) at the server. Once that holds, the block can only ever list a DIFFERENT issue, so it was structurally unable to help with what it was added for. #7922 then gave the coordination question a correct answer — `issue runs --active` / `--siblings`, cross-agent, family-scoped, and read at the moment it is asked — which is the same information as a pull instead of a per-turn push. Removed end to end rather than hidden behind a flag: the prompt block, the claim payload field on both sides of the wire, and ListActiveSiblingIssueTasks. The field was `omitempty` and the daemon already rendered nothing for an empty list, so old and new server/daemon pairs stay compatible in both directions; nothing outside server/ ever read active_sibling_runs. Kept: the self-assignment suppression and its tests, `--no-start`, `issue run-messages`, and `issue runs --active/--siblings`. Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
9fce92f427 |
MUL-6846: feat(cli): expose active and cross-issue agent runs on issue runs (#7922)
* feat(cli): expose active and cross-issue agent runs on `issue runs` An agent could already read one issue's task runs, but only as the full execution history — every past run, hydrated with attribution and token usage — with no way to ask the one question coordination actually needs: who is working right now. And nothing at all could answer it across a parent's sub-issues, which is the case #7768 describes: several agents claim sibling children in parallel and each starts overlapping work without knowing the others exist. The claim-time `## Active sibling runs` block does not cover this. It is scoped by agent_id — your OWN other in-flight tasks — so a different agent running on a sibling issue is invisible, and it is a snapshot taken before the turn started. Two query params on the existing task-runs endpoint, and the flags that drive them: multica issue runs <id> --active # in-flight runs on this issue multica issue runs <id> --siblings # ...and across the sub-issue family `--siblings` roots at the issue's parent, or at the issue itself when it has none. One rule covers both directions: a child sees its siblings, a parent sees its children, a standalone issue degenerates to its own active runs — so a caller does not have to know an issue's shape before choosing a flag. Rows carry the issue they belong to, ordered running-first and bounded, because a parent that fanned out hundreds of children must not turn an advisory read into an unbounded response. Both reads are advisory. Nothing here reserves an issue or serialises anything; they tell a run whom to coordinate with, and coordination still happens in comments. With neither param the response is byte-identical to before, which the issue-detail execution log and the CLI's short-task-ID resolver both depend on. Co-authored-by: multica-agent <[email protected]> * fix(cli): address review on active/sibling run reads Four fixes from review, all on the same theme: a coordination read has to be cheap and has to be honest about what it did not tell you. - `--active` no longer hydrates usage. ListIssueTaskUsage returns a row per (task, provider, model) for every task the issue ever ran, so the active path was still paying the full-history cost the filter exists to remove — for a column that is near-empty on runs that have not finished. The execution log keeps it. - An invalid `active` value is a 400 instead of a silent fallback. `active=tru` used to answer "who is here now?" with the whole history under a 200, and a pile of finished runs reads as the live ones. Same reason the unknown-scope check already existed, same shape as the comment-list boolean params. - The 50-row family cap now announces itself. The query asks for one row past the cap and sets X-Active-Runs-Truncated when it comes back; the CLI warns on stderr. Without it, "the answer was cut off" and "no run on that sibling" are the same response, and an agent reads the second. The cap is also documented in the flag help, the skill and CLI_AND_DAEMON.md, along with the ordering it depends on. - The skill's new agent-facing contract is pinned in builtin_skills_test.go: that the claim-time block is same-agent only, the `--siblings` command itself, the cap disclosure, and the advisory/no-lock boundary — the last a negative anchor, because an agent that reads these as a lock skips the coordination they exist to prompt. Also reuses service.IssueIdentifier instead of re-deriving the format. Co-authored-by: multica-agent <[email protected]> * fix(server): expose the truncation header to CORS, correct the cap rationale Both from re-review, and both about the cap telling the truth. - HeaderActiveRunsTruncated was missing from corsExposedHeaders. No browser client requests scope=family today, so nothing is broken right now — but a custom response header that is not exposed is unreadable from JS, which is the exact silent-truncation failure the header was added to remove. The registry and its guard test exist because this drifted once already (MUL-5492); completing both is two lines. - The comment justifying 50 was wrong. It claimed real family concurrency is bounded far below the cap so 50 sits above any legitimate answer. It does not: agentconfig.MaxMaxConcurrentTasks lets ONE agent run 50 concurrent tasks, this feature exists for the case where several agents work a family at once, and the returned set includes queued / dispatched / waiting rows that no execution slot bounds at all — a parent that fans out 200 children can have 200 enqueued. So 50 is a response-size budget and nothing more, truncation is an ordinary outcome rather than a pathological one, and that is precisely why it has to be reported. Comment and PR body now say that. Co-authored-by: multica-agent <[email protected]> * refactor(cli): give the family read its own compact row, cap at 20 The family read was reusing AgentTaskResponse — the execution log's row. Measured against real data that is ~1645 bytes per run, of which the caller reads almost nothing: attribution (625 bytes, and its own extra query), result, work_dir, relative_work_dir, trigger_summary, coalesced comment ids. An agent asking "is anyone working next to me?" was spending ~20k tokens of its own context on the answer. ActiveRunSummary carries what the question needs and nothing else: task, issue, agent, status, and timestamps. 367 bytes per row. It is a separate type on purpose — not AgentTaskResponse, and not ActiveSiblingRunData (the claim payload's row, which has no agent field because those runs all belong to the claiming agent by construction). Reusing either would mean an optional field only one caller ever sets, which is the shape this commit removes from AgentTaskResponse. The query now names its columns instead of embedding the task row, so Postgres stops detoasting the JSONB blobs too, and the handler drops attribution hydration along with the field. Cap goes 50 -> 20 (maintainer call): a handful of runs is the normal case, so the budget is set to the normal case. Truncation was already reported and still is, which is what makes a small cap safe. Worst-case response: 80 KB / ~20k tokens -> 7 KB / ~1.8k tokens. The CLI's family table drops COMPLETED and ERROR, which are empty on every row of a read that only returns work still in flight. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
f1ce9d4542 |
MUL-6883 perf(server): stop rescanning settled delegated-failure recoveries (#7820)
* feat(server): retire settled delegated-failure recoveries from the outbox scan ListPendingDelegatedFailureRecoveries had no indexable notion of a finished recovery. Its partial index (343) selected every recovery comment ever written, so each sweeper tick re-proved through four joins and two NOT EXISTS subqueries that history was already handled. LIMIT bounds the rows RETURNED, never the rows CHECKED, so a tick that found nothing still grew more expensive as history accumulated. Add comment.recovery_settled_at and a partial index that holds only unsettled recovery comments, and write the marker inside the transaction that makes a task terminal. Every other exclusion the scan applies is reversible — an issue can leave 'done', a cancelled retry can be superseded — so none of them can be frozen into an index predicate. A delivery receipt on a terminal task cannot be taken back, which is what makes it safe to record as durable state. The marker is deliberately written only once the task is terminal. SetTaskDeliveredCommentIDs replaces rather than appends a receipt while a task is still dispatched, so a reclaim by a differently-capable daemon can drop an id it will not deliver; settling at receipt-write time would freeze that legitimate transient window into a permanently lost recovery. Exhaustion is the one terminal outcome whose receipt lands on a row that may still be running, so it retires the comment directly in the same transaction. An unmarked row is the safe direction: it is scanned exactly as it was before this column existed. That is what makes the rolling-deploy window harmless — an older replica that writes a receipt without the marker only leaves work for the backfill. Co-authored-by: multica-agent <[email protected]> * feat(server): backfill delegated-failure recoveries settled before the marker The marker only covers tasks that reach a terminal status after the writing code ships. History written before that keeps matching the unsettled index predicate and keeps being re-checked on every sweeper tick, which is the cost this change set exists to remove. The walk retires a candidate only when a terminal task already recorded it in delivered_comment_ids — the same monotonic condition the runtime path uses. Its keyset watermark advances over every candidate EXAMINED rather than every candidate marked: a recovery that is genuinely still pending must be left alone, and keying on updated rows would hand back the same watermark forever and never terminate. Separate from migrations and startup, in the shape the other backfill commands already use, so a large comment table cannot block a deploy. Running it is a performance recovery step, not a correctness step — until it finishes the scan behaves exactly as it does today. Co-authored-by: multica-agent <[email protected]> * fix(server): close the review gaps in the settled-recovery backfill and paths Three fixes from review. 1. The backfill reproduced the bug it was written to fix. Its keyset ordered by id while idx_comment_delegated_failure_unsettled is (created_at, id), so the candidate scan read every unsettled row and sorted it to return LIMIT of them; and the receipt test ran a per-candidate EXISTS over agent_task_queue, the largest table in the database. The keyset now matches the index, and the receipt lookup is inverted into one agent_task_queue read per BATCH that hash-joins delivered ids back against the candidates. Measured on 500k comments / 200k tasks / 50k unsettled recoveries: 17,393ms and 816,963 temp blocks before, 235ms and no temp spill after. The candidate scan goes from reading 50,000 rows to reading the 1,000 it needs. Total walk time is flat across batch size (50x228ms, 10x1016ms, 5x2199ms — all ~11s), so batch size only trades how long one statement holds row locks. 1000 stays, now for a measured reason rather than a guessed one. 2. Bulk cancellation paths could strand a delivered receipt permanently unsettled, which lets the new index regrow the history it exists to remove. Manual rerun's pending-slot clear and agent archive were the two named in review; auditing the rest found comment-edit cancellation, both chat-session cancellations, workspace revoke and runtime teardown in the same position. All now settle — inside the caller's transaction where one exists, and through CaptureCancelledTasks for bulk paths that finalize outside one. The invariant is stated on the exported helper so the next such path has somewhere to read it. 3. --max-batches could not actually resume. The watermark lived only in the process, so a bounded rerun restarted at the front of the index — which is precisely where the rows this walk never settles live, so repeated bounded runs would rescan the same prefix and never reach the settleable history behind it. Adds --checkpoint-file (atomic replace, read on start) and explicit --after-created-at/--after-id, and logs the resume cursor on every exit path including SIGTERM. Co-authored-by: multica-agent <[email protected]> * fix(server): make delegated-failure settlement terminal-checked and atomic Two fixes from review. 1. The settlement SQL trusted its caller. "A delivery receipt is only immutable once the task is terminal" is the safety premise of the whole design, but the exported helper enforced nothing: passing a dispatched or running task would settle a receipt a reclaiming daemon can still replace, and the marker is monotonic, so nothing could undo it. The predicate now lives in the SQL, so a non-terminal task updates no rows instead of destroying the obligation. 2. Bulk settlement was post-commit best-effort, and the comment claiming the sweeper would replay a failed one was simply wrong: the covering task is terminal and already holds the receipt, so ListPendingDelegatedFailureRecoveries excludes the comment. It would neither replay nor settle — it would sit in the partial index forever, restoring the unbounded scan this change set removes. Every terminal write now commits with its settlement. CancelTasksForIssue / ForAgent / ByTriggerComment and the rerun pending-slot clear run in a transaction; CancelQueuedChatTasks settles inside the one it already had; the sweeper and daemon-recovery bulk fails move behind TaskService wrappers that own the transaction; agent archive gets CancelTasksForArchivedAgent, which cancels and settles atomically while still skipping the per-task broadcast the archive event makes redundant. BroadcastCancelledTasks, CaptureCancelledTasks and HandleFailedTasks run after their caller committed, so they no longer settle at all — a failure there could be neither rolled back nor repaired. The best-effort helper is deleted rather than left available; every remaining call site passes a qtx. Tests: a dispatched/running task with a receipt is not settled (fails without the SQL predicate), a bulk cancel whose settlement fails rolls the cancel back, and the archive path settles through its atomic entry point. Co-authored-by: multica-agent <[email protected]> * docs(server): correct settlement invariant comment and drop leaked test function The SettleDeliveredDelegatedFailureRecoveries doc still described the pre-fix contract — settlement reachable through CaptureCancelledTasks or a post-commit best-effort call — which is exactly the shape the atomicity fix removed. Restate the rule as it now stands (same qtx as the terminal write, never from a post-commit helper) so the stale text cannot guide a future change back into stranding rows in the partial index. TestBulkCancelRollsBackWhenSettlementFails dropped its trigger but left reject_recovery_settlement() behind in the shared test database; drop the function too. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]> Co-authored-by: Eve <[email protected]> |
||
|
|
db86c75e12 |
MUL-6878: avoid per-row issue status resolution (#7814)
* perf(server): avoid per-row issue status resolution Co-authored-by: multica-agent <[email protected]> * fix(server): address issue status query review nits Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
64ec7f5416 |
MUL-6737: fix(timeouts): close the two gaps the 2h inactivity budget left open (#7699)
* fix(timeouts): close the two gaps the 2h inactivity budget left open (MUL-6737) Two independent places where a timeout still answered the wrong question. Codex runs a semantic-inactivity timer inside the app-server protocol, and unlike the daemon's watchdog it cannot see that a tool is in flight: a commandExecution that prints nothing for the window trips it anyway. At a fixed 10m ceiling that timer killed a quiet test suite or an output-buffering `docker build` long before the 2h daemon budget meant to protect it, which made "we raised the budget to 2h" false for Codex users. It now derives from the daemon budget, taking the larger of the idle and tool windows because it stands in for both. A tool budget of 0 means "never force-stop during a tool", which this timer cannot express, so it falls back to the idle budget rather than running unbounded; with the watchdog suite disabled entirely Codex keeps its own built-in default, as it always has. The env override and flag are unchanged. Queued tasks expired on a pure wall clock: queued longer than a TTL meant failed. That conflated "nobody is coming for this" with "the queue ahead is long", and only the first is a failure. MUL-6558 was the second — a self-hosted runtime with low concurrency held its own queue past 2h and healthy work died as queued_expired. The TTL knob added then only moved the cliff. Expiry now keys on whether the owning runtime still proves liveness, the same signal FailTasksForOfflineRuntimes already uses for dispatched/running rows, so a daemon going down retires its queued and its in-flight work on one clock instead of two. A busy runtime keeps its backlog for as long as it takes. MULTICA_TASK_QUEUED_TTL is removed with the wall clock it configured. Heartbeat age is read directly instead of gating on runtime.status='online' so a row stuck at 'online' with a dead heartbeat still releases its queue, and rows with no usable runtime binding — runtime_id IS NULL from before migration 251, whose CHECK landed NOT VALID, or a runtime row that is gone — are expired on sight rather than stranded forever by a JOIN that matches nothing. Tests: the queued sweep test now runs both phases against the same rows, asserting the 5h-old task survives while its runtime heartbeats (the MUL-6558 regression) and only expires once that runtime goes silent. Assertions are scoped to the test's own task ids, because a runtime-keyed sweep legitimately also collects queued rows other tests left on the shared fixture runtime. Co-authored-by: multica-agent <[email protected]> * fix(sweeper): give a queued task its own reconnect grace before expiring it Review catch: keying expiry on runtime liveness alone was too aggressive in one direction. Enqueue binds a task to agent.runtime_id without checking that the runtime is up (task.go CreateAgentTask call sites), so for a runtime that has already been dark for longer than the grace, the liveness clause is satisfied the moment a new task is created — assigning an issue to a laptop closed overnight would fail it inside one 30s sweep tick instead of waiting for the machine to come back. That is the opposite of what "reconnect grace" promises. Require the row's own age as well: a queued task now waits a full grace, counted from when it started waiting, before it can be given up on. A heartbeating runtime still never expires its backlog, so the MUL-6558 fix is unchanged. The age bound also stops the sweep re-evaluating the runtime subquery against every queued row on every tick. Also remove the deploy surfaces the previous commit missed — the knob was deleted from the server but helm (values + configmap), the self-host compose file, and the CI-run helm render assertions still advertised it, so operators would have kept setting a variable nothing reads. Same for five orphaned comment lines in .env.example describing the removed variable. Correct the fail-closed comment on the unbound-runtime arms: they are not reachable for "rows predating migration 251" — runtime_id was NOT NULL from migration 004 until 251 replaced it with a CHECK that every insert and update is verified against, and the migration-004 FK is ON DELETE RESTRICT. The arms stay as defence against a future schema change, which is what the comment now says. Tests: a third phase covers the reported case — a task enqueued against an already-dead runtime survives, then expires once it has waited a full grace of its own. Phase 2's expectation is corrected accordingly: the row created at now() no longer expires when its runtime dies, because it has not waited its own grace yet. Co-authored-by: multica-agent <[email protected]> * docs(timeouts): state both queued-expiry conditions and correct the FK claim Review follow-up on |
||
|
|
09a2410e88 |
refactor(agents): rename starter_prompts to conversation_starters (MUL-6709) (#7592)
* refactor(agents): rename starter_prompts to conversation_starters
The UI, docs and all four locales call this feature "Conversation starters"
(#7573), but every identifier behind it still said `starter_prompts` — the DB
column, the wire field, the `/api/config` capability flag, the TS types, the
i18n keys, and the `?focus=` token that a user can read in their own address
bar. The old name is also the one #7573 rejected on the way in: it promises a
prompt, while selecting a starter only prefills the composer.
Rename it everywhere so the identifier matches the product.
This is a breaking rename with no compatibility shim, and it is free exactly
once: the column arrived in migration 404 (#7261) after the last release tag,
so nothing has shipped. No deployed client reads the old name, no stored data
uses it, and `agent_conversation_starters_supported` has no version skew to
straddle. After this release, the same change would cost a wire alias, a
dual-written capability flag, and older Desktop/Mobile builds silently hiding
the editor.
Migration 404 stays as-shipped; 432 renames the column and its CHECK
constraint, so an already-migrated dev or staging database converges with a
fresh one.
Co-authored-by: multica-agent <[email protected]>
* docs(skills): name the conversation-starters surface for agents
`multica-creating-agents` mentioned `conversation_starters` only as a wire
field in two tables. It never said what the feature is called in the product,
where a human configures it, or what selecting one does — so an agent reading
this skill would tell a user about "conversation starters" as a JSON field and
leave them with nowhere to go.
Add the product name (both languages), the Instructions-tab deep link, and the
behaviour that makes the name accurate: selecting one only fills the composer.
Also record that an agent with none configured still shows three built-in
defaults, so "the Chat shows suggestions" is not evidence this agent has any.
Document one gap the field table implied away: the accepted-body list is not a
CLI surface. `conversation_starters` has no `agent create` / `agent update`
flag — only `agent copy` carries an existing value forward — so an agent
following the old text would build a create command that silently drops it.
Co-authored-by: multica-agent <[email protected]>
* fix(onboarding): keep the released starter_prompt field on the pre-v3 shim
The rename in ec44cc0ef swept up `starter_prompt` (singular) on
`bootstrapOnboardingRuntimeRequest`. That is a different contract from the
per-agent `starter_prompts` this PR renames: it is a field on the endpoint
marked "DEPRECATED, kept for desktop < v3", and unlike the per-agent one it
HAS shipped — it is in v0.4.34, released since
|
||
|
LinYushen
|
3b1018e9c9 |
Revert "MUL-6016: feat(task): retry exact Codex capacity failures (#6747)" (#7586)
This reverts commit
|
||
|
|
bbba9c5460 |
MUL-6697: enforce runtime access on task claims (#7571)
* fix(tasks): enforce runtime access on claims Co-authored-by: multica-agent <[email protected]> * fix(tasks): settle invalid runtime claims explicitly Co-authored-by: multica-agent <[email protected]> * test(tasks): cover runtime owner claim mismatch Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
8fda4f22d5 |
MUL-6016: feat(task): retry exact Codex capacity failures (#6747)
* docs: record Codex capacity retry design * docs: add Codex capacity retry implementation plan * fix(taskfailure): recognize exact Codex capacity error * feat(task): retry exact Codex capacity failures * feat(server): configure Codex capacity retries * docs: expose Codex capacity retry setting * fix(task): align capacity retry comments --------- Co-authored-by: cuijs <[email protected]> Co-authored-by: Claude <[email protected]> |
||
|
Yuri GuiandAgent Op
|
f8ec870f30 |
MUL-6629: feat(agents): add per-agent starter prompts for chat (#7261)
* feat(agents): add per-agent starter prompts Co-authored-by: Agent Op <[email protected]> * fix(agents): default omitted starter prompts Co-authored-by: Agent Op <[email protected]> * fix(migrations): renumber agent starter prompts Co-authored-by: Agent Op <[email protected]> * fix(chat): make starter prompt prefills reliable Co-authored-by: Agent Op <[email protected]> * fix(agents): preserve starter prompt writes Keep dirty starter-prompt edits across agent refetches and fail closed when an older server cannot persist them. Clarify that copied starter prompts have no dedicated CLI override. Co-authored-by: Agent Op <[email protected]> * fix(migrations): avoid starter prompt prefix collision Main now uses migration prefixes through 403, so move the starter-prompts migration to the next available prefix before requesting another review. Co-authored-by: Agent Op <[email protected]> * fix(agents): preserve edits after failed save Ignore optimistic instruction snapshots while a save is pending so a rejected update can roll back the cache without erasing the user's submitted starter prompts. Cover the delayed optimistic-update and rollback sequence for both label and prompt fields. Co-authored-by: Agent Op <[email protected]> --------- Co-authored-by: Agent Op <[email protected]> |
||
|
YYClaw
|
8c563b49be |
MUL-6555 feat(issues): preserve source context for sub-issues (#7461)
* feat(issues): preserve source context for sub-issues * fix(issues): address source context review findings * fix(migrations): guard source context rollback preflight |
||
|
|
1cc46b2697 |
MUL-6516 fix(channels): enforce durable context boundaries for /new (#7362)
* fix(channels): enforce new context boundaries * fix(channels): address context boundary review findings * fix(channels): enforce causal new context boundaries * fix(channels): scope outbound provenance to target --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
Bohan Jiangandmultica-agent
|
8c9b7503a1 |
MUL-6146 fix(rerun): stop cancelling an in-flight agent run on manual rerun (#7282)
* fix(rerun): stop cancelling an in-flight agent run on manual rerun (MUL-6146) RerunIssue cancelled every active task for the (issue, agent) pair before enqueueing the replacement, including 'running' and 'waiting_local_directory'. That made a manual rerun destructive: asking an agent for another pass silently killed the pass it was still working on. Neither status appears in idx_one_pending_task_per_issue_agent_v2, so a fresh queued row can be inserted alongside an executing one, and ClaimAgentTask's per-(issue, agent) serialization already holds it there until the active run reaches a terminal state. Narrow the cancel to rows that have not begun executing and let the queue do the rest. Rows that have not started are still cancelled and replaced rather than reused, which keeps the new run attributed to the rerunning member (MUL-4302 §5) instead of inheriting the attribution of whoever created the pending row. Co-authored-by: multica-agent <[email protected]> * fix(rerun): skip auto-retry when a manual rerun already holds the pending slot (MUL-6146) Letting a manual rerun queue behind a running task opened a collision the previous commit did not account for. If that running task then failed for a reason whose retry child is created immediately (timeout, runtime_recovery, codex_semantic_inactivity, skill_bundle_unavailable, or provider_network's first attempt), CreateRetryTask inserted a SECOND queued row for the same (issue, agent) and violated idx_one_pending_task_per_issue_agent_v2. That insert shares FailTask's transaction, so the unique violation rolled the parent's failed status back with it: the daemon's terminal callback saw a 500 and the task was left stuck in 'running'. Guard both auto-retry paths with hasRunnableSuccessor. When something is already pending for the pair, skip the retry — a runnable successor is exactly what the retry was for, and the rerun is it. The FailTask check runs inside the failure transaction so a concurrently committing rerun cannot take the slot between check and insert. runtime_offline and provider_network's final attempt were never affected: retryDelayForAttempt defers those, and a deferred child is outside the index. Co-authored-by: multica-agent <[email protected]> * fix(rerun): make the retry insert yield the pending slot instead of raising (MUL-6146) The previous commit guarded both auto-retry paths with a successor check and claimed the in-transaction check prevented a concurrent rerun from taking the slot. It does not: hasRunnableSuccessor is a plain count at READ COMMITTED and takes no lock, and rerun's cancel and enqueue are separate commits, so a rerun can always commit between the check and the insert. The unique violation then aborts FailTask's transaction, which also carries the parent's failed status, leaving the task stuck in 'running' — the very bug the guard was meant to fix. Move correctness into the insert: CreateRetryTask now yields via ON CONFLICT DO NOTHING, so it can never abort its caller's transaction. That makes the policy explicit — whoever already holds the slot keeps it, so a deliberate human rerun wins over an automatic retry. Both callers read the resulting zero rows as 'no retry created' and still commit their own work; this also stops the pre-existing workspace-teardown fence from aborting a failure the same way. For the reverse order, RerunIssue re-clears and enqueues once when a retry beat it into the slot, so the human action still wins. Bounded to one extra attempt. The successor checks stay as an optimisation, with comments corrected to say so rather than claiming a mutual exclusion they never provided. Co-authored-by: multica-agent <[email protected]> * fix(rerun): fence deferred promotion and recognise the normalized slot-taken error (MUL-6146) Two gaps left by the previous commit, both reachable now that a manual rerun can sit in the pending slot while a task is still running. Deferred retries escaped the conflict fence entirely. runtime_offline and provider_network's final attempt arm the retry as a 'deferred' row, which is not covered by idx_one_pending_task_per_issue_agent_v2, so CreateRetryTask's ON CONFLICT never fires and the row is created alongside the rerun. Promotion then flipped it to 'queued' unconditionally and hit the index — and because the claim loop promotes before it claims, that error blocked every claim on the runtime, starving the rerun the operator was waiting for. Both promote queries now skip a row whose (issue, agent) slot is occupied, and promote at most one row per pair so a single statement cannot collide with itself. Skipped rows stay deferred and promote on a later tick once the slot frees, so no retry is lost. RerunIssue's reclaim only matched the raw pgconn violation, but enqueueMentionTaskWithCommentPlan normalizes it to the bare ErrDuplicatePendingTask sentinel — the path taken by every rerun whose target is not the issue's current agent assignee (squad leader, an agent re-fired by task_id after being displaced, a mentioned agent). Those never entered the reclaim branch, so losing the race surfaced as a hard error. pendingSlotTakenErr now matches both shapes. Co-authored-by: multica-agent <[email protected]> * fix(rerun): cancel superseded deferred retries and tolerate promotion slot conflicts (MUL-6146) Keeps the one product invariant this change exists for: one rerun click means exactly one more run. The previous commit only declined to promote a deferred retry while the slot was occupied, and a test froze that as correct. It is not. Once the manual rerun starts running it stops occupying the queued slot, so the stale retry promotes and runs the agent a SECOND time — duplicate comments, duplicate side effects, duplicate cost, and nothing anywhere to signal it. That also contradicted this file's own 'a manual rerun replaces the pending plan' and FailTask's 'a runnable successor already exists, so no retry is needed'. A superseded deferred retry is now cancelled before promotion instead. Scope is deliberately tight — retry clones only (retry_of_task_id set), never assignee-fallback escalations or channel-media pending rows, which are supposed to coexist with an active primary. Promotion also tolerates the unique violation now. The in-query NOT EXISTS fence cannot see an enqueue that has not committed yet, so promotion can still collide; that is self-healing (the next tick sees the committed row) but the error used to fail the claim outright, and for the batch path it failed every runtime in the set. One contended row now skips promotion for a tick instead. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> |
||
|
|
b4bfe083be |
MUL-6407: mint UUIDv7 primary keys for append-heavy tables (#7230)
* perf(db): mint UUIDv7 primary keys for the append-heavy tables
Random v4 primary keys scatter every insert across the whole primary-key
B-tree, so each row dirties a different page. On the tables that grow
continuously — the task queue, messages, comments, activity and audit rows —
that is pure write amplification. UUIDv7 puts a big-endian millisecond
timestamp in the leading 48 bits, so consecutive inserts share a page.
The 12 tables converted here are the append-only hot paths:
agent_task_queue, activity_log, inbox_item, comment, issue, chat_session,
chat_message, autopilot_run, webhook_delivery, task_token,
channel_inbound_audit, sys_cron_executions
Ids come from the new pkg/dbid, which returns a pgtype.UUID ready to assign to
a sqlc params struct. Upsert-dominated tables (task_usage, agent_runtime, ...)
are deliberately left alone: v7 only buys locality for rows that are actually
inserted. Lease and claim tokens keep gen_random_uuid() — they must stay
unpredictable and carry no timestamp.
No migration, and no backfill. Every converted INSERT binds its id as
COALESCE(sqlc.narg('id')::uuid, gen_random_uuid()), so the column default stays
reachable: a caller that passes no id inserts exactly as it does today. Tables
will hold a mix of v4 and v7 ids permanently, which the uuid type and every
parser in the repo already accept.
Verification: - pkg/dbid: version/variant, uniqueness, and a non-decreasing timestamp
prefix; a shape test asserting every converted query keeps the COALESCE
fallback; and a source scan asserting every production params literal mints
an id, so a call site added later cannot silently fall back to v4.
- internal/handler: round-trips issue, comment, activity_log, inbox_item,
chat_session, chat_message, agent_task_queue, task_token and
channel_inbound_audit against Postgres and asserts the stored id is v7,
plus one case proving an unset id still inserts (as v4).
- internal/scheduler: covers the one hand-written INSERT, asserting the
execution id is v7 while lease_token stays v4.
Co-authored-by: multica-agent <[email protected]>
* fix(db): address UUIDv7 review feedback
Co-authored-by: multica-agent <[email protected]>
* fix(db): cover Telegram audit UUIDv7
Co-authored-by: multica-agent <[email protected]>
---------
Co-authored-by: Sol-Boy <[email protected]>
Co-authored-by: multica-agent <[email protected]>
|
||
|
Wu Changming
|
7fdc854c26 |
fix(issues): copy durable local path for worktree tasks (MUL-6378) (#7176)
A worktree-mode local_directory task runs in a disposable git worktree that Finalize removes on every exit path, so the task's `work_dir` pointed at a directory that no longer existed. "Copy local workdir path" handed the user a path they could not `cd` into. The daemon now reports a `durable_work_dir` handoff instead of having clients infer one: - `removeLocalWorktreeDir` returns an error and confirms removal with `Lstat`, so a broken symlink cannot be mistaken for an absent directory. Finalize fails when removal cannot be confirmed. - `durable_work_dir` is written only after Finalize succeeds and the worktree is verifiably gone. A preserved worktree, an in-place run, a running task, a squad-leader task, and any older daemon all leave it unset, keeping the real `work_dir` authoritative. - The field travels on the complete, fail, and cancel-ack callbacks and is persisted on the task row (migration 362, additive nullable column). Writes use COALESCE for replay idempotency; the cancel-ack path CASes on `status = 'cancelled'` so a late ack cannot touch a completed or failed row. - The server derives `relative_durable_work_dir` through the same home-stripping helper as `relative_work_dir`; the absolute value is reserved for explicit clipboard actions and is never rendered. - Clients select the durable path only for a terminal task that carries the field. The issue actions menu and the transcript dialog share one resolver, and the label and toast say the task worktree was reclaimed, naming the delivery branch when one exists. The stored path is a point-in-time delivery snapshot: later renames or detachments of the project resource do not rewrite historical tasks. |
||
|
|
afd05ca221 |
MUL-6063: recover terminal delegated task failures (#6844)
* fix: recover terminal delegated task failures Co-authored-by: multica-agent <[email protected]> * fix(tasks): make delegated failure recovery durable Co-authored-by: multica-agent <[email protected]> * fix(tasks): bound delegated failure recovery Co-authored-by: multica-agent <[email protected]> * fix(tasks): address delegated recovery review nits Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Eve <[email protected]> Co-authored-by: multica-agent <[email protected]> Co-authored-by: Sol-Boy <[email protected]> |
||
|
|
22db0b0a2c |
MUL-5651: preserve tasks through runtime network partitions (#7129)
* fix(runtime): preserve tasks through network partitions Co-authored-by: multica-agent <[email protected]> * fix(runtime): bound reconnect retry waiting Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
b5a48eca13 |
fix(agent tasks): prevent duplicate self-assignment work (MUL-6168) (#6956)
* fix(agent tasks): avoid duplicate self-assignment runs Co-authored-by: multica-agent <[email protected]> * fix(agent tasks): address duplicate-run review Co-authored-by: multica-agent <[email protected]> * fix(agent tasks): resolve sibling-run nits Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Sol-Boy <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
c9727e6b2b |
feat(daemon): worktree mode for local_directory resources (MUL-5707) (#6759)
* feat(daemon): worktree mode for local_directory resources (MUL-5707) Local directory resources serialise every task on a per-path mutex, so two tasks on one folder queue behind each other. This adds an opt-in worktree mode: each task gets its own git worktree of the user's repo inside the daemon's env root, so tasks run concurrently and deliver their work as a branch instead of editing the user's working copy. Opt in per resource with execution_mode=worktree (default stays in_place, so existing resources are unchanged). Requires a git repo with at least one commit; a non-git directory fails with an actionable message rather than silently falling back to serial execution. Design points that make the mode safe to use: - The agent starts from what the user sees, not HEAD. Tracked edits are captured with `git stash create` (which writes a commit object without touching the index, working tree, or stash list) and replayed into the worktree; untracked non-ignored files are copied, bounded at 2000 files / 200MB with the skipped count logged at warn level. - That replayed state is committed as a baseline, so "did this task change anything?" has an exact answer and `git diff <baseline>..<branch>` is precisely the agent's work. - Nothing is silently discarded: leftovers are committed to the branch before the worktree is removed, including when the task fails partway. - A task that changes nothing leaves no branch behind. - Sidecars stay out of the branch by running the existing CleanupRuntimeConfig + CleanupSidecars pass over the worktree before finalizing. The .git/info/exclude approach repocache uses is not usable here: a linked worktree resolves info/exclude to the user's own common git dir, so it would change what `git status` hides in their checkout. - Worktree-mode env roots do not get the local_directory GC exemption, so they age out normally instead of accumulating one env root per task. - `git worktree prune` on create self-heals registrations orphaned by a crashed daemon. Adds --execution-mode to `project resource add/update`, and updates the docs and the projects-and-resources built-in skill. Co-authored-by: multica-agent <[email protected]> * fix(daemon): close data-loss and isolation gaps in worktree mode (MUL-5707) Review follow-up on #6759. Six issues, all on failure or skew paths. 1. Preserve work when the final commit fails. Finalize logged a warning and then ran `git worktree remove --force`, destroying the agent's changes. commit.gpgSign with no key available to the daemon triggers this, and --no-verify does not disable signing. Finalize now returns an error and deliberately keeps the worktree — it stays registered, so `git worktree list` points at the work — and the daemon fails the task instead of reporting success with a branch that is missing the changes. 2. Fail closed on an unsupported execution_mode. Treating an unknown mode as in_place was wrong: execution_mode is how a user asks for ISOLATION, so the "safe" fallback would edit the working copy they asked us to stay out of. An unrecognised mode now fails the task naming the version skew. 3. Keep Codex conversations alive across turns. The per-issue session store was keyed on LocalWorkDir != "", so worktree tasks — which also never reuse an env root — got a fresh empty CODEX_HOME per turn and silently lost the thread. Covered by a two-turn regression test that fails without the fix. 4. Fail closed on an incomplete snapshot. A failed stash create/apply, a failed untracked copy, or hitting the copy bounds all degraded to "agent sees HEAD" with only a daemon log line. A half-replayed tree looks like a working checkout, so nothing downstream questions it while the agent reads different code than the user has. All of these now abort the prepare and tear the partial worktree back down. 5. Do not snapshot a directory mid-mutation. The same real path can be attached to another project as an in_place resource; snapshotting under a running in_place task captured a half-written tree plus that task's in-flight .agent_context/.multica sidecars, which then reached the branch and could feed the agent another issue's brief. Worktree prepare now takes the per-path mutex for the snapshot alone (released before the run, so tasks still overlap), and the untracked replay skips Multica's own sidecars. 6. Stop the test helper from masking regressions. gitRun skipped on any git failure, including post-finalize assertions, so a lost branch could report as a skip. Git availability is now checked once at repo setup; every later git failure is fatal. Also rebased onto main, keeping both MulticaConfigRoot and LocalWorktree in the Environment initialiser. Co-authored-by: multica-agent <[email protected]> * fix(daemon): close sidecar-commit and wait-visibility gaps in worktree mode; gate worktree saves on daemon version (MUL-5707) - runTask: register the sidecar-cleanup defer immediately after the Finalize defer. Previously two early returns (temp-dir setup, StartTask failure) sat between them, and on those paths Finalize auto-committed the sidecars Prepare had just written — delivering a branch whose only content is Multica's own runtime files (in_place: leaving them in the user's tree). - runTask: give the worktree snapshot wait the same plumbing as the in-place acquire — waiting_local_directory status, resourceWaitTasks accounting, and a cancellation poller. The holder can be an hours-long in_place task; the wait was invisible and a user cancel went unnoticed until the 5-minute prepare timeout. - isMulticaSidecarPath: match sidecar dirs as whole path segments at any depth, not only at the repo root. An in_place resource pointed at a SUBDIRECTORY leaves its sidecars below that subdirectory; replaying them committed another issue's brief into the delivered branch. - handler: refuse saving execution_mode=worktree while the owning daemon's registered cli_version is below MinLocalWorktreeCLIVersion (422, daemon_version_unsupported). An old daemon json-skips the field and runs tasks IN PLACE — the daemon-side unknown-mode refusal cannot protect a daemon that predates the field, so the server has to. Residual (documented): a daemon downgraded after the save needs a claim-time gate. - copyUntrackedFiles: read the -z listing raw instead of TrimSpace-ing it, which corrupted filenames beginning/ending with whitespace into paths that silently vanished from the replay. - docs + skill: describe the save-time version gate; drop the stale claim that old runtimes fail the task themselves. Co-authored-by: multica-agent <[email protected]> * feat(views): UI switch for local_directory execution mode (MUL-5707) Worktree mode shipped CLI/API-only, which for its target user — someone running several tasks at once from Desktop — means it does not exist. This adds the switch to the resources panel. The choice is framed by what the user gets back, not by the mode identifiers: "Edit this folder directly" (one task at a time, changes land in your working copy) vs "Run in parallel, isolated" (tasks run at the same time and each hands back an agent/… branch). `in_place` / `worktree` stay visible as a secondary hint for anyone cross-referencing the CLI or docs. Two entry points, matching where the decision is actually made: - The add-a-folder flow now asks for the mode after the folder is picked and validated, instead of creating the resource and leaving the mode to be discovered later. - Every local_directory row has a mode control. It is deliberately NOT gated on being the owning machine — switching the mode only rewrites a field, so it works from the web app too, unlike rename or the folder picker. Both blocked states are handled where the user chooses, not after they commit: - Non-git folder: the desktop validator now reports whether the path is inside a git working tree (walking up for `.git`, so a subdirectory of a repo counts), and the option is disabled with that reason. Unknown — an older desktop build, or an existing row — stays permissive; the daemon re-checks authoritatively. - Daemon too old: the version floor is mirrored from the server as MIN_LOCAL_WORKTREE_CLI_VERSION and checked against the runtime for the resource's OWN daemon, not the machine the browser happens to be on. A server rejection is rendered inline in the dialog rather than as a toast, so the dialog stays actionable. A "Parallel" badge on the row is always visible (not hover-only): without it there is no way to tell whether tasks on a folder edit it directly or hand back a branch, which is the first thing anyone asks when a task queues. Also adds `execution_mode` to LocalDirectoryResourceRef — the field existed server-side but had no TypeScript counterpart — and brings the zh/ja/ko project-resources docs in line with the English ones, which were the only locale updated when the mode landed. Co-authored-by: multica-agent <[email protected]> * fix(daemon,handler): close worktree isolation and delivery gaps (MUL-5707) Second review round. Five must-fix items, all on paths where the mode's promises could still break. 1. A failed sidecar cleanup no longer ships Multica's own files as the user's branch. The cleanup errors were logged and Finalize then ran `git add -A`, so a permission/disk/lock failure could deliver a branch whose entire content was the runtime brief and issue context. Worktree mode now escalates a cleanup failure to an abort: nothing is committed, the worktree is kept for inspection, and the task fails with the preserved path. In place it stays a warning — no commit, no branch, and the leftovers are visible in the user's own tree. 2. The untracked replay no longer drops files silently. `Lstat` failures and symlinks were `continue`d without counting, so the caller read a partial snapshot as complete. Both now count as skipped, which the existing fail-closed check turns into a refused prepare. Non-content entries (sockets, FIFOs, devices) stay uncounted deliberately. 3. Prepare rolls back a worktree it created. Any later failure — context files, provider home, MCP config — returned without an Environment, so nothing downstream knew a worktree existed, leaving a registration and a branch no task ever ran in. A deferred Discard now runs unless Prepare reached its successful return. 4. The daemon-version gate moved to where it can actually hold. Save-time validation cannot see a machine downgraded afterwards, and an old daemon json-skips execution_mode entirely — it would run the task in place, editing the working copy the user asked to isolate. Claims are now checked against the runtime that is actually claiming, and a blocked task is cancelled with a reason rather than left to be redelivered to the same too-old runtime forever. 5. branch_name is delivered end to end. The daemon sent it, but TaskCompleteRequest had no field so it vanished at decode, and the failure path never sent it at all. It now has a column, travels on both terminal callbacks, and renders as a copyable row in Run details. The failure path matters most: worktree mode commits the agent's leftovers before tearing the worktree down, so a run that died partway still has work worth finding. Docs (all four locales) and the projects-and-resources skill are updated for the claim-time gate and the branch surface. Co-authored-by: multica-agent <[email protected]> * fix(handler,daemon): migration renumber, cancel reasons, branch on every terminal path (MUL-5707) Third review round. 1. Rebased onto main and renumbered the migration to 285 — main had taken 273 and is now at 284. sqlc regenerated on top of the rebased queries; the conflicts were confined to generated files. 2. The claim gate now persists why it cancelled. It called CancelTask, which writes only status='cancelled', and returned the reason as a 4xx that reaches the daemon's log and nothing else — batch claim drops the task from its response entirely. A run refused for a durable reason the user has to act on cannot present as a bare "cancelled", so this adds CancelAgentTaskWithReason and records error + failure_reason on the row. 3. branch_name now survives every terminal path, not just the two direct callbacks. Three were dropping it: - the context-exhausted complete→fail reroute did not copy it; - the daemon's complete→fail fallback (server rejected the complete, agent had succeeded) did not set it; - both cancellation paths discarded the result wholesale, even though the worktree is finalized — and its work committed — before the daemon learns of the cancel. The branch now rides the cancel-ack and is recorded by SetAgentTaskBranchName, which never overwrites a name a terminal callback already wrote. 4. Rollout/rollback compatibility. Isolation is enforced by two server-side gates, so a daemon downgraded at any point after the resource was saved is caught. The gap they cannot close is rolling the SERVER back past this change while sub-floor daemons are still connected: an old server has no gate and an old daemon ignores execution_mode. That constraint is now documented where an operator will meet it (the migration and the docs), and the version floor is pinned across server and frontend by test so the two gates cannot silently drift apart. Co-authored-by: multica-agent <[email protected]> * fix(server,views): route claim-gate cancels through TaskService, make cancel reasons visible, harden cancel-ack branch delivery (MUL-5707) Closes the fourth-round review findings on worktree mode's cancellation chain: - Claim gate: cancel through a new TaskService.CancelTaskWithReason instead of a raw CancelAgentTaskWithReason query, so a gate-cancelled task gets the full cancellation side effects — audit capture, chat settle, agent status reconcile, task:cancelled broadcast, NotifyTaskFinished. Added REAL claim integration tests: singular claim (422 + row cancelled with reason), batch claim (task omitted + row cancelled with reason), and a current-runtime control. - Cancel reasons are now user-visible: execution log rows, agent activity rows, and the transcript dialog label a SERVER-cancelled task (persisted error/failure_reason) like a failed one, with the actionable text on hover and a Reason row in Run details. User-initiated cancels stay a plain "Cancelled". New cancelReasonLabel helper + local_directory_error label; details_reason locale key in en/zh-Hans/ja/ko; widened AgentTask.failure_reason to the open string the wire actually carries. - Cancel-ack is now a terminal delivery: the daemon retries it on transient errors (same schedule as complete/fail), the server fails LOUD (5xx) when persisting the branch/error fails instead of warn-and-200, and re-broadcasts task:cancelled after a successful write so clients that already refetched a branchless row pick up the delivery. Replays are idempotent (never overwrite). - Cancel + Finalize-abort no longer swallows the preserved-worktree pointer: the finalize error is wrapped in worktreePreservedError (joined onto an existing run error, never displacing it), extracted on the cancel path, and delivered via the ack into agent_task_queue.error through the new SetAgentTaskErrorIfEmpty query — never overwriting a reason recorded by a fail callback or the claim gate. Other run errors on a cancelled run stay discarded as before: they are expected noise. - Docs: the self-hosted rollback constraint callout now exists in zh/ja/ko, not just English. Co-authored-by: multica-agent <[email protected]> * fix(handler): guard cancel-ack writes to cancelled rows; requeue when the claim gate's cancel fails (MUL-5707) Fifth-round review fixes: - SetAgentTaskBranchName / SetAgentTaskErrorIfEmpty now carry a status='cancelled' CAS. The daemon acks on EVERY terminal status it observes, so a late or replayed ack from a stale run could previously backfill its branch or preserved-path error into a completed/failed row whose own callback — the authoritative channel — recorded nothing, without any event invalidating the change. A CAS-refused write is a deliberate no-op (still 200: nothing to retry), and the rebroadcast stays guarded by the same status check. Regression tests: a late ack against a completed row with empty fields and against a failed row leaves both untouched. - When the worktree claim gate's CancelTaskWithReason fails, the claimed row no longer strands in dispatched (daemon refused, nothing to release it but stale reclaim). The gate requeues via RequeueTaskAfterClaimFailure — the next claim re-runs the gate and retries the cancel — and reports 5xx so the singular daemon reads a transient server problem, not a claim refusal; the batch path omits the task as before, now with the row requeued. Fault injection tests (row-scoped trigger raising on the cancel UPDATE while letting the requeue through) cover singular and batch. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: Bohan-J <[email protected]> Co-authored-by: multica-agent <[email protected]> |
||
|
|
c4f0d8cc8c |
MUL-5999: make workspace teardown index-driven, bounded and fenced (#6754)
* perf(workspace): make teardown task discovery index-driven
Workspace teardown found a workspace's tasks with a three-way
`OR agent_id IN (…) OR issue_id IN (…) OR runtime_id IN (…)`, three times
per delete, and swept task_token with the same shape. PostgreSQL costs
that as a Seq Scan on agent_task_queue — the largest table in the
database (5.08M rows / 8.3 GB in production) — so deleting even a tiny
workspace paid for the whole table, inside the transaction that holds the
workspace row lock and global advisory lock 4246.
Resolve the task set one owner key at a time instead, in the handler, and
reuse that snapshot for every task-keyed step. Measured on a 1.2M-row
synthetic copy of the production shape (937 agents / 109 issues / 12
runtimes): the OR form and a 937-element `= ANY(array)` form both Seq
Scan; single-key equality is a Bitmap Index Scan. A UNION of parent-key
joins is worse than the OR (three hash joins, each seq scanning the
table), which is why that earlier suggestion was dropped.
All three ownership paths are preserved: nothing enforces that a task's
agent, runtime and issue share a workspace, so dropping a path would turn
a performance fix into a tenant-cleanup bug.
task_token needs no loop. Its workspace_id has been NOT NULL since
migration 108, so the workspace key alone covers this workspace's rows,
and the two extra OR arms only ever matched tokens owned by another
workspace whose task or agent lives here — both NOT NULL FKs with
ON DELETE CASCADE, so the later task/agent deletes remove exactly those.
Two indexes make the new shapes index-eligible:
- 273 agent_task_queue(runtime_id): every existing runtime_id index is
partial, so all-status consumers had none. This also covers the FK's
own ON DELETE CASCADE probe, which runs per deleted agent_runtime row
on the ordinary delete path.
- 274 task_token(workspace_id): the column has never been indexed.
Measured block usage for the same workspace, before → after:
agent_task_queue discovery 25,665 → 14,485 blocks; task_token sweep
4,086 → 3 blocks. The point is not the ratio at this size but that the
cost now scales with the workspace being deleted instead of with the
global table.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): fence, bound and un-cascade the teardown task sweep
Addresses all four blocking findings from the MUL-5999 review.
1. Concurrency (correctness / tenant isolation). The task set is no longer
a lock-free snapshot reused across steps. Teardown now locks every row
through which a task can belong to the workspace — its agents, issues
and runtimes — FOR UPDATE, and holds them for the rest of the
transaction. Inserting an agent_task_queue row takes FOR KEY SHARE on
each referenced owner, which conflicts with FOR UPDATE, so a concurrent
enqueue (or a reassignment onto one of these runtimes) can no longer
slip a task in behind the sweep. Locking the tasks alone could not do
that: a row that does not exist yet cannot be locked. This is the same
technique the handler already uses on the workspace row to keep
CreateChatSession out of the delete window.
The per-batch page query also takes FOR UPDATE, so under READ COMMITTED
PostgreSQL re-checks ownership after each lock: a task a concurrent
commit already moved elsewhere is skipped rather than deleted on a
stale claim, and a task we locked cannot move before we delete it.
Deadlock (40P01) joins lock_timeout (55P03) as a retryable 503, because
the owner locks can be taken in the opposite order by a concurrent task
update.
2. Cleanup ownership (no cascade reliance). task_token keeps all three
explicit paths, split into separate index-driven statements instead of
one OR: workspace_id in DeleteWorkspaceLeafData, task_id in
DeleteTaskBatch, agent_id in DeleteTaskTokensByAgent. Nothing in the
teardown graph now depends on ON DELETE CASCADE to finish the job.
3. Bounded memory. The whole-workspace task array is gone. The sweep pages
through one owner key at a time with LIMIT 1000 and deletes exactly the
page it read, so process memory, the array parameter and each DELETE's
row count are bounded by the batch size rather than by a workspace's
lifetime task count. The task-keyed arms moved out of the leaf-data
statement into DeleteTaskBatch, which runs per page, and the separate
whole-workspace task delete is gone.
4. Migration retryability. The concurrent-index cleanup hooks are now
derived from a version -> index-name map that covers 257, 261 and
273-277, so an interrupted CREATE INDEX CONCURRENTLY cannot be mistaken
for success on retry.
Three more indexes, each an unindexed FK referencing column that the
per-batch statements need and that the FK's own ON DELETE action already
scanned per row on ordinary deletes: task_token(agent_id),
chat_draft_restore(task_id), autopilot_run(task_id).
Plans re-verified on a 1.2M-row synthetic copy of the prod shape with the
child tables populated: page query is a Bitmap Index Scan under LockRows;
task_usage, chat_draft_restore, task_token and autopilot_run arms are all
index scans.
Tests: the ownership-path test now asserts the sweep is self-sufficient
inside the transaction, before any owner row is deleted, so a cascade
cannot stand in for teardown. New tests cover the enqueue / reassignment
fence (blocked writers, unrelated writer unaffected), multi-page sweeps
and the page bound, plus a migration-retry repair test and a check that
every cleanup hook names the index its migration builds.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): keyset-page the teardown sweep and fail closed on leaks
Addresses the second review round on MUL-5999.
1. Bounded scan, not just bounded result. The page query was
`owner = $1 ORDER BY id LIMIT n` against indexes that cannot produce id
order, so the executor had to read the owner's whole task set before
applying LIMIT — quadratic in pages for a busy owner, which is the case
this issue is about. Measured on a 550k-row probe: the old shape spent
11,630 buffers per page (planner chose a full pkey index scan; on other
statistics it chose an explicit Sort), the keyset shape spends ~2,000
and a later page costs the same as the first.
Pages are now `owner = $1 AND id > $cursor ORDER BY id LIMIT n` against
composite indexes: 273 becomes (runtime_id, id), 278 adds (agent_id, id),
279 adds (issue_id, id) and 280 drops the single-column issue_id index it
supersedes so the hot table does not carry both.
2. Owner enumeration is bounded too. The three LockWorkspaceTaskOwner*
statements no longer return ids at all — they are a set-based write fence
and now cost the API process nothing. Owners are walked separately in
keyset pages of 500 (281-283 add the (workspace_id, id) indexes those
pages need), so a workspace with a huge agent or issue set never has its
whole owner list held in memory.
The keyset column is id rather than a natural key on purpose: a cursor
column must be immutable, and renaming an owner mid-teardown would
otherwise let it sort behind the cursor and escape the sweep.
3. The fence is no longer the only guarantee. Blocking a concurrent enqueue
still relies on an INSERT taking FOR KEY SHARE on the owner rows it
references, which is a property of the legacy FKs rather than a protocol
every enqueue path opts into. Two things now backstop it:
- every owner is re-swept until a pass finds nothing, and teardown fails
closed (rolling the whole transaction back) if tasks keep arriving,
instead of committing a partially cleaned tenant;
- a schema-contract test asserts the three owner FKs still exist, so a
migration that removes them turns the missing fence into a red build
rather than a silent hole.
An explicit deleting-state protocol shared with all seven enqueue paths
remains the right long-term fix and is out of scope here.
End-to-end teardown, real handler, identical synthetic data, main vs this
branch (1.2M-row agent_task_queue, 200k task_token, 300k task_usage):
small workspace, 5k tasks: 29,247 ms -> 1,013 ms (4.38M -> 0.27M blocks)
busy workspace, 60k tasks: 243,863 ms -> 1,724 ms (5.76M -> 2.23M blocks)
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): fence task writes in-transaction, independent of the FKs
Addresses the third review round on MUL-5999.
1. The re-scan did not close the window, and the reviewer is right that it
cannot: a pass that comes back empty returns success, and a task
inserted after that final scan but before COMMIT is not caught by any
number of passes. The only thing that closed that window was the
FOR KEY SHARE an INSERT takes on the agent / issue / runtime rows it
references — a side effect of foreign keys that workspace_delete.sql
itself documents as an expand-phase net a later contract removes.
Migration 284 replaces it with an explicit fence: a BEFORE INSERT OR
UPDATE OF (agent_id, issue_id, runtime_id) trigger on agent_task_queue
that takes FOR KEY SHARE on the workspace row behind every owner the row
references, and raises if one is gone. That conflicts with the FOR UPDATE
LockWorkspaceForDelete already holds for the whole teardown transaction,
so a task write against a workspace being torn down blocks until teardown
finishes and then either proceeds (rolled back) or fails closed
(committed).
A trigger rather than a per-call-site check because the fence has to be
in the same transaction as the write: there are seven INSERTs and several
ownership-changing UPDATEs across agent.sql, autopilot.sql, chat.sql and
runtime.sql, several of them single autocommit statements, so a separate
"take the fence" statement from the caller would be its own transaction
and protect nothing. In-transaction by construction, covers every current
path, and cannot be forgotten by a path added later.
TestTaskWriteFence_BlocksOnWorkspaceLockAlone isolates the fence from the
FKs: the holder locks ONLY the workspace row — no agent, issue or runtime
row — so no FK on agent_task_queue has anything to block on, since none of
them references workspace. Inserts through all three ownership paths and a
reassignment all block; an unrelated workspace's insert and a status-only
update still pass. Dropping the trigger fails that test.
Cost, measured with 5,000 bare inserts: 147 ms without the fence,
238 ms with it — about 18 us per inserted row for up to three indexed
lookups and a shared row lock. It does not fire on the hot claim /
dispatch / complete path, which updates status and timestamps rather than
ownership.
The verification passes stay as a self-check, no longer described as the
concurrency guarantee.
2. Migration 280 is withdrawn. Its down path rebuilt the superseded index
with CREATE INDEX CONCURRENTLY, and the runner only runs INVALID-index
cleanup hooks in the up direction, so an interrupted rollback would leave
an invalid index that a later `down` skips — and rolling back 279 too
would then leave no usable issue_id index at all. The redundant
single-column index stays for now; dropping it needs its own migration
with a direction-aware cleanup. This also settles the stale comment in
task_usage.sql, which still names the old index and remains correct.
3. Keyset cursors no longer start from the all-zero uuid, which is itself a
valid uuid and would have made a row with that id invisible forever.
Every walk now has an explicit first-page query with no cursor predicate.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(workspace): move the task-write fence into the write statements
Addresses the fourth review round on MUL-5999.
1. The trigger is gone. Migration 284 now defines
lock_task_owner_workspaces(agent_id, issue_id, runtime_id), a predicate
that every statement writing agent_task_queue ownership calls in its own
WHERE clause:
INSERT INTO agent_task_queue (...)
SELECT ... WHERE lock_task_owner_workspaces($1, $3, $2)
Nine statements now carry it: the seven task INSERTs (agent.sql x5,
autopilot.sql, chat.sql), LinkTaskToIssue, and ReassignTasksToRuntime —
the last through a MATERIALIZED CTE so a bulk UPDATE evaluates the
VOLATILE predicate once instead of per row. Clearing runtime_id is
exempt: it cannot move a task into another workspace.
Calling it from the writing statement keeps the lock in the writer's own
transaction, which a separate caller-issued statement could not do for
the autocommit enqueue paths. TestTaskOwnershipWritesCallTheFence is the
trade for dropping the trigger: it scans the query files and fails the
build if a statement writes ownership without the fence.
2. The fence no longer defers to the foreign keys. The trigger version
counted only owner references that still resolved, so a deleted owner
made itself invisible: with all three gone it returned early, and with
one of three gone the count still matched. The predicate now compares
resolved owners against the number of owner arguments supplied and
returns false on any mismatch, so a stale write is rejected on its own
terms. TestLockTaskOwnerWorkspaces_RejectsUnresolvableOwner covers each
shape, including the one-of-three case.
3. Deterministic lock order. The workspace rows are locked with
ORDER BY w.id, so mirrored cross-workspace writes cannot take the same
pair in opposite orders. Honest note on the test: 25 mirrored concurrent
pairs never deadlock either way, because the current plan happens to
return the rows in a stable order — the ORDER BY is what makes that a
guarantee rather than an accident of the plan, and the test is a
regression test for the symptom.
4. The fail-closed test now tests what its name says. The previous one only
inserted a neighbour-owned task and could not have failed. The
replacement runs the actual race: a writer calls the real
CreateAgentTask, blocks on the workspace lock, teardown deletes the
workspace and commits, and the writer must come back with pgx.ErrNoRows
— the fence's own verdict. An FK doing the work instead surfaces as
SQLSTATE 23503, which the assertion rejects explicitly.
Cost: 5,000 predicate calls in 53.7 ms, about 11 us per fenced write, for
up to three indexed lookups and a shared row lock. Status, dispatch and
completion updates do not call it.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): stop the legacy merge when the task fence refuses
Addresses the P1 from the fifth review round on MUL-5999.
ReassignTasksToRuntime returned only a row count, so a fenced refusal was
indistinguishable from "the old runtime had no tasks". mergeLegacyRuntimes
read both as success and went on to DeleteAgentRuntime(old), where
agent_task_queue's ON DELETE CASCADE silently took the history the
reassignment exists to preserve — and once that FK goes away, the same path
would instead leave tasks pointing at a deleted runtime. Either way the
merge was leaning on the cascade, which is what this PR set out to stop.
The query now returns the fence verdict alongside the count:
SELECT (SELECT ok FROM fence) AS fence_ok,
(SELECT count(*) FROM reassigned) AS reassigned_tasks
and the merge is a single helper, mergeLegacyRuntime, that aborts with
errRuntimeMergeFenced before touching anything else when fence_ok is false.
Tasks now move FIRST, because that is the step carrying the fence: nothing
else in the merge — agent reassignment, legacy daemon_id bookkeeping, the
old runtime delete — runs unless the fence agreed. The next daemon
registration re-runs the merge, by which time the workspace is either gone
or its teardown rolled back.
TestMergeLegacyRuntime_KeepsOldRuntimeWhenFenceRefuses covers it: with a
vanished merge target the old runtime and its task both survive and the task
stays on the old runtime, while a merge onto a live runtime still completes.
Removing the abort fails that test — and fails it with SQLSTATE 23503 from
agent_runtime_id_fkey, which is precisely the "an FK is doing the work"
shape the review objected to.
Also replaced the fixed 300 ms sleep in the queued-writer race with a poll
on pg_stat_activity for a backend actually parked on a lock, so the test
cannot pass without the race being set up (it now fails on timeout instead).
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): make the legacy runtime merge a single transaction
Addresses the P1 from the sixth review round on MUL-5999.
The merge ran four autocommit statements, so the fence's workspace lock
ended with the statement that took it. Two ways that left a tenant
inconsistent while daemon registration still reported success:
* a transient failure on the agent reassignment committed the task move
and nothing else, leaving tasks on the new runtime and their agents on
the old one until some later registration happened to retry;
* in the gap after the task commit, a workspace teardown could lock and
delete the target runtime — and its UnbindTasksFromRuntime step clears
runtime_id on exactly the tasks that had just moved. The next merge
looks tasks up by the OLD runtime id, so that history was detached for
good.
mergeLegacyRuntime now opens a transaction and runs the whole merge on one
qtx: fenced task reassignment, agent reassignment, the legacy daemon_id
audit write, and the old-runtime delete, then commits. The fence stays the
first statement, so the workspace lock it takes is held until COMMIT — a
teardown cannot start deleting the target mid-merge — and the lock order
against teardown is unchanged. Any failure returns an error and rolls the
whole thing back.
The audit write is no longer best-effort: inside a transaction a failed
statement poisons the rest, so it either lands with the merge or the merge
is abandoned.
TestMergeLegacyRuntime_RollsBackWhenAStepFails injects the failure in the
database rather than through a code seam — a trigger makes the agent move
raise — and asserts the task, the agent, both runtime rows and the audit
column are all exactly as they were, then that the same merge completes
once the injection is removed. Reverting to autocommit fails it with
"task moved to <new runtime>; the task reassignment was not rolled back".
Also took the non-blocking suggestion: the queued-writer race now has the
writer report its own pg_backend_pid() over a channel and waits for that
specific backend to park on a lock, instead of matching statement text
against every backend on a shared test database.
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* fix(runtime): fence late writes against the runtime a merge is collapsing
Addresses the P1 from the seventh review round on MUL-5999, which the
reviewer reproduced rather than inferred.
The fence locked the workspace row and nothing else. A task write takes
FOR KEY SHARE there, and so did the legacy runtime merge — and two KEY SHARE
locks do not conflict. So an enqueue against the OLD runtime went straight
through after the merge had scanned its tasks, and DeleteAgentRuntime then
removed that brand-new task through agent_task_queue's ON DELETE CASCADE.
Making the merge one transaction widened the window rather than creating it:
the agent's new runtime_id is invisible until COMMIT, so concurrent writers
keep resolving the old runtime for the whole merge.
Both sides of the contract now lock the owner rows too, in one order:
workspace rows (by id), then agent, issue, runtime.
* lock_task_owner_rows (migration 284, renamed from
lock_task_owner_workspaces) takes FOR KEY SHARE on each owner row after
the workspace rows, and re-checks that they still exist. That is what
conflicts with a delete.
* The merge takes FOR KEY SHARE on the workspace and then FOR UPDATE on
both runtime rows in id order, before the task scan, and holds them to
COMMIT. FOR UPDATE is the level that conflicts with a writer's KEY
SHARE; both runtimes in one ordered statement so two merges running in
opposite directions cannot take the pair in opposite orders. Locking
fewer rows than asked for means one disappeared, and the merge is
abandoned.
A late writer therefore either commits before the merge takes the lock — in
which case the scan that follows sees its task and moves it — or waits, and
then finds the runtime gone and writes no row.
TestRuntimeMerge_LateWriteToOldRuntimeIsFencedNotCascaded drives the merge
statement by statement so it can stop where the window was (after the agent
update, before the old runtime is deleted), enqueues against the old runtime
through the generated query, and requires that the writer is parked on a
lock, that it comes back as pgx.ErrNoRows, and that the merge's own history
ends up on the target. It explicitly rejects both 23503 and a row that
lands and then vanishes.
Both halves are load-bearing: dropping the runtime lock from the fence, or
the FOR UPDATE from the merge, fails that test — the latter with "the late
write was rejected by a foreign key (23503), not by the fence".
Cost: 5,000 predicate calls in 74.4 ms, ~15 us per fenced write (was ~11 us
with the workspace lock alone).
Refs MUL-5999
Co-authored-by: multica-agent <[email protected]>
* docs(workspace): correct the fence comment and drop a trailing newline
Two review nits, no behaviour change.
lockWorkspaceTaskOwners still described migration 284 as a BEFORE trigger on
agent_task_queue. It became a called predicate, lock_task_owner_rows, in
|