Files
13eb4bda93 MUL-8154 fix(daemon): enforce runtime ownership on daemon routes (#9423)
* fix(daemon): enforce runtime ownership on daemon routes (MUL-8154)

/api/daemon/* accepts any workspace member's PAT/JWT, and authorization only
checked workspace membership. Any member could claim another member's tasks
(receiving the agent's custom_env, MCP config and an owner-scoped task token),
write task results/messages, and take a runtime over by re-registering its
daemon_id.

- Runtime-level check on every runtime/task daemon route, HTTP heartbeat,
  deregister, batch claim (and so the WS claim RPC) and the WS handshake:
  user credentials must own the runtime (NULL owner fails closed); daemon
  tokens must match a non-empty runtime daemon_id.
- Register: pre-check that every existing row on the daemon_id belongs to the
  caller, plus a required_owner_id fence in both upserts (incl. the
  failed-profile path) so a concurrent registration cannot win the row.
  Daemon-token registration keeps the existing owner.
- Legacy merge: only for user-credential registrations, and only when the
  caller owns both rows, verified on the FOR UPDATE-locked rows.
- Tasks without a runtime are not writable through daemon routes; status
  stays readable.
- GetTaskStatus / WS lease queries carry runtime owner/daemon so the hot
  paths keep one round trip.

Co-authored-by: multica-agent <[email protected]>

* test(daemon): authenticate fixtures as the runtime owner (MUL-8154)

Most handler tests drove daemon routes with a daemon token whose daemon_id
did not match the fixture runtime, relying on the workspace-wide token scope
this fix removes. They now authenticate the way real daemons do, with the
runtime owner's user credential (newOwnerDaemonRequest). Tests about daemon
token scoping keep mdt_ and get fixtures with a matching daemon_id; the
ownerless-runtime tests also assert that a user credential is refused.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): serialize registration per daemon and keep Qoder WS claims working (MUL-8154)

Review of #9423:

- P1: the owner pre-check and the per-provider/profile upserts hit different
  conflict keys, so two members registering different providers of the same
  daemon could both pass the pre-check and both insert, leaving one daemon
  with two owners (and a daemon token for either runtime reaching the
  other's). Registration now runs the pre-check and every runtime write in one
  transaction under a pg_advisory_xact_lock on (workspace_id, daemon_id), on a
  single connection so queued registrations cannot starve the holder. Legacy
  merges, analytics and the register event run after commit; failed-profile
  records use savepoints so they stay best-effort.
- P2: the Qoder revocation check read the Authorization header, which the WS
  RPC bridge's in-process request does not carry, so valid Qoder WS claims
  silently returned nothing. DaemonAuth now records the mdt_ token hash in the
  request context, the WS identity keeps it, the RPC bridge re-attaches it,
  and the check validates that hash on every call (revocation still applies
  to an open connection).
- Regression tests for cross-provider and provider+profile concurrent
  registration, and for valid/revoked Qoder credentials over WS RPC.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): bind daemon tokens to the runtime owner they were issued for (MUL-8154)

Second review of #9423: a daemon_id can be re-registered by another member
after its runtimes are deleted, and a daemon token was bound only to
(workspace, daemon_id), so the previous owner's Remote MCP token reached the
new owner's runtime (heartbeat, claim, owner-scoped task token).

- daemon_token.owner_id (migration 588). Remote MCP tokens are bound to the
  locked runtime owner inside FinalizeTaskClaim; live tokens are backfilled
  only where their daemon has exactly one owner.
- DaemonAuth carries the owner binding (DB and cache; cache key bumped to v2
  so pre-upgrade entries are never read), the WS identity keeps it and the
  RPC bridge replays it.
- Daemon-token authorization now also requires token owner == runtime owner.
  Qoder-managed tokens stay unbound (governed by cloud_agent_connection), and
  user credentials can no longer register the reserved qoder-managed-* ids.
- Tests: the reuse lifecycle over HTTP, a new WS handshake and an already
  open WS connection; the reserved daemon_id; the concurrency barrier now
  hooks the registration transaction (TxStarter) so the interleaving is real.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): owner-bind daemon-token registration; drop unsafe token backfill (MUL-8154)

Third review of #9423:

- Migration 588 no longer backfills daemon_token.owner_id from the daemon's
  current owner: that owner may be a later re-registrant, which re-bound the
  previous owner's live token to them. Existing non-Qoder tokens stay NULL and
  are refused (24h Remote MCP credentials, re-issued on the next claim).
- owner_id is a plain UUID column, no foreign key / cascade (AGENTS.md).
- Register via a non-Qoder daemon token now uses the token's owner for the
  in-transaction pre-check and the SQL owner fence, and refuses unbound
  tokens. Qoder keeps its connection-based check; daemon tokens still never
  trigger legacy merges.
- Tests: an old owner's bound token and an unbound legacy token can neither
  register over nor heartbeat the replacement owner's runtime.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): grant the Qoder registration exemption only with a live connection (MUL-8154)

Fourth review of #9423: register skipped the token owner binding for any
qoder-managed-* daemon token, but checked the Qoder connection only when the
body declared a qoder_cloud runtime. A pre-upgrade Remote MCP token under that
prefix (no owner, no connection) could register type=claude and rewrite
another member's runtime.

Register now verifies the token against a live cloud_agent_connection before
anything else whenever the token is in the Qoder namespace, regardless of the
declared runtimes, and restricts that namespace to the bridge's own shape: a
built-in qoder_cloud runtime, no custom profiles, no failed_profiles.

Test covers built-in, custom-profile, failed-profiles-only and qoder_cloud
bodies with an unconnected token, plus the connected token's allowed and
refused shapes.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: multica-eve <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-10-22 16:47:06 +08:00

313 lines
16 KiB
Go

package handler
import (
"context"
"fmt"
"net/http"
"testing"
"github.com/multica-ai/multica/server/internal/events"
"github.com/multica-ai/multica/server/internal/testutil"
"github.com/multica-ai/multica/server/pkg/protocol"
)
func agentMention(name, id string) string {
return fmt.Sprintf("[@%s](mention://agent/%s)", name, id)
}
func postSteeredComment(t *testing.T, issueID, parentID, content string, steerTaskIDs ...string) *testutil.Response {
t.Helper()
body := map[string]any{"content": content, "steer_task_ids": steerTaskIDs}
if parentID != "" {
body["parent_id"] = parentID
}
return testutil.Call(t, testHandler.CreateComment, withURLParam(
newRequest(http.MethodPost, "/api/issues/"+issueID+"/comments", body), "id", issueID))
}
// runningSteerableTask starts another agent's running, steer-capable turn on
// the fixture's issue.
func runningSteerableTask(t *testing.T, f supplementFixture, name string) (agentID, taskID string) {
t.Helper()
runtimeID := dbfx.Runtime(t, "steer-"+name, testutil.Cols{"provider": "claude"})
agentID = dbfx.Agent(t, name, runtimeID)
taskID = dbfx.Task(t, agentID, testutil.Cols{
"issue_id": f.issueID, "runtime_id": runtimeID, "status": "running", "started_at": testutil.Raw("now()"),
})
dbfx.Exec(t, `INSERT INTO task_supplement_capability (task_id, workspace_id, issue_id, capability) VALUES ($1, $2, $3, $4)`,
taskID, testWorkspaceID, f.issueID, protocol.DaemonCapabilityTaskSupplementV1)
dbfx.Cleanup(t, `DELETE FROM task_supplement_capability WHERE task_id = $1`, taskID)
dbfx.Cleanup(t, `DELETE FROM task_supplement WHERE task_id = $1`, taskID)
return agentID, taskID
}
func completeTaskViaDaemon(t *testing.T, taskID string) {
t.Helper()
req := newOwnerDaemonRequest(http.MethodPost, "/api/daemon/tasks/"+taskID+"/complete",
map[string]any{"output": "done"})
testutil.Call(t, testHandler.CompleteTask, withURLParam(req, "taskId", taskID)).Want(http.StatusOK)
}
func outcomeFor(outcomes []CommentTriggerOutcome, agentID string) (CommentTriggerOutcome, bool) {
for _, outcome := range outcomes {
if outcome.TargetType == "agent" && outcome.TargetID == agentID {
return outcome, true
}
}
return CommentTriggerOutcome{}, false
}
func TestCreateCommentSteersTheThreadAgentsRunningTurn(t *testing.T) {
for _, provider := range []string{"codex", "grok", "cursor", "kiro"} {
t.Run(provider, func(t *testing.T) {
f := newSupplementFixture(t, provider, "running", true)
dbfx.Exec(t, `UPDATE comment SET content = $2 WHERE id = $1`, f.triggerID,
agentMention("Supplement", f.agentID)+" original objective")
dbfx.Exec(t, `UPDATE agent_task_queue SET delivered_comment_ids = ARRAY[trigger_comment_id] WHERE id = $1`, f.taskID)
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
var created CommentResponse
postSteeredComment(t, f.issueID, f.triggerID, "Only fix web.", f.taskID).Want(http.StatusCreated).JSON(&created)
if len(created.Supplements) != 1 {
t.Fatalf("receipts = %+v, want one for the running turn", created.Supplements)
}
receipt := created.Supplements[0]
if receipt.TaskID != f.taskID || receipt.AgentID != f.agentID || receipt.Status != "pending" {
t.Fatalf("receipt = %+v, want pending for task %s / agent %s", receipt, f.taskID, f.agentID)
}
if created.SupplementTaskID != f.taskID || created.SupplementStatus != "pending" {
t.Fatalf("legacy receipt fields = %s/%s, want the first receipt", created.SupplementTaskID, created.SupplementStatus)
}
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE issue_id = $1`, f.issueID); n != 1 {
t.Fatalf("steering created %d runs, want the running turn only", n)
}
completeTaskViaDaemon(t, f.taskID)
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE issue_id = $1`, f.issueID); n != 1 {
t.Fatalf("completion replayed a steered comment into %d runs", n)
}
})
}
}
func TestCreateCommentSteerKeepsTheNormalTriggerWhenTheTurnCannotTakeInput(t *testing.T) {
for _, tc := range []struct {
name string
status string
negotiated bool
}{
{name: "turn without the capability", status: "running", negotiated: false},
{name: "turn not started", status: "dispatched", negotiated: true},
} {
t.Run(tc.name, func(t *testing.T) {
f := newSupplementFixture(t, "codex", tc.status, tc.negotiated)
dbfx.Exec(t, `UPDATE comment SET content = $2 WHERE id = $1`, f.triggerID,
agentMention("Supplement", f.agentID)+" original objective")
dbfx.Exec(t, `UPDATE agent_task_queue SET delivered_comment_ids = ARRAY[trigger_comment_id] WHERE id = $1`, f.taskID)
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
var created CommentResponse
postSteeredComment(t, f.issueID, f.triggerID, "Only fix web.", f.taskID).Want(http.StatusCreated).JSON(&created)
if len(created.Supplements) != 0 {
t.Fatalf("bound a turn that cannot take input: %+v", created.Supplements)
}
if n := dbfx.Count(t, `SELECT count(*) FROM task_supplement WHERE comment_id = $1`, created.ID); n != 0 {
t.Fatalf("stored %d receipts", n)
}
if tc.status != "running" {
return
}
// The comment keeps its deferred trigger and becomes the follow-up.
completeTaskViaDaemon(t, f.taskID)
var trigger string
dbfx.QueryRow(t, `SELECT trigger_comment_id FROM agent_task_queue WHERE issue_id = $1 AND id <> $2`,
f.issueID, f.taskID).Scan(&trigger)
if trigger != created.ID {
t.Fatalf("follow-up trigger = %s, want the unsteered comment %s", trigger, created.ID)
}
})
}
}
func TestCreateCommentSteersOnlyRunningRecipients(t *testing.T) {
f := newSupplementFixture(t, "codex", "running", true)
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
idleAgentID := dbfx.Agent(t, "Idle recipient", f.runtimeID)
content := agentMention("Supplement", f.agentID) + " only web; " + agentMention("Idle recipient", idleAgentID) + " check desktop"
var created CommentResponse
postSteeredComment(t, f.issueID, "", content, f.taskID).Want(http.StatusCreated).JSON(&created)
if len(created.Supplements) != 1 || created.Supplements[0].AgentID != f.agentID {
t.Fatalf("receipts = %+v, want the running agent only", created.Supplements)
}
if outcome, ok := outcomeFor(created.TriggerOutcomes, f.agentID); !ok || outcome.Status != DispatchSteered || outcome.ReasonCode != ReasonSteered {
t.Fatalf("running recipient outcome = %+v (%v), want steered", outcome, ok)
}
if outcome, ok := outcomeFor(created.TriggerOutcomes, idleAgentID); !ok || outcome.Status == DispatchSteered || outcome.Status == DispatchBlocked {
t.Fatalf("idle recipient outcome = %+v (%v), want a normal trigger", outcome, ok)
}
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE agent_id = $1 AND trigger_comment_id = $2`, idleAgentID, created.ID); n != 1 {
t.Fatalf("idle recipient has %d runs for the comment, want 1", n)
}
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE agent_id = $1`, f.agentID); n != 1 {
t.Fatalf("steered recipient has %d runs, want its running turn only", n)
}
}
func TestCreateCommentSteersSeveralRunningTurns(t *testing.T) {
f := newSupplementFixture(t, "codex", "running", true)
otherAgentID, otherTaskID := runningSteerableTask(t, f, "Second runner")
content := agentMention("Supplement", f.agentID) + " and " + agentMention("Second runner", otherAgentID) + " stop touching desktop"
var created CommentResponse
postSteeredComment(t, f.issueID, "", content, f.taskID, otherTaskID).Want(http.StatusCreated).JSON(&created)
got := map[string]string{}
for _, receipt := range created.Supplements {
got[receipt.AgentID] = receipt.TaskID
}
if len(created.Supplements) != 2 || got[f.agentID] != f.taskID || got[otherAgentID] != otherTaskID {
t.Fatalf("receipts = %+v, want one per running turn", created.Supplements)
}
var timeline []TimelineEntry
testutil.Call(t, testHandler.ListTimeline, withURLParam(
newRequest(http.MethodGet, "/api/issues/"+f.issueID+"/timeline", nil), "id", f.issueID)).Want(http.StatusOK).JSON(&timeline)
for _, entry := range timeline {
if entry.ID == created.ID && len(entry.Supplements) != 2 {
t.Fatalf("timeline receipts = %+v, want both runs", entry.Supplements)
}
}
}
func TestCompletionReplaysASteeredCommentOnlyForAgentsItDidNotSteer(t *testing.T) {
f := newSupplementFixture(t, "codex", "running", true)
dbfx.Exec(t, `UPDATE agent_task_queue SET delivered_comment_ids = ARRAY[trigger_comment_id] WHERE id = $1`, f.taskID)
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
otherAgentID, otherTaskID := runningSteerableTask(t, f, "Waits its turn")
content := agentMention("Supplement", f.agentID) + " only web; " + agentMention("Waits its turn", otherAgentID) + " next round"
var created CommentResponse
postSteeredComment(t, f.issueID, "", content, f.taskID).Want(http.StatusCreated).JSON(&created)
if len(created.Supplements) != 1 || created.Supplements[0].AgentID != f.agentID {
t.Fatalf("receipts = %+v, want the steered agent only", created.Supplements)
}
completeTaskViaDaemon(t, otherTaskID)
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE agent_id = $1 AND trigger_comment_id = $2`, otherAgentID, created.ID); n != 1 {
t.Fatalf("unsteered recipient got %d follow-ups, want 1", n)
}
completeTaskViaDaemon(t, f.taskID)
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE agent_id = $1`, f.agentID); n != 1 {
t.Fatalf("steered recipient has %d runs, want no follow-up", n)
}
}
func TestCreateCommentSteerNeverRetargetsAnotherTurn(t *testing.T) {
f := newSupplementFixture(t, "codex", "running", true)
dbfx.Exec(t, `UPDATE comment SET content = $2 WHERE id = $1`, f.triggerID, agentMention("A", f.agentID)+" original thread")
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
// The chosen turn ends before the send arrives, and the same agent has
// meanwhile started a turn for another thread.
dbfx.Exec(t, `UPDATE agent_task_queue SET status = 'completed', completed_at = now() WHERE id = $1`, f.taskID)
otherRoot := dbfx.Comment(t, f.issueID, agentMention("A", f.agentID)+" unrelated thread")
successor := dbfx.Task(t, f.agentID, testutil.Cols{
"issue_id": f.issueID, "runtime_id": f.runtimeID, "trigger_comment_id": otherRoot,
"status": "running", "started_at": testutil.Raw("now()"),
})
dbfx.Exec(t, `INSERT INTO task_supplement_capability (task_id, workspace_id, issue_id, capability) VALUES ($1, $2, $3, $4)`,
successor, testWorkspaceID, f.issueID, protocol.DaemonCapabilityTaskSupplementV1)
dbfx.Cleanup(t, `DELETE FROM task_supplement_capability WHERE task_id = $1`, successor)
dbfx.Cleanup(t, `DELETE FROM task_supplement WHERE task_id = $1`, successor)
var sent CommentResponse
postSteeredComment(t, f.issueID, f.triggerID, "Only change the original task", f.taskID).Want(http.StatusCreated).JSON(&sent)
if len(sent.Supplements) != 0 {
t.Fatalf("the chosen turn ended, but the send steered another turn: %+v", sent.Supplements)
}
if n := dbfx.Count(t, `SELECT count(*) FROM task_supplement WHERE task_id = $1`, successor); n != 0 {
t.Fatalf("successor turn received %d steered messages", n)
}
}
func TestCompletionKeepsAnAgentTheAuthorChoseNotToStart(t *testing.T) {
f := newSupplementFixture(t, "codex", "running", true)
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
otherAgent, otherTask := runningSteerableTask(t, f, "Other recipient")
content := agentMention("A", f.agentID) + " and " + agentMention("B", otherAgent) + " only do this once"
dbfx.Exec(t, `UPDATE comment SET content = $2 WHERE id = $1`, f.triggerID, content)
dbfx.Exec(t, `UPDATE agent_task_queue SET trigger_comment_id = $2, comment_thread_id = $2, delivered_comment_ids = ARRAY[$2::uuid] WHERE id = $1`, otherTask, f.triggerID)
// B is working in this thread; the reply goes to A only ("Won't start
// this time" for B, or "Send as a new run" for a receipt A missed).
body := map[string]any{"content": content, "parent_id": f.triggerID, "suppress_agent_ids": []string{otherAgent}}
testutil.Call(t, testHandler.CreateComment, withURLParam(
newRequest(http.MethodPost, "/api/issues/"+f.issueID+"/comments", body), "id", f.issueID)).Want(http.StatusCreated)
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE agent_id = $1`, otherAgent); n != 1 {
t.Fatalf("the reply started another B run: %d", n)
}
completeTaskViaDaemon(t, otherTask)
if n := dbfx.Count(t, `SELECT count(*) FROM agent_task_queue WHERE agent_id = $1`, otherAgent); n != 1 {
t.Fatalf("B's completion replayed the reply into %d runs, want 1", n)
}
}
func TestCreateCommentSteerIgnoresATurnOfAnotherIssue(t *testing.T) {
f := newSupplementFixture(t, "codex", "running", true)
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
// The same agent is also running on another issue.
otherIssueID := dbfx.Issue(t, "Another issue")
elsewhere := dbfx.Task(t, f.agentID, testutil.Cols{
"issue_id": otherIssueID, "runtime_id": f.runtimeID, "status": "running", "started_at": testutil.Raw("now()"),
})
dbfx.Exec(t, `INSERT INTO task_supplement_capability (task_id, workspace_id, issue_id, capability) VALUES ($1, $2, $3, $4)`,
elsewhere, testWorkspaceID, otherIssueID, protocol.DaemonCapabilityTaskSupplementV1)
dbfx.Cleanup(t, `DELETE FROM task_supplement_capability WHERE task_id = $1`, elsewhere)
content := agentMention("Supplement", f.agentID) + " only web"
var created CommentResponse
postSteeredComment(t, f.issueID, "", content, elsewhere).Want(http.StatusCreated).JSON(&created)
if len(created.Supplements) != 0 {
t.Fatalf("receipts = %+v, want none for a turn of another issue", created.Supplements)
}
if n := dbfx.Count(t, `SELECT count(*) FROM task_supplement WHERE task_id = $1`, elsewhere); n != 0 {
t.Fatalf("another issue's turn received %d messages", n)
}
if outcome, ok := outcomeFor(created.TriggerOutcomes, f.agentID); !ok || outcome.Status == DispatchSteered {
t.Fatalf("recipient outcome = %+v (%v), want its normal trigger", outcome, ok)
}
}
func TestUpdateCommentKeepsTheLatestEditsSuppression(t *testing.T) {
f := newSupplementFixture(t, "codex", "running", true)
dbfx.Cleanup(t, `DELETE FROM agent_task_queue WHERE issue_id = $1`, f.issueID)
commentID := dbfx.Comment(t, f.issueID, "initial body")
h := *testHandler
h.Bus = events.New()
laterContent := agentMention("A", f.agentID) + " latest edit"
nested := false
h.Bus.Subscribe(protocol.EventCommentUpdated, func(events.Event) {
if nested {
return
}
nested = true
// A second editor saves right after the first edit's text commits.
later := withURLParam(newRequest(http.MethodPut, "/api/comments/"+commentID, map[string]any{
"content": laterContent, "expected_revision": 2, "suppress_agent_ids": []string{},
}), "commentId", commentID)
testutil.Call(t, h.UpdateComment, later).Want(http.StatusOK)
})
first := withURLParam(newRequest(http.MethodPut, "/api/comments/"+commentID, map[string]any{
"content": agentMention("A", f.agentID) + " first edit", "expected_revision": 1, "suppress_agent_ids": []string{f.agentID},
}), "commentId", commentID)
testutil.Call(t, h.UpdateComment, first).Want(http.StatusOK)
saved, err := testHandler.Queries.GetComment(context.Background(), parseUUID(commentID))
if err != nil {
t.Fatal(err)
}
if saved.Content != laterContent || saved.Revision != 3 {
t.Fatalf("latest body/revision = %q/%d", saved.Content, saved.Revision)
}
if len(saved.SuppressedAgentIds) != 0 {
t.Fatalf("the latest edit starts A, but an earlier edit's choice stayed: %v", saved.SuppressedAgentIds)
}
}