Files
multica/server/internal/handler/daemon_context_exhausted_test.go
13eb4bda93 MUL-8154 fix(daemon): enforce runtime ownership on daemon routes (#9423)
* fix(daemon): enforce runtime ownership on daemon routes (MUL-8154)

/api/daemon/* accepts any workspace member's PAT/JWT, and authorization only
checked workspace membership. Any member could claim another member's tasks
(receiving the agent's custom_env, MCP config and an owner-scoped task token),
write task results/messages, and take a runtime over by re-registering its
daemon_id.

- Runtime-level check on every runtime/task daemon route, HTTP heartbeat,
  deregister, batch claim (and so the WS claim RPC) and the WS handshake:
  user credentials must own the runtime (NULL owner fails closed); daemon
  tokens must match a non-empty runtime daemon_id.
- Register: pre-check that every existing row on the daemon_id belongs to the
  caller, plus a required_owner_id fence in both upserts (incl. the
  failed-profile path) so a concurrent registration cannot win the row.
  Daemon-token registration keeps the existing owner.
- Legacy merge: only for user-credential registrations, and only when the
  caller owns both rows, verified on the FOR UPDATE-locked rows.
- Tasks without a runtime are not writable through daemon routes; status
  stays readable.
- GetTaskStatus / WS lease queries carry runtime owner/daemon so the hot
  paths keep one round trip.

Co-authored-by: multica-agent <[email protected]>

* test(daemon): authenticate fixtures as the runtime owner (MUL-8154)

Most handler tests drove daemon routes with a daemon token whose daemon_id
did not match the fixture runtime, relying on the workspace-wide token scope
this fix removes. They now authenticate the way real daemons do, with the
runtime owner's user credential (newOwnerDaemonRequest). Tests about daemon
token scoping keep mdt_ and get fixtures with a matching daemon_id; the
ownerless-runtime tests also assert that a user credential is refused.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): serialize registration per daemon and keep Qoder WS claims working (MUL-8154)

Review of #9423:

- P1: the owner pre-check and the per-provider/profile upserts hit different
  conflict keys, so two members registering different providers of the same
  daemon could both pass the pre-check and both insert, leaving one daemon
  with two owners (and a daemon token for either runtime reaching the
  other's). Registration now runs the pre-check and every runtime write in one
  transaction under a pg_advisory_xact_lock on (workspace_id, daemon_id), on a
  single connection so queued registrations cannot starve the holder. Legacy
  merges, analytics and the register event run after commit; failed-profile
  records use savepoints so they stay best-effort.
- P2: the Qoder revocation check read the Authorization header, which the WS
  RPC bridge's in-process request does not carry, so valid Qoder WS claims
  silently returned nothing. DaemonAuth now records the mdt_ token hash in the
  request context, the WS identity keeps it, the RPC bridge re-attaches it,
  and the check validates that hash on every call (revocation still applies
  to an open connection).
- Regression tests for cross-provider and provider+profile concurrent
  registration, and for valid/revoked Qoder credentials over WS RPC.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): bind daemon tokens to the runtime owner they were issued for (MUL-8154)

Second review of #9423: a daemon_id can be re-registered by another member
after its runtimes are deleted, and a daemon token was bound only to
(workspace, daemon_id), so the previous owner's Remote MCP token reached the
new owner's runtime (heartbeat, claim, owner-scoped task token).

- daemon_token.owner_id (migration 588). Remote MCP tokens are bound to the
  locked runtime owner inside FinalizeTaskClaim; live tokens are backfilled
  only where their daemon has exactly one owner.
- DaemonAuth carries the owner binding (DB and cache; cache key bumped to v2
  so pre-upgrade entries are never read), the WS identity keeps it and the
  RPC bridge replays it.
- Daemon-token authorization now also requires token owner == runtime owner.
  Qoder-managed tokens stay unbound (governed by cloud_agent_connection), and
  user credentials can no longer register the reserved qoder-managed-* ids.
- Tests: the reuse lifecycle over HTTP, a new WS handshake and an already
  open WS connection; the reserved daemon_id; the concurrency barrier now
  hooks the registration transaction (TxStarter) so the interleaving is real.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): owner-bind daemon-token registration; drop unsafe token backfill (MUL-8154)

Third review of #9423:

- Migration 588 no longer backfills daemon_token.owner_id from the daemon's
  current owner: that owner may be a later re-registrant, which re-bound the
  previous owner's live token to them. Existing non-Qoder tokens stay NULL and
  are refused (24h Remote MCP credentials, re-issued on the next claim).
- owner_id is a plain UUID column, no foreign key / cascade (AGENTS.md).
- Register via a non-Qoder daemon token now uses the token's owner for the
  in-transaction pre-check and the SQL owner fence, and refuses unbound
  tokens. Qoder keeps its connection-based check; daemon tokens still never
  trigger legacy merges.
- Tests: an old owner's bound token and an unbound legacy token can neither
  register over nor heartbeat the replacement owner's runtime.

Co-authored-by: multica-agent <[email protected]>

* fix(daemon): grant the Qoder registration exemption only with a live connection (MUL-8154)

Fourth review of #9423: register skipped the token owner binding for any
qoder-managed-* daemon token, but checked the Qoder connection only when the
body declared a qoder_cloud runtime. A pre-upgrade Remote MCP token under that
prefix (no owner, no connection) could register type=claude and rewrite
another member's runtime.

Register now verifies the token against a live cloud_agent_connection before
anything else whenever the token is in the Qoder namespace, regardless of the
declared runtimes, and restricts that namespace to the bridge's own shape: a
built-in qoder_cloud runtime, no custom profiles, no failed_profiles.

Test covers built-in, custom-profile, failed-profiles-only and qoder_cloud
bodies with an unconnected token, plus the connected token's allowed and
refused shapes.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: multica-eve <[email protected]>
Co-authored-by: multica-agent <[email protected]>
2026-10-22 16:47:06 +08:00

228 lines
8.7 KiB
Go

package handler
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"testing"
"github.com/go-chi/chi/v5"
"github.com/jackc/pgx/v5"
db "github.com/multica-ai/multica/server/pkg/db/generated"
"github.com/multica-ai/multica/server/pkg/taskfailure"
)
// TestCompleteTask_ContextExhaustionFromOlderDaemonIsRecordedAsFailed is the
// mixed-version regression for GH #6402.
//
// A daemon whose backend does not read the provider's structured terminal
// reason reports a context-exhausted run through /complete, with the CLI's
// "your context window is full" notice as the answer. Installed daemons upgrade
// on their own cadence, so without a server-side rule an un-upgraded host keeps
// publishing that notice as the agent's reply AND keeps the dead session as the
// resume pointer — every later trigger on the issue resumes it and reproduces
// the same non-answer. That is a permanently stuck (agent, issue) pair, not a
// mislabelled row, which is why the normalization lives at the boundary rather
// than only in the daemon.
//
// Asserts the whole chain: the task lands failed with the reason the resume
// blacklist covers, and the saturated session stops being handed to the next
// task.
func TestCompleteTask_ContextExhaustionFromOlderDaemonIsRecordedAsFailed(t *testing.T) {
if testHandler == nil || testPool == nil {
t.Skip("database not available")
}
ctx := context.Background()
var agentID, runtimeID string
if err := testPool.QueryRow(ctx, `
SELECT a.id, a.runtime_id FROM agent a WHERE a.workspace_id = $1 LIMIT 1
`, testWorkspaceID).Scan(&agentID, &runtimeID); err != nil {
t.Fatalf("setup: get agent: %v", err)
}
var issueID string
if err := testPool.QueryRow(ctx, `
INSERT INTO issue (workspace_id, title, status, priority, creator_id, creator_type, number, position)
VALUES ($1, 'gh-6402 context exhaustion fixture', 'in_progress', 'none', $2, 'member', 6402, 0)
RETURNING id
`, testWorkspaceID, testUserID).Scan(&issueID); err != nil {
t.Fatalf("setup: create issue: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM issue WHERE id = $1`, issueID) })
var taskID string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_task_queue (agent_id, runtime_id, issue_id, status, priority, started_at)
VALUES ($1, $2, $3, 'running', 0, now())
RETURNING id
`, agentID, runtimeID, issueID).Scan(&taskID); err != nil {
t.Fatalf("setup: create task: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM agent_task_queue WHERE id = $1`, taskID) })
// Exactly what an older daemon posts: a successful completion whose output
// is the provider's notice, carrying the saturated session id.
w := httptest.NewRecorder()
req := newOwnerDaemonRequest("POST", "/api/daemon/tasks/"+taskID+"/complete",
map[string]any{
"output": "Prompt is too long · the request is ~274931 tokens (limit 200000) but this conversation is only ~1597 tokens — " +
"the rest is system prompt, tool definitions, and attachment content. A single-exchange conversation cannot be " +
"compacted; reduce attached files/tools or start with less context.",
"session_id": "sess-saturated-6402",
"work_dir": "/tmp/gh6402",
},
)
rctx := chi.NewRouteContext()
rctx.URLParams.Add("taskId", taskID)
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, rctx))
testHandler.CompleteTask(w, req)
if w.Code != http.StatusOK {
t.Fatalf("CompleteTask: expected 200, got %d: %s", w.Code, w.Body.String())
}
var status, failureReason string
if err := testPool.QueryRow(ctx, `
SELECT status, COALESCE(failure_reason, '') FROM agent_task_queue WHERE id = $1
`, taskID).Scan(&status, &failureReason); err != nil {
t.Fatalf("read task: %v", err)
}
if status != "failed" {
t.Fatalf("task status = %q, want failed — a context-exhausted run produced no work", status)
}
if failureReason != string(taskfailure.ReasonAgentContextOverflow) {
t.Fatalf("failure_reason = %q, want %q", failureReason, taskfailure.ReasonAgentContextOverflow)
}
// The payoff: the next task on this (agent, issue) must not be handed the
// session that just proved it cannot answer. This is the automated form of
// the reporter's manual workaround — archiving the transcript on disk.
got, err := testHandler.Queries.GetLastTaskSession(ctx, db.GetLastTaskSessionParams{
AgentID: parseUUID(agentID),
IssueID: parseUUID(issueID),
})
if err == nil && got.SessionID.Valid && got.SessionID.String == "sess-saturated-6402" {
t.Fatal("the saturated session is still offered as the resume pointer")
}
if err != nil && !errors.Is(err, pgx.ErrNoRows) {
t.Fatalf("GetLastTaskSession: %v", err)
}
// And nothing was published as the agent's answer: the run failed, so the
// completion path's fallback comment never ran.
var agentComments int
if err := testPool.QueryRow(ctx, `
SELECT count(*) FROM comment
WHERE issue_id = $1 AND author_type = 'agent' AND author_id = $2 AND type = 'comment'
`, issueID, agentID).Scan(&agentComments); err != nil {
t.Fatalf("count agent comments: %v", err)
}
if agentComments != 0 {
t.Fatalf("expected the provider notice NOT to be posted as an answer, got %d agent comment(s)", agentComments)
}
}
// TestCompleteTask_RealAnswerStillCompletes is the companion guard: outputs a
// real agent can legitimately produce go through the completion path untouched.
//
// The second case is the one that shaped the classifier. "Prompt is too long"
// IS a wording the CLI emits, but it is also a complete, ordinary English
// answer to "is my prompt too long?" — so the success-path predicate must not
// claim it. Matching it would fail a task that succeeded and retire a healthy
// session, which is strictly worse than the miss: the real provider frame
// carries is_error, so that shape reaches the server through /fail, where
// Classify routes it to context_overflow anyway.
func TestCompleteTask_RealAnswerStillCompletes(t *testing.T) {
if testHandler == nil || testPool == nil {
t.Skip("database not available")
}
ctx := context.Background()
var agentID, runtimeID string
if err := testPool.QueryRow(ctx, `
SELECT a.id, a.runtime_id FROM agent a WHERE a.workspace_id = $1 LIMIT 1
`, testWorkspaceID).Scan(&agentID, &runtimeID); err != nil {
t.Fatalf("setup: get agent: %v", err)
}
tests := []struct {
name string
number int32
output string
}{
{
name: "an answer that mentions compacting",
number: 6403,
output: "Ran /compact first because the session was getting long, then fixed the redirect and pushed.",
},
{
name: "an answer that IS the provider's bare sentence",
number: 6404,
output: "Prompt is too long",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
var issueID string
if err := testPool.QueryRow(ctx, `
INSERT INTO issue (workspace_id, title, status, priority, creator_id, creator_type, number, position)
VALUES ($1, 'gh-6402 control fixture', 'in_progress', 'none', $2, 'member', $3, 0)
RETURNING id
`, testWorkspaceID, testUserID, tc.number).Scan(&issueID); err != nil {
t.Fatalf("setup: create issue: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM issue WHERE id = $1`, issueID) })
var taskID string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_task_queue (agent_id, runtime_id, issue_id, status, priority, started_at)
VALUES ($1, $2, $3, 'running', 0, now())
RETURNING id
`, agentID, runtimeID, issueID).Scan(&taskID); err != nil {
t.Fatalf("setup: create task: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM agent_task_queue WHERE id = $1`, taskID) })
w := httptest.NewRecorder()
req := newOwnerDaemonRequest("POST", "/api/daemon/tasks/"+taskID+"/complete",
map[string]any{"output": tc.output, "session_id": "sess-healthy"},
)
rctx := chi.NewRouteContext()
rctx.URLParams.Add("taskId", taskID)
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, rctx))
testHandler.CompleteTask(w, req)
if w.Code != http.StatusOK {
t.Fatalf("CompleteTask: expected 200, got %d: %s", w.Code, w.Body.String())
}
var status string
if err := testPool.QueryRow(ctx, `
SELECT status FROM agent_task_queue WHERE id = $1
`, taskID).Scan(&status); err != nil {
t.Fatalf("read task: %v", err)
}
if status != "completed" {
t.Fatalf("task status = %q, want completed", status)
}
// And the healthy session stays available to the next task.
got, err := testHandler.Queries.GetLastTaskSession(ctx, db.GetLastTaskSessionParams{
AgentID: parseUUID(agentID),
IssueID: parseUUID(issueID),
})
if err != nil {
t.Fatalf("GetLastTaskSession: %v", err)
}
if got.SessionID.String != "sess-healthy" {
t.Fatalf("expected the healthy session to remain resumable, got %q", got.SessionID.String)
}
})
}
}