mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-09 01:46:46 +08:00
* fix(agent): own the runtime process tree on every backend Cancelling a task on Windows left the agent running. The daemon marked the task cancelled, revoked its mat_ token, and killed what it thought was the CLI — but on Windows the direct child is usually a cmd.exe shim and the real CLI is already a grandchild, so it survived. One reporter's orphan kept working for 40 minutes, read the daemon owner's profile PAT after its own token started returning 401, and posted comments as the member (GH #7522). Whole-tree ownership already existed on both platforms. It was opt-in, and opt-in rotted: 8 of the 27 places this package starts a process asked for a Unix process group, and 3 of the 23 backends took Windows Job Object ownership. Adding claude to that list would have left 19 backends with the same hole, so the fix is structural instead. - newRuntimeCmd, at the single point where a runtime process is built, now puts every command in its own process group and replaces os/exec's leader-only Cancel with a group-wide kill. A backend that wants a graceful shutdown still assigns its own cmd.Cancel and wins. This also covers the --version and model-discovery probes, which cmd.Output() runs without ever reaching a start helper. - startOwnedProcessTree is now the only way this package starts a process, so Windows Job Object ownership is not something a backend can forget. Each start is paired with releaseProcessGroup after the reap, which drops the job handle and kills whatever outlived it. - Two structural tests enforce both halves, the way TestOnlyLaunchGoSpawnsRuntimeProcesses already enforces the argv boundary: a new backend calling cmd.Start fails, and one that takes ownership without releasing it fails. The Unix half is #5918 for the 20 backends the original fix did not reach: their CLI was never in its own group at all, so no group-wide signal could have reached it. Co-authored-by: multica-agent <[email protected]> * fix(agent): own the process tree for synchronous probes too Review found the ownership boundary had a hole the PR claimed to cover. os/exec's Run, Output and CombinedOutput call Start themselves, so a probe written with cmd.Output() never reaches startOwnedProcessTree and owns nothing on Windows — where the direct child of a `--version` probe is typically the npm shim and the real CLI is already a grandchild. That is the same escape #7522 was reported for, in 15 call sites nobody was looking at; detectCLIVersion's own comment already describes a broken CLI leaving grandchildren that hold the stdout pipe. - runOwned / outputOwned / combinedOutputOwned give a synchronous probe the same start, reap and release a task launch gets. They keep the stdlib contract: stdout returned, stderr attached to the ExitError, combined output through a single writer so os/exec gives it one pipe. The stderr sample is bounded, as os/exec bounds its own. - All 15 probe sites route through them, and the structural guard now rejects Run/Output/CombinedOutput alongside Start. The release guard did not hold its invariant either: it only asked whether a file contained any releaseProcessGroup, so a second unreleased start in a file that already had a pair would pass — and models.go is exactly that shape. It now works per function: one owned start each, paired with at least one release. Equal counts would be the wrong rule, since dsh's Execute legitimately releases on two exit paths. Both guards are now pure functions over an AST, with tests that feed them synthetic sources — including the second-unreleased-start case that the file-level version missed — so the guards fail closed without needing a real regression committed to find out. Adds the Windows probe regression (descendant must be in the job and die with a cancelled probe) plus unix equivalents, and names the consequence in the degraded-ownership warning, which is the only signal an operator gets that cancellation on that host is back to killing the leader alone. Co-authored-by: multica-agent <[email protected]> * fix(agent): bound a finished probe's wait on pipes its descendants hold Review caught that runOwned could wait for its own cleanup and never return, and it reproduces: a descendant that inherited the output pipes holds them open after the leader exits, cmd.Wait blocks until the copy goroutines see EOF, and what would close those pipes is the release that runs after Wait. Cancellation breaks the cycle, which is why every test in the last round missed this — they all cancelled first, and they all set WaitDelay by hand. Normal exit has nothing to break it, and checkOpenclawVersion runs on the caller's context before runContext establishes any task timeout, so a version wrapper that exits cleanly while leaving a pipe-holding descendant would hang the task until a human cancelled it. runOwned now fills in a bounded WaitDelay when the caller left it zero — Go defines WaitDelay for precisely "a child process that exits but leaves its I/O pipes unclosed" — and signals the group before dropping ownership, so the descendant is reaped on Unix too, where releasing a process group is a no-op. A caller with its own bound keeps it; detectCLIVersion has set one by hand since MUL-3812 for this same shape. Adds the normal-exit regression on both platforms: the leader exits holding nothing, the descendant keeps the pipes, and outputOwned has to return in bounded time with the descendant gone. The Windows descendant spawner gains a spawn-and-exit mode and now hands its stdio to the descendant, which is what makes it hold the pipes at all. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]>
203 lines
6.3 KiB
Go
203 lines
6.3 KiB
Go
//go:build unix
|
|
|
|
package agent
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"log/slog"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strings"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
// TestRuntimeCommandsGetTheirOwnProcessGroup pins the default that GH #7522
|
|
// showed was missing. Before, each backend had to remember to ask for a process
|
|
// group and most did not, so a group-wide signal could not reach their CLI at
|
|
// all. The group now comes from the one place a runtime process is built, which
|
|
// means a backend cannot be launched without it.
|
|
func TestRuntimeCommandsGetTheirOwnProcessGroup(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
cmd := NewCommand("/bin/sh", nil).exec(context.Background(), "-c", "true")
|
|
if cmd.SysProcAttr == nil || !cmd.SysProcAttr.Setpgid {
|
|
t.Fatal("a runtime command must be built with Setpgid so cancellation can signal the whole tree")
|
|
}
|
|
}
|
|
|
|
// TestRuntimeCommandCancellationKillsDescendants is the behaviour the default
|
|
// buys, on a command with no backend-specific cancellation logic at all: the
|
|
// tool subprocesses an agent spawned must die with it.
|
|
//
|
|
// os/exec's own Cancel kills the leader alone, which is what left a cancelled
|
|
// agent's descendants running. The fake here spawns a grandchild that outlives
|
|
// its parent, so killing the leader is not enough to pass.
|
|
func TestRuntimeCommandCancellationKillsDescendants(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tempDir := t.TempDir()
|
|
pidFile := filepath.Join(tempDir, "pids")
|
|
fakePath := filepath.Join(tempDir, "runtime")
|
|
writeTestExecutable(t, fakePath, []byte("#!/bin/sh\n"+
|
|
`( sleep 300 ) </dev/null >/dev/null 2>&1 &
|
|
printf '%s %s\n' "$$" "$!" > "$1"
|
|
sleep 300
|
|
`))
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
cmd := NewCommand(fakePath, nil).exec(ctx, pidFile)
|
|
if err := startOwnedProcessTree(cmd, slog.Default()); err != nil {
|
|
t.Fatalf("start: %v", err)
|
|
}
|
|
defer releaseProcessGroup(cmd)
|
|
|
|
pids := waitForPids(t, pidFile)
|
|
cancel()
|
|
|
|
done := make(chan struct{})
|
|
go func() { _ = cmd.Wait(); close(done) }()
|
|
select {
|
|
case <-done:
|
|
case <-time.After(10 * time.Second):
|
|
t.Fatal("the cancelled process was never reaped")
|
|
}
|
|
for _, pid := range pids {
|
|
waitProcessGone(t, pid)
|
|
}
|
|
}
|
|
|
|
// TestProbeOutputCancellationKillsDescendants is the probe half of GH #7522.
|
|
//
|
|
// outputOwned exists because cmd.Output() calls Start itself and so can never
|
|
// reach the ownership boundary. detectCLIVersion's own comment describes the
|
|
// shape this reproduces: a CLI shim that leaves a grandchild behind. Cancelling
|
|
// the probe has to reap both, not just the shim.
|
|
func TestProbeOutputCancellationKillsDescendants(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tempDir := t.TempDir()
|
|
pidFile := filepath.Join(tempDir, "pids")
|
|
fakePath := filepath.Join(tempDir, "probe")
|
|
writeTestExecutable(t, fakePath, []byte("#!/bin/sh\n"+
|
|
`( sleep 300 ) </dev/null >/dev/null 2>&1 &
|
|
printf '%s %s\n' "$$" "$!" > "$1"
|
|
sleep 300
|
|
`))
|
|
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
defer cancel()
|
|
|
|
cmd := NewCommand(fakePath, nil).exec(ctx, pidFile)
|
|
cmd.WaitDelay = 5 * time.Second
|
|
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
_, err := outputOwned(cmd, slog.Default())
|
|
done <- err
|
|
}()
|
|
|
|
pids := waitForPids(t, pidFile)
|
|
cancel()
|
|
|
|
select {
|
|
case <-done:
|
|
case <-time.After(20 * time.Second):
|
|
t.Fatal("outputOwned never returned after the probe was cancelled")
|
|
}
|
|
for _, pid := range pids {
|
|
waitProcessGone(t, pid)
|
|
}
|
|
}
|
|
|
|
// TestOutputOwnedMatchesStdlibContract keeps the owned probe helpers drop-in:
|
|
// callers were using cmd.Output() and cmd.CombinedOutput() and must keep
|
|
// getting the same bytes back.
|
|
func TestOutputOwnedMatchesStdlibContract(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tempDir := t.TempDir()
|
|
fakePath := filepath.Join(tempDir, "noisy")
|
|
writeTestExecutable(t, fakePath, []byte("#!/bin/sh\n"+
|
|
`printf 'to-stdout\n'
|
|
printf 'to-stderr\n' >&2
|
|
exit 3
|
|
`))
|
|
|
|
out, err := outputOwned(NewCommand(fakePath, nil).exec(context.Background()), slog.Default())
|
|
if string(out) != "to-stdout\n" {
|
|
t.Errorf("stdout = %q, want %q", out, "to-stdout\n")
|
|
}
|
|
var exitErr *exec.ExitError
|
|
if !errors.As(err, &exitErr) {
|
|
t.Fatalf("err = %v, want an *exec.ExitError", err)
|
|
}
|
|
if exitErr.ExitCode() != 3 {
|
|
t.Errorf("exit code = %d, want 3", exitErr.ExitCode())
|
|
}
|
|
if !strings.Contains(string(exitErr.Stderr), "to-stderr") {
|
|
t.Errorf("ExitError.Stderr = %q, want it to carry the probe's stderr", exitErr.Stderr)
|
|
}
|
|
|
|
combined, err := combinedOutputOwned(NewCommand(fakePath, nil).exec(context.Background()), slog.Default())
|
|
if err == nil {
|
|
t.Error("combinedOutputOwned should report the non-zero exit")
|
|
}
|
|
for _, want := range []string{"to-stdout", "to-stderr"} {
|
|
if !strings.Contains(string(combined), want) {
|
|
t.Errorf("combined output %q missing %q", combined, want)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestProbeReturnsWhenLeaderExitsHoldingPipes is the normal-exit counterpart to
|
|
// the cancellation tests: nobody cancels anything here.
|
|
//
|
|
// The fake exits straight away but leaves a descendant holding the stdout and
|
|
// stderr write ends. cmd.Wait() cannot return until those close, and on Windows
|
|
// what closes them is releaseProcessGroup — which runs after Wait. A probe that
|
|
// waits for its own cleanup never returns, and checkOpenclawVersion runs on the
|
|
// caller's context before any task timeout exists, so the task would hang until
|
|
// someone cancelled it by hand.
|
|
func TestProbeReturnsWhenLeaderExitsHoldingPipes(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
tempDir := t.TempDir()
|
|
pidFile := filepath.Join(tempDir, "pids")
|
|
fakePath := filepath.Join(tempDir, "wrapper")
|
|
// The descendant inherits stdout/stderr on purpose — that is what holds the
|
|
// pipe open after the leader is gone.
|
|
writeTestExecutable(t, fakePath, []byte("#!/bin/sh\n"+
|
|
`( sleep 300 ) &
|
|
printf '%s %s\n' "$$" "$!" > "$1"
|
|
printf 'v1.2.3\n'
|
|
exit 0
|
|
`))
|
|
|
|
done := make(chan []byte, 1)
|
|
go func() {
|
|
out, _ := outputOwned(NewCommand(fakePath, nil).exec(context.Background(), pidFile), slog.Default())
|
|
done <- out
|
|
}()
|
|
|
|
pids := waitForPids(t, pidFile)
|
|
|
|
select {
|
|
case out := <-done:
|
|
if string(out) != "v1.2.3\n" {
|
|
t.Errorf("stdout = %q, want the version line", out)
|
|
}
|
|
case <-time.After(15 * time.Second):
|
|
t.Fatal("outputOwned never returned: the leader exited, but Wait is still waiting for a pipe " +
|
|
"that only releaseProcessGroup can close, and releaseProcessGroup runs after Wait")
|
|
}
|
|
|
|
// Nothing the probe spawned may outlive the answer.
|
|
for _, pid := range pids {
|
|
waitProcessGone(t, pid)
|
|
}
|
|
}
|