mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-09 01:46:46 +08:00
* fix(agent): own the runtime process tree on every backend Cancelling a task on Windows left the agent running. The daemon marked the task cancelled, revoked its mat_ token, and killed what it thought was the CLI — but on Windows the direct child is usually a cmd.exe shim and the real CLI is already a grandchild, so it survived. One reporter's orphan kept working for 40 minutes, read the daemon owner's profile PAT after its own token started returning 401, and posted comments as the member (GH #7522). Whole-tree ownership already existed on both platforms. It was opt-in, and opt-in rotted: 8 of the 27 places this package starts a process asked for a Unix process group, and 3 of the 23 backends took Windows Job Object ownership. Adding claude to that list would have left 19 backends with the same hole, so the fix is structural instead. - newRuntimeCmd, at the single point where a runtime process is built, now puts every command in its own process group and replaces os/exec's leader-only Cancel with a group-wide kill. A backend that wants a graceful shutdown still assigns its own cmd.Cancel and wins. This also covers the --version and model-discovery probes, which cmd.Output() runs without ever reaching a start helper. - startOwnedProcessTree is now the only way this package starts a process, so Windows Job Object ownership is not something a backend can forget. Each start is paired with releaseProcessGroup after the reap, which drops the job handle and kills whatever outlived it. - Two structural tests enforce both halves, the way TestOnlyLaunchGoSpawnsRuntimeProcesses already enforces the argv boundary: a new backend calling cmd.Start fails, and one that takes ownership without releasing it fails. The Unix half is #5918 for the 20 backends the original fix did not reach: their CLI was never in its own group at all, so no group-wide signal could have reached it. Co-authored-by: multica-agent <[email protected]> * fix(agent): own the process tree for synchronous probes too Review found the ownership boundary had a hole the PR claimed to cover. os/exec's Run, Output and CombinedOutput call Start themselves, so a probe written with cmd.Output() never reaches startOwnedProcessTree and owns nothing on Windows — where the direct child of a `--version` probe is typically the npm shim and the real CLI is already a grandchild. That is the same escape #7522 was reported for, in 15 call sites nobody was looking at; detectCLIVersion's own comment already describes a broken CLI leaving grandchildren that hold the stdout pipe. - runOwned / outputOwned / combinedOutputOwned give a synchronous probe the same start, reap and release a task launch gets. They keep the stdlib contract: stdout returned, stderr attached to the ExitError, combined output through a single writer so os/exec gives it one pipe. The stderr sample is bounded, as os/exec bounds its own. - All 15 probe sites route through them, and the structural guard now rejects Run/Output/CombinedOutput alongside Start. The release guard did not hold its invariant either: it only asked whether a file contained any releaseProcessGroup, so a second unreleased start in a file that already had a pair would pass — and models.go is exactly that shape. It now works per function: one owned start each, paired with at least one release. Equal counts would be the wrong rule, since dsh's Execute legitimately releases on two exit paths. Both guards are now pure functions over an AST, with tests that feed them synthetic sources — including the second-unreleased-start case that the file-level version missed — so the guards fail closed without needing a real regression committed to find out. Adds the Windows probe regression (descendant must be in the job and die with a cancelled probe) plus unix equivalents, and names the consequence in the degraded-ownership warning, which is the only signal an operator gets that cancellation on that host is back to killing the leader alone. Co-authored-by: multica-agent <[email protected]> * fix(agent): bound a finished probe's wait on pipes its descendants hold Review caught that runOwned could wait for its own cleanup and never return, and it reproduces: a descendant that inherited the output pipes holds them open after the leader exits, cmd.Wait blocks until the copy goroutines see EOF, and what would close those pipes is the release that runs after Wait. Cancellation breaks the cycle, which is why every test in the last round missed this — they all cancelled first, and they all set WaitDelay by hand. Normal exit has nothing to break it, and checkOpenclawVersion runs on the caller's context before runContext establishes any task timeout, so a version wrapper that exits cleanly while leaving a pipe-holding descendant would hang the task until a human cancelled it. runOwned now fills in a bounded WaitDelay when the caller left it zero — Go defines WaitDelay for precisely "a child process that exits but leaves its I/O pipes unclosed" — and signals the group before dropping ownership, so the descendant is reaped on Unix too, where releasing a process group is a no-op. A caller with its own bound keeps it; detectCLIVersion has set one by hand since MUL-3812 for this same shape. Adds the normal-exit regression on both platforms: the leader exits holding nothing, the descendant keeps the pipes, and outputOwned has to return in bounded time with the descendant gone. The Windows descendant spawner gains a spawn-and-exit mode and now hands its stdio to the descendant, which is what makes it hold the pipes at all. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]>
450 lines
14 KiB
Go
450 lines
14 KiB
Go
package agent
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"io"
|
|
"os/exec"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"time"
|
|
)
|
|
|
|
// qoderBlockedArgs are flags hardcoded by the daemon that must not be
|
|
// overridden by user-configured custom_args. Qoder enters ACP mode via the
|
|
// global `--acp` flag; letting users strip or duplicate it would break the
|
|
// daemon↔Qoder transport contract. `--yolo` is daemon-owned so headless ACP
|
|
// always runs in bypass-permissions mode (see Qoder ACP docs). The legacy `acp`
|
|
// subcommand style is blocked so custom_args cannot switch the binary mode.
|
|
var qoderBlockedArgs = map[string]blockedArgMode{
|
|
"--acp": blockedStandalone,
|
|
"acp": blockedStandalone,
|
|
"--yolo": blockedStandalone,
|
|
}
|
|
|
|
// qoderBackend implements Backend for both Qoder CLI binaries by spawning
|
|
// `<binary> --yolo --acp` and communicating via the ACP (Agent Communication
|
|
// Protocol) JSON-RPC 2.0 transport over stdin/stdout.
|
|
//
|
|
// Qoder CLI uses global flags (`--yolo`, `--acp`), not an `acp` subcommand. We
|
|
// reuse hermesClient like Hermes/Kimi/Kiro and mirror their streaming gate so
|
|
// history replay flushed during session/setup does not corrupt the streamed
|
|
// output or leave the UI stuck on a stale assistant chunk.
|
|
type qoderBackend struct {
|
|
cfg Config
|
|
defaultExecutable string
|
|
}
|
|
|
|
func qoderDefaultBinary(providerType string) string {
|
|
if providerType == "qoderclicn" {
|
|
return "qoderclicn"
|
|
}
|
|
return "qodercli"
|
|
}
|
|
|
|
var qoderReaderDrainGrace = 2 * time.Second
|
|
|
|
type qoderMessageStream struct {
|
|
ch chan Message
|
|
mu sync.Mutex
|
|
closed bool
|
|
}
|
|
|
|
func newQoderMessageStream(size int) *qoderMessageStream {
|
|
return &qoderMessageStream{ch: make(chan Message, size)}
|
|
}
|
|
|
|
func (s *qoderMessageStream) send(msg Message) {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
if s.closed {
|
|
return
|
|
}
|
|
trySend(s.ch, msg)
|
|
}
|
|
|
|
func (s *qoderMessageStream) close() {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
if s.closed {
|
|
return
|
|
}
|
|
s.closed = true
|
|
close(s.ch)
|
|
}
|
|
|
|
func (b *qoderBackend) Execute(ctx context.Context, prompt string, opts ExecOptions) (*Session, error) {
|
|
execPath := b.cfg.ExecutablePath
|
|
if execPath == "" {
|
|
execPath = b.defaultExecutable
|
|
}
|
|
if _, err := exec.LookPath(execPath); err != nil {
|
|
return nil, fmt.Errorf("%s executable not found at %q: %w", b.defaultExecutable, execPath, err)
|
|
}
|
|
|
|
// Translate the agent's mcp_config (Claude-style object of objects) into
|
|
// the array shape ACP session/new and session/resume expect. Reuse the
|
|
// shared converter so remote MCP `headers` (e.g. Authorization) survive as
|
|
// [{name, value}] and output is deterministic. Fail closed on malformed
|
|
// JSON so the launch surfaces the real error instead of silently dropping
|
|
// every MCP server.
|
|
mcpServers, err := buildACPMcpServers(opts.McpConfig, b.cfg.Logger)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("qoder: invalid mcp_config: %w", err)
|
|
}
|
|
|
|
timeout := opts.Timeout
|
|
runCtx, cancel := runContext(ctx, timeout)
|
|
|
|
qoderArgs := append(
|
|
[]string{"--yolo", "--acp"},
|
|
filterCustomArgs(opts.CustomArgs, qoderBlockedArgs, b.cfg.Logger)...,
|
|
)
|
|
cmd := b.cfg.commandAt(execPath).exec(runCtx, qoderArgs...)
|
|
hideAgentWindow(cmd)
|
|
b.cfg.logAgentCommand(cmd, newAgentCommandLogArgs(qoderArgs))
|
|
if opts.Cwd != "" {
|
|
cmd.Dir = opts.Cwd
|
|
}
|
|
cmd.Env = buildEnv(b.cfg.Env)
|
|
|
|
stdout, err := cmd.StdoutPipe()
|
|
if err != nil {
|
|
cancel()
|
|
return nil, fmt.Errorf("qoder stdout pipe: %w", err)
|
|
}
|
|
stdin, err := cmd.StdinPipe()
|
|
if err != nil {
|
|
cancel()
|
|
return nil, fmt.Errorf("qoder stdin pipe: %w", err)
|
|
}
|
|
// StderrPipe + an explicit copier give us a join point (`stderrDone`)
|
|
// that fires before the failure-promotion decision; see the matching
|
|
// comment in hermes.go for why the io.MultiWriter form races with
|
|
// stopReason=end_turn under load (a terminal provider error can land
|
|
// after we've already read finalOutput and reported "completed").
|
|
providerErr := newACPProviderErrorSniffer("qoder")
|
|
stderr, err := cmd.StderrPipe()
|
|
if err != nil {
|
|
cancel()
|
|
return nil, fmt.Errorf("qoder stderr pipe: %w", err)
|
|
}
|
|
|
|
if err := startOwnedProcessTree(cmd, b.cfg.Logger); err != nil {
|
|
cancel()
|
|
return nil, fmt.Errorf("start qoder: %w", err)
|
|
}
|
|
|
|
stderrSink := io.MultiWriter(newLogWriter(b.cfg.Logger, "[qoder:stderr] "), providerErr)
|
|
stderrDone := make(chan struct{})
|
|
go func() {
|
|
defer close(stderrDone)
|
|
_, _ = io.Copy(stderrSink, stderr)
|
|
}()
|
|
|
|
b.cfg.Logger.Info("qoder acp started", "pid", cmd.Process.Pid, "cwd", opts.Cwd)
|
|
|
|
msgStream := newQoderMessageStream(256)
|
|
resCh := make(chan Result, 1)
|
|
|
|
// Qoder emits interim narration and the final answer as the same ACP
|
|
// AgentMessageChunk type; the tracker keeps only the post-tool-call block
|
|
// for Result.Output while retaining the full text for error detection.
|
|
var deliverable acpDeliverableTracker
|
|
var streamingCurrentTurn atomic.Bool
|
|
|
|
promptDone := make(chan hermesPromptResult, 1)
|
|
activity := make(chan struct{}, 1)
|
|
|
|
c := &hermesClient{
|
|
cfg: b.cfg,
|
|
stdin: stdin,
|
|
pending: make(map[int]*pendingRPC),
|
|
pendingTools: make(map[string]*pendingToolCall),
|
|
acceptNotification: func(string) bool {
|
|
return streamingCurrentTurn.Load()
|
|
},
|
|
onActivity: func() {
|
|
select {
|
|
case activity <- struct{}{}:
|
|
default:
|
|
}
|
|
},
|
|
onMessage: func(msg Message) {
|
|
if !streamingCurrentTurn.Load() {
|
|
return
|
|
}
|
|
if msg.Type == MessageToolUse {
|
|
msg.Tool = kimiToolNameFromTitle(msg.Tool)
|
|
}
|
|
deliverable.observe(msg)
|
|
msgStream.send(msg)
|
|
},
|
|
onPromptDone: func(result hermesPromptResult) {
|
|
if !streamingCurrentTurn.Load() {
|
|
return
|
|
}
|
|
select {
|
|
case promptDone <- result:
|
|
default:
|
|
}
|
|
},
|
|
}
|
|
|
|
readerDone := make(chan struct{})
|
|
go func() {
|
|
defer close(readerDone)
|
|
scanner := newAgentStreamScanner(stdout)
|
|
for scanner.Scan() {
|
|
line := strings.TrimSpace(scanner.Text())
|
|
if line == "" {
|
|
continue
|
|
}
|
|
c.handleLine(line)
|
|
}
|
|
c.closeAllPending(fmt.Errorf("qoder process exited"))
|
|
}()
|
|
|
|
go func() {
|
|
defer cancel()
|
|
defer msgStream.close()
|
|
defer close(resCh)
|
|
defer func() {
|
|
stdin.Close()
|
|
_ = cmd.Wait()
|
|
releaseProcessGroup(cmd)
|
|
}()
|
|
|
|
startTime := time.Now()
|
|
finalStatus := "completed"
|
|
var finalError string
|
|
var sessionID string
|
|
// Set when the ACP runtime refuses the session we asked to
|
|
// resume. Only that is curable by starting a fresh session, so
|
|
// handshake/network failures below must leave it false.
|
|
var resumeRejected bool
|
|
effectiveModel := strings.TrimSpace(opts.Model)
|
|
|
|
initResult, err := c.request(runCtx, "initialize", map[string]any{
|
|
"protocolVersion": 1,
|
|
"clientInfo": map[string]any{
|
|
"name": "multica-agent-sdk",
|
|
"version": "0.2.0",
|
|
},
|
|
"clientCapabilities": map[string]any{},
|
|
})
|
|
if err != nil {
|
|
finalStatus = "failed"
|
|
finalError = fmt.Sprintf("qoder initialize failed: %v", err)
|
|
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
|
|
return
|
|
}
|
|
|
|
// Drop MCP entries whose remote transport the runtime didn't
|
|
// advertise. See the matching comment in hermes.go for why
|
|
// unconditionally sending http/sse to a stdio-only ACP runtime
|
|
// tanks the whole session/new.
|
|
mcpServers = filterACPMcpServersByCapability(mcpServers, extractACPMcpCapabilities(initResult), "qoder", b.cfg)
|
|
|
|
cwd := opts.Cwd
|
|
if cwd == "" {
|
|
cwd = "."
|
|
}
|
|
|
|
if opts.ResumeSessionID != "" {
|
|
result, err := c.request(runCtx, "session/resume", map[string]any{
|
|
"cwd": cwd,
|
|
"sessionId": opts.ResumeSessionID,
|
|
"mcpServers": mcpServers,
|
|
})
|
|
if err != nil {
|
|
finalStatus = "failed"
|
|
finalError = fmt.Sprintf("qoder session/resume failed: %v", err)
|
|
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
|
|
return
|
|
}
|
|
var changed bool
|
|
sessionID, changed = resolveResumedSessionID(opts.ResumeSessionID, result)
|
|
if changed {
|
|
b.cfg.Logger.Warn("agent returned a different session id on resume — original was likely lost; continuing with the new id",
|
|
"backend", "qoder",
|
|
"requested", opts.ResumeSessionID,
|
|
"actual", sessionID,
|
|
)
|
|
}
|
|
if effectiveModel == "" {
|
|
effectiveModel = extractACPCurrentModelID(result)
|
|
}
|
|
} else {
|
|
result, err := c.request(runCtx, "session/new", map[string]any{
|
|
"cwd": cwd,
|
|
"mcpServers": mcpServers,
|
|
})
|
|
if err != nil {
|
|
finalStatus = "failed"
|
|
finalError = fmt.Sprintf("qoder session/new failed: %v", err)
|
|
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
|
|
return
|
|
}
|
|
sessionID = extractACPSessionID(result)
|
|
if sessionID == "" {
|
|
finalStatus = "failed"
|
|
finalError = "qoder session/new returned no session ID"
|
|
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
|
|
return
|
|
}
|
|
if effectiveModel == "" {
|
|
effectiveModel = extractACPCurrentModelID(result)
|
|
}
|
|
}
|
|
|
|
c.sessionID = sessionID
|
|
b.cfg.Logger.Info("qoder session created", "session_id", sessionID)
|
|
|
|
if opts.Model != "" {
|
|
if _, err := c.request(runCtx, "session/set_model", map[string]any{
|
|
"sessionId": sessionID,
|
|
"modelId": opts.Model,
|
|
}); err != nil {
|
|
b.cfg.Logger.Warn("qoder set_session_model failed", "error", err, "requested_model", opts.Model)
|
|
finalStatus = "failed"
|
|
finalError = fmt.Sprintf("qoder could not switch to model %q: %v", opts.Model, err)
|
|
if opts.ResumeSessionID != "" && isACPSessionNotFound(err) {
|
|
// On a resumed session with a model override, the dead
|
|
// session surfaces here instead of at session/prompt.
|
|
// Clear the id so the daemon's resume-failure fallback
|
|
// retries fresh and stores the replacement session.
|
|
b.cfg.Logger.Warn("resumed session not found at set_model time; clearing session id so the daemon retries fresh",
|
|
"backend", "qoder",
|
|
"session_id", sessionID,
|
|
)
|
|
sessionID = ""
|
|
resumeRejected = true
|
|
}
|
|
resCh <- Result{
|
|
Status: finalStatus,
|
|
Error: finalError,
|
|
DurationMs: time.Since(startTime).Milliseconds(),
|
|
SessionID: sessionID,
|
|
ResumeRejected: resumeRejected,
|
|
}
|
|
return
|
|
}
|
|
b.cfg.Logger.Info("qoder session model set", "model", opts.Model)
|
|
}
|
|
|
|
userText := prompt
|
|
if opts.SystemPrompt != "" {
|
|
userText = opts.SystemPrompt + "\n\n---\n\n" + prompt
|
|
}
|
|
|
|
// Flip just before session/prompt so history replay flushed during setup
|
|
// is dropped; every notification for this turn is processed afterward.
|
|
streamingCurrentTurn.Store(true)
|
|
_, err = c.request(runCtx, "session/prompt", map[string]any{
|
|
"sessionId": sessionID,
|
|
"prompt": []map[string]any{
|
|
{"type": "text", "text": userText},
|
|
},
|
|
})
|
|
if err != nil {
|
|
if runCtx.Err() == context.DeadlineExceeded {
|
|
finalStatus = "timeout"
|
|
finalError = fmt.Sprintf("qoder timed out after %s", timeout)
|
|
} else if runCtx.Err() == context.Canceled {
|
|
finalStatus = "aborted"
|
|
finalError = "execution cancelled"
|
|
} else {
|
|
finalStatus = "failed"
|
|
finalError = fmt.Sprintf("qoder session/prompt failed: %v", err)
|
|
if opts.ResumeSessionID != "" && isACPSessionNotFound(err) {
|
|
// The runtime may echo the requested id from
|
|
// session/resume and only reject it at prompt time.
|
|
// Empty SessionID lets the daemon retry with a fresh
|
|
// session instead of pinning future runs to the stale id.
|
|
b.cfg.Logger.Warn("resumed session not found at prompt time; clearing session id so the daemon retries fresh",
|
|
"backend", "qoder",
|
|
"session_id", sessionID,
|
|
)
|
|
sessionID = ""
|
|
resumeRejected = true
|
|
}
|
|
}
|
|
} else {
|
|
select {
|
|
case pr := <-promptDone:
|
|
if pr.stopReason == "cancelled" {
|
|
finalStatus = "aborted"
|
|
finalError = "qoder cancelled the prompt"
|
|
}
|
|
c.mergeUsage(pr.usage)
|
|
default:
|
|
}
|
|
waitForACPNotificationQuiescence(runCtx, activity, readerDone, acpNotificationQuietTime, qoderReaderDrainGrace)
|
|
}
|
|
|
|
duration := time.Since(startTime)
|
|
b.cfg.Logger.Info("qoder finished", "pid", cmd.Process.Pid, "status", finalStatus, "duration", duration.Round(time.Millisecond).String())
|
|
|
|
stdin.Close()
|
|
cancel()
|
|
|
|
// Qoder ACP may keep the process — and the stdout/stderr pipes — open
|
|
// after session/prompt returns (it can leave a child holding the
|
|
// inherited fds). The prompt response is already terminal, so bound the
|
|
// drain: wait for both the stdout reader and the stderr copier, but no
|
|
// longer than the grace window. CommandContext cancellation above is
|
|
// what actually tears the process down. Draining stderr here is what
|
|
// makes the provider-error promotion below see a terminal marker before
|
|
// we decide the final status.
|
|
drainCtx, drainCancel := context.WithTimeout(context.Background(), qoderReaderDrainGrace)
|
|
select {
|
|
case <-readerDone:
|
|
case <-drainCtx.Done():
|
|
}
|
|
select {
|
|
case <-stderrDone:
|
|
case <-drainCtx.Done():
|
|
}
|
|
drainCancel()
|
|
// The stdout reader may still run after the grace window. Flip the
|
|
// stream gate before this goroutine's defer closes msgStream; if a
|
|
// late reader already passed the gate, qoderMessageStream serializes
|
|
// send and close so the late send is dropped instead of panicking.
|
|
streamingCurrentTurn.Store(false)
|
|
|
|
finalOutput, providerErrorOutput := deliverable.result()
|
|
|
|
// Promote completed→failed when stderr or the agent text stream show a
|
|
// terminal upstream-LLM failure (HTTP 4xx / rate-limit / expired token).
|
|
// Mirrors hermes/kimi/kiro; without it a run that exhausts retries still
|
|
// reports "completed" because session/prompt ends with stopReason=end_turn
|
|
// even though qodercli wrote a terminal error to stderr.
|
|
finalStatus, finalError = promoteACPResultOnProviderError(finalStatus, finalError, providerErrorOutput, providerErr)
|
|
|
|
u := c.accumulatedUsage()
|
|
|
|
var usageMap map[string]TokenUsage
|
|
if acpUsagePresent(u) {
|
|
model := effectiveModel
|
|
if model == "" {
|
|
model = "unknown"
|
|
}
|
|
usageMap = map[string]TokenUsage{model: u}
|
|
}
|
|
|
|
resCh <- Result{
|
|
Status: finalStatus,
|
|
Output: finalOutput,
|
|
Error: finalError,
|
|
DurationMs: duration.Milliseconds(),
|
|
SessionID: sessionID,
|
|
ResumeRejected: resumeRejected,
|
|
Usage: usageMap,
|
|
}
|
|
}()
|
|
|
|
return &Session{Messages: msgStream.ch, Result: resCh}, nil
|
|
}
|