Files
multica/server/pkg
Multica Eveandmultica-agent 9310690e0a MUL-7933: fix(wakeups): drain inputs and due checks in one pass (#9202)
* fix(wakeups): drain pending inputs and due checks in one dispatch pass

A dispatch pass visited a single batch of 100 rules ordered by updated_at.
With ~1,160 ready rules in production, a status-change hint waited about
a dozen 30s passes behind condition checks with nothing new, so
--until-status handoffs fired minutes late (MUL-7933, #9139).

Each pass now takes batches from two lanes until both drain or a limit
is hit: rules with pending inputs (at most 50 per batch, not filtered by
enabled) and due schedules/deadlines/condition checks (the rest of each
batch). Rules are visited at most once per pass; the pass stops after 15s
or 1,500 rules so the sequential scheduler keeps time for other jobs.
Global passes log lane counts, duration, stop reason and the oldest
pending input age (warn when cut short or older than 2 minutes).

Co-authored-by: multica-agent <[email protected]>

* fix(wakeups): keep due checks moving behind slow inputs; report held inputs apart

Review follow-up on #9202.

- A pass ran a whole batch's inputs before its due rules and checked only
  the overall 15s budget, so a few inputs near the 2s dispatch cap could
  starve schedules, deadlines and condition checks. Dispatches now
  alternate between the lanes, and while due work remains the input lane
  stops at half the pass budget; either lane uses the rest once the other
  drains.
- The pass log counted every unprocessed receipt as backlog, but dispatch
  keeps inputs pending on purpose while a run holds them (reserved by a
  claim, the rule's own run claimed, or waiting to join a queued run). Those
  are now logged as held_inputs, and only the oldest input the scheduler
  could act on (oldest_ready_input_age_s) can raise the 2-minute warning.

Co-authored-by: multica-agent <[email protected]>

* fix(wakeups): count only inputs a run can still take as held

Review follow-up on #9202. The pass log's held count hid real backlog in
two cases, so a stalled wakeup could sit for minutes without a warning:

- An input whose task_id pointed at a run that ended (cancelled/failed)
  before starting was counted as held, but takenReceipts releases it on
  the next dispatch. Reserved now means the run has neither started nor
  ended, as takenReceipts decides; an input of a run that started is
  consumed on the next dispatch and also counts as backlog.
- A platform rule's input was held by any queued run on the issue. It is
  now held only by a queued run of the issue's current target agent, run
  as the person the platform rule's run would run as, reusing
  resolveWakeTarget, childDoneRunAs and hasWaitingRun as dispatchSystem
  does.

PendingWakeupInputs is replaced by ListPendingWakeupInputGroups (per-rule
groups, oldest first, bounded) plus pendingInputs in Go.

Co-authored-by: multica-agent <[email protected]>

* fix(wakeups): keep tick tests and passes stable under load

CI failed the tick tests with "timeout: context deadline exceeded": under
load the 100ms scan-order touch after each dispatch timed out, and the
pass reported that as an error. The touch only affects scan order (the
rule's outcome is already committed and an unfinished rule stays a
candidate), so a failed touch is now logged at debug, not returned.

The tests also depended on wall-clock speed. The drain tests now run with
a generous budget, since they pin which rules a pass reaches; the slow
input test checks dispatch order (at most two slow inputs before the last
due check) instead of what fits in 1.5s. Without the input budget it
still fails, with all ten inputs ahead of the last due check.

Co-authored-by: multica-agent <[email protected]>

* fix(wakeups): scan every pending input group; warn when the scan is incomplete

Review follow-up on #9202. The pending scan read only the oldest 1000
groups, so a ready input behind a page of held groups left
oldest_ready_input_age_s empty, and a scan that hit its 1s timeout was
dropped without a trace; either way a stalled wakeup did not warn.

- ListPendingWakeupInputGroups is keyset-paged on (oldest, wakeup_id,
  reserved); pendingInputs reads pages until a short one.
- The scan gets 3s. On an error or timeout the pass logs
  pending_scan=incomplete with the error, at warn level, together with
  the partial counts; a finished scan logs pending_scan=complete.
- A pass with dispatch failures also logs at warn level.
- Platform-rule target checks are cached per issue.

Co-authored-by: multica-agent <[email protected]>

---------

Co-authored-by: multica-agent <[email protected]>
2026-10-13 14:15:10 +08:00
..