mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-11 10:56:45 +08:00
* feat(wecom): answer in the bubble the question opened
WeCom's smart-bot protocol has no typing indicator, no reaction and no
read receipt. The one affordance it has is the streaming message, so the
reply is opened early and filled in later: an aibot_respond_msg frame
with finish=false paints a bubble the client renders as "working", and a
later frame carrying the same stream id replaces that bubble's body in
place.
Two frames per round, and no more. A message that is ingested opens a
stream and writes a think tag into it, which renders as the client's own
loading dots; the answer, the failure notice or the cancellation writes
finish=true onto the same stream and the spinner becomes the words. The
store between them (stream_store.go) is a cache keyed by chat session:
it carries the callback's req_id, which only the WebSocket read loop
ever sees and which every frame of the stream has to echo, across to an
event-bus subscriber holding nothing but a session id minutes later.
Every way a turn can end closes the bubble, because a stream that never
finishes is a spinner in the user's chat for good: the answer from the
chat-done subscriber, task:failed and task:cancelled from the typing
indicator's own subscriptions, and the settled flush for a round that
never became a run. A round with no bubble left — the process restarted
mid-run, the opening frame was refused, the stream ran past the
protocol's ten-minute window — says the same words as an ordinary
message addressed by the task's own delivery row, so the degradation is
one whole message rather than a silence.
The window is measured, not guessed: the live tenant took a frame at
600.0s and refused the one at 630.0s with 846608. It belongs to the
stream and nothing written into the stream extends it, so a run longer
than that gives its handle up before writing rather than spend a refusal
the reader cannot unsee.
Two things this fixes on the way:
- One frame in flight per req_id (ws_sender.go, awaitAck). Two frames
of one req_id are answered in whatever order the server likes, and a
verdict handed to the wrong frame sends an answer down the fallback
path that did not need it.
- A closing frame whose ack never came is written again, up to three
more times. A lost ack says nothing about whether the frame landed,
and re-sending the identical frame is accepted either way; the side
chosen is a possible duplicate over an answer that was never sent.
The bubble's copy is selected per reader (strings.go, language.go): a
1:1 reads that person's Multica profile language, a room reads the
deployment's, set from MULTICA_WECOM_DEFAULT_LOCALE.
Filling the bubble in while the run is still going — the tool calls and
reasoning rendered as a scrolling step list, with the redaction and the
audience gate that has to come with it, and the hand-over onto a fresh
stream that keeps a run longer than the window on screen — is a separate
layer on top of this one and follows separately.
* feat(wecom): count bubbles opened, so a stranded one is visible
stream_finished and stream_fell_back describe endings, and a bubble
nobody ever ends moves neither. On a multi-replica deployment the answer
is produced off-lease and arrives as its own message; a process
restarted mid-run loses the handle; a closing frame may never happen. In
all three the spinner is left for the server to end at the ten-minute
mark, and from the two ending counters alone that is indistinguishable
from an hour in which nobody asked anything.
stream_opened is recorded where the handle is KEPT, which is not where a
frame was accepted: an opening frame whose ack never came back keeps its
handle on purpose, because re-sending that stream id later creates the
message if the frame was lost. A stream the server refuses outright is
not counted — no bubble exists to owe an ending.
opened minus finished minus fell_back is the outstanding count. It does
not settle to zero at any instant, since bubbles in flight sit in the
difference; it is a backlog read over time, not a balance.
Three tests, two of them reverse-verified: a bubble nobody ends leaves
the difference at one, an answered one leaves it at zero, and a refused
opening frame counts nothing while its answer still reaches the user as
the plain message main sends today.
* feat(wecom): bind a bubble to its run off the bus, not off the engine
The streaming bubble needs two facts the engine used to carry for it: which
messages are one run, and which task answers them. Carrying them meant changes
in channel/engine and in every other adapter that implements TypingNotifier —
six files touched for one platform's feature. This reverts all of that. The
engine is back to upstream/main, byte for byte, and so are dingtalk, lark,
slack and telegram; OnIngested and OnSettled are main's signatures again.
The run announces itself instead. task:queued carries a task id and a chat
session id (service.taskEvent) and every enqueue path already publishes it:
FinalizeChatTaskEnqueue for the debounced flush, the deferred sweeper for a
turn that waited on its media. So the store issues its own round identity — a
per-session sequence number — and pairs the two sequences by position.
Three rules, and what each of them stops:
1. A message arriving while a round is still waiting for a run JOINS it. Two
messages in one debounce window are one run and one answer, and a second
bubble for them is a bubble nobody ever closes. A message arriving when
nothing is waiting opens a bubble immediately: a wait with nothing on
screen reads as a message that was lost.
2. task:queued binds the OLDEST round with no run, and a run that finds none
waits in a per-session queue for the next bubble — the Router detaches the
ingest goroutine and a session's first message enqueues inside dispatch, so
the event routinely beats the bubble. Two kinds of run are refused: one
carrying an issue id, which answers somewhere other than this chat and
would seal the asker's question with a stranger's answer, and one whose
ending has already been handled, which would bind a bubble with no closer
left. Everything in the handler is a map operation: events.Bus dispatches
synchronously on the publisher's goroutine, so a query here would be
charged to whoever enqueued the task.
3. An attempt being auto-retried releases its round instead of closing it
(retryUnbind). The clone is a new task row with a new id and a NULL issue
id — indistinguishable on the bus from a fresh turn — so without this it
would bind the next question's bubble and then seal it with the previous
question's answer. Releasing the round makes the clone take the round it is
replacing, in either publish order: FailTask broadcasts the clone's
task:queued BEFORE the parent's task:failed, while a backoff child is
queued minutes later by the deferred sweeper.
A released round is not a collecting one: a new message must not join it, and
OnSettled must not close it as "never started" when its replacement is already
queued. That is what roundEntry.everBound separates.
OnSettled carries no name any more — a flush that started no task has none to
give — so it closes the session's oldest round that never became a run, which
is the only round such a flush can be speaking for.
wecom.NewResolverSet goes back to main's four parameters: the engine's own
tests build a WeCom set without a typing notifier. Boot assigns set.Typing
instead, and hands the assembled set to RouterOptions.WecomResolverSetBuilt so
the boot-wiring guard can still read back what boot actually built.
* fix(wecom): the answer is sent once, and only what the gate can attribute
Four findings from review, all of them cases where a comment states the
right rule and the code a few lines away does not implement it.
THE ANSWER IS NOT SENT TWICE. The fallback treated any seal error as
proof the frame missed, so a lost ack put the same words out again as a
plain message — and cancelAck's own note makes a lost ack the cascade
rather than the exception, one per retry, in a chat with no unsend. It
now falls back only on proof: a stated refusal (846605 / 846608) or a
provably-unsent error. Written-but-unacked is recorded as unknown.
provablyNotSent did not know errStreamAckTimeout and answered "not
sent" for it by default, which is the step that let the duplicate out.
THE ANSWER IS NOT LOST. seal's retry no longer starts an attempt the
remaining deadline cannot finish — at the shipped constants three
retries need 26s against a 10s budget, so the last two were unreachable
and the first spent the budget the plain message needed. The fallback
takes its own deadline when what is left cannot cover one push.
A VERDICT IS NOT GUESSED. An ack carries only req_id, so position
identifies a frame only while the server owes nothing else on that
req_id — and acks come back grouped by outcome, not in write order
(STRATEGY §6.4). A caller giving up used to free the gate while the
server still owed, and the closing frame could then read an abandoned
opener's acceptance as its own. No frame goes out while a verdict is
owed; seal's retries are exempt because an identical frame is not a
second frame, measured 2026-09-03 (§6.5).
AN ENDING IS ATTRIBUTED, NOT ASSUMED. A round on the open list stopped
being proof of origin when binding moved to task:queued: a question
typed in Multica on the same session publishes an identical event and
can bind the room's bubble. The gate now runs for every ending, and
cancellation has one at all. Its three answers are told apart — not
ours releases the round, unknown withholds the notice and releases
nothing, because an unreachable database is not evidence about where a
run came from. A run left pending is bounded by the ingest goroutine's
own clock rather than the protocol's ten minutes, so it cannot pair
with a much later question's bubble.
AND THE NOTICE REACHES A ROOM FROM ANY REPLICA. main routed it as an
ordinary reply when the local replica held no socket; the bubble path
lost that, and the WS lease makes "no socket here" the normal case on
any multi-replica deployment. The indicator routes again.
* fix(wecom): keep the writer probe main uses working on a channel lock
main asserts that a trace line is recorded while the writer is held, and
asks with sync.Mutex.TryLock. This branch replaced that mutex with a
channel semaphore, because lockWriter has to be waitable on a context.
Neither side is wrong on its own and neither is red on its own; the
merge CI builds does not compile.
tryLockWriter answers the same question the same way, and the probe
asks it instead.
* fix(wecom): retire the bookkeeping of a turn that settled without sealing
pruneStreamsLocked retired an entry only if it was both sealed and past
streamMaxAge, so a turn whose closing frame never went out kept its
counters for the life of the connection. After the seal and fallback
rules those are not rare: the gate declines a frame while a verdict is
owed, the answer falls back, and nothing ever seals.
The condition that matters is settled, not sealed. With acked == sent
the server owes this req_id nothing, so no verdict can arrive later to
be read against counters that have been reset. An entry still owed one
stays however old it is.
Age alone is not enough, and the shorter argument is worth refuting
because it is tempting: the platform ends a STREAM at ten minutes, but a
new stream id on the same req_id is accepted afterwards — measured
2026-08-09, STRATEGY §6.1 — so age cannot establish that no frame will
follow. TestPruneKeepsLiveTurnsEvenOverTheCap is what caught that; the
first version of this change would have dropped a live turn.
* fix(wecom): read one rule at every site that shares it
Seven findings from the last round were five rules, each fixed at one of
the two or three places it applies. This closes the rest.
A frame the owed gate lets through must be able to read its own verdict:
the rewrite exemption removed the gate and left the waiter unaddressable,
so a stated refusal came back as a timeout and the answer, classified
unconfirmed, was never said.
A failure that may have reached the socket is never read as proof it did
not. writeStreamFrame kept giving a frame's place back on errWriteAttempted,
and the opening frame dropped its handle on it - the one site in the
package still treating it as a definite refusal.
Every closer now shares one reading of a seal's outcome and one
fallback-budget rule: classifySeal and fallbackBudget, used by the answer,
by the failure and cancellation notices, and by a delivery routed from
another replica.
The origin gate is asked before anything can return without it, and one
delivery-row read now has three answers rather than two - no row, another
platform's, ours - so a foreign failure still costs one lookup while the
row-less runs the gate exists for finally reach it.
A round released for a retry is no longer thrown into the pool the next
task:queued draws from. It records the attempt it waits for, and the
clone's ending names it through chat_input_task_id, which is the only
authoritative name either run has.
A relayed reply seals the round it belongs to instead of pushing a second
message under it.
Bookkeeping retires an entry once nothing can write for its req_id again,
which cancelAck's deliberate debt could never reach on settlement alone.
* test(wecom): the user-binding double answers by the id it is given
Production returns pgx.ErrNoRows for a sender nobody has bound, so a
double that hands back a binding for every id cannot express a room where
one speaker is bound and another is not.
* fix(wecom): a queued round's empty answer makes no claim about why
QueuedBehind records that another round was open when this one was
painted. It is not evidence that the reply ahead covered this message, so
the notice claimed a merge that never happened. Dropping it from the
switch removes the claim without adding state, and it puts the attachment
case back in charge when files are carried.
The field stays: what it records is true and cheap. What was wrong was
the sentence built on it.
RouterOptions.WecomResolverSetBuilt now says what it asserts - that the
set was built, not that the engine registered it. The accessor that would
assert the stronger thing is a shared-engine change.
* fix(wecom): every ending reaches the bubble, including the ones with no words
The relay could carry a reply that HAS words. Every other ending had no way
to reach the replica holding the bubble, so off-lease it left a spinner
claiming work was in progress for the rest of the protocol's window - and
nothing ends one: the sweep writes no frame, OnSettled needs an unbound
round, and the next question opens its own bubble.
Three of them, one rule:
a cancellation returned at !holding() or at !HasBubble and relayed nothing
a completion with nothing to say returned before the only relay site
an answer that is only files had its take gated on hasVisibleChar
relayKindSeal is what they route through. It is deliberately NOT a reply with
an empty body: a reply whose round is gone falls through to an ordinary push,
so one cancel-all click would put 这次处理已取消 into every chat in the
deployment. A seal frame with no round does nothing at all.
It carries the REASON, not the words. The words are the round's and the
round's locale was captured when its bubble was opened, so only the holder
can say the sentence in the language the asker reads. It also carries no
installation id: it is filtered by who holds the round, which is the same
question as who holds the socket, asked without the address lookup the
cancellation path deliberately refuses.
Outbound.relay widens to the noticeRouter interface the indicator already
uses, because a concrete type makes 'was this ending routed at all'
untestable - which is what these three gaps hid behind. That widening
silently removed the nil-receiver guard inside RelayOutbound.publish that
three call sites were standing on, so routeFrame now checks in one place.
* test(wecom): make the seal-frame case able to fail
It asserted that a seal frame with no round puts nothing in the room, and
passed with the seal branch turned into a reply that pushes - because the
frame carried no installation id, so deliverRelayed returned at the id parse
before any push could happen. Addressable now, so the assertion is standing
on the path it names.
---------
Co-authored-by: Seacen Zhao <[email protected]>
202 lines
7.8 KiB
Go
202 lines
7.8 KiB
Go
package wecom
|
|
|
|
// task_failed_test.go — what a failed run says in the chat, and who says it.
|
|
//
|
|
// #7952 established the words: the platform's own redacted reason, behind a
|
|
// warning mark, the way DingTalk and Lark deliver theirs. It sent them from
|
|
// Outbound, because at the time nothing else was listening.
|
|
//
|
|
// Here the typing indicator sends them instead, and that is not a preference:
|
|
// a failed run has a bubble waiting on it, and the indicator is the only
|
|
// party that can seal it. Sending from Outbound as well would put two messages
|
|
// in the chat for one failure — the bubble's ending and a plain message under
|
|
// it — which is what TestAFailedRunProducesExactlyOneMessage guards.
|
|
|
|
import (
|
|
"errors"
|
|
"testing"
|
|
|
|
"github.com/multica-ai/multica/server/internal/events"
|
|
"github.com/multica-ai/multica/server/pkg/protocol"
|
|
)
|
|
|
|
// failedWithReason publishes the task:failed FailTask broadcasts for a run
|
|
// whose failure carries the platform's redacted text.
|
|
func (r *bubbleRig) failedWithReason(t *testing.T, taskName, reason string) {
|
|
t.Helper()
|
|
id := taskUUID(t, taskName)
|
|
r.bus.Publish(events.Event{
|
|
Type: protocol.EventTaskFailed,
|
|
ChatSessionID: bubbleSession,
|
|
TaskID: id,
|
|
Payload: map[string]any{
|
|
"task_id": id,
|
|
"failure_reason": "provider_network",
|
|
"retry_pending": false,
|
|
"error": reason,
|
|
},
|
|
})
|
|
}
|
|
|
|
// The reason is the whole value of the notice: "上下文超出模型限制" tells a
|
|
// person what to do next where a generic line does not.
|
|
//
|
|
// REVERSE VERIFICATION: make failureText return copyFor(l).StreamFailed
|
|
// unconditionally and this fails with the generic line.
|
|
func TestAFailureCarriesThePlatformsOwnReason(t *testing.T) {
|
|
t.Parallel()
|
|
rig := newBoundRoomRig(t)
|
|
rig.askedInTheRoom(t, "task-1")
|
|
|
|
rig.failedWithReason(t, "task-1", "上下文超出模型限制")
|
|
|
|
got := pushedTexts(t, rig.conn)
|
|
if len(got) != 1 || got[0] != "⚠️ 上下文超出模型限制" {
|
|
t.Fatalf("the asker read %q, want exactly the platform's reason", got)
|
|
}
|
|
}
|
|
|
|
// The same reason, written into the bubble rather than under it, when the
|
|
// round still has one.
|
|
func TestAFailureWithAReasonSealsTheBubbleWithIt(t *testing.T) {
|
|
t.Parallel()
|
|
rig := newBoundRoomRig(t)
|
|
rig.askedInTheRoom(t, "task-1")
|
|
rig.ran(t, "REQ-1", "task-1")
|
|
|
|
rig.failedWithReason(t, "task-1", "上下文超出模型限制")
|
|
|
|
frames := rig.conn.streamFrames(t)
|
|
if len(frames) != 2 || frames[1]["finish"] != true || frames[1]["content"] != "⚠️ 上下文超出模型限制" {
|
|
t.Fatalf("the bubble was left as %v, want it sealed with the platform's reason", frames)
|
|
}
|
|
if got := pushedTexts(t, rig.conn); len(got) != 0 {
|
|
t.Fatalf("the reason ALSO went out as a plain message %q — the asker reads the same failure twice", got)
|
|
}
|
|
}
|
|
|
|
// A failure that arrives with no reason still ends the turn. This is the one
|
|
// place this tree departs from #7952, which stayed silent: silence is fine
|
|
// when nothing is waiting, and wrong here, because the bubble the question
|
|
// opened would spin until the platform's own window closed it.
|
|
func TestAFailureWithNoReasonStillEndsTheTurn(t *testing.T) {
|
|
t.Parallel()
|
|
rig := newBoundRoomRig(t)
|
|
rig.askedInTheRoom(t, "task-1")
|
|
|
|
rig.failed(t, "task-1", false)
|
|
|
|
got := pushedTexts(t, rig.conn)
|
|
if len(got) != 1 || got[0] != streamCopyFailed {
|
|
t.Fatalf("the asker read %q, want the generic ending %q", got, streamCopyFailed)
|
|
}
|
|
}
|
|
|
|
// An attempt the platform is already retrying is not an ending: its
|
|
// replacement is queued, and the retry's answer would land underneath a
|
|
// bubble that had already declared failure.
|
|
func TestARetryPendingFailureSaysNothing(t *testing.T) {
|
|
t.Parallel()
|
|
rig := newBoundRoomRig(t)
|
|
rig.askedInTheRoom(t, "task-1")
|
|
rig.ran(t, "REQ-1", "task-1")
|
|
|
|
rig.failed(t, "task-1", true)
|
|
|
|
if got := pushedTexts(t, rig.conn); len(got) != 0 {
|
|
t.Fatalf("the asker read %q for an attempt the platform is already retrying", got)
|
|
}
|
|
frames := rig.conn.streamFrames(t)
|
|
if len(frames) != 1 {
|
|
t.Fatalf("%d stream frames, want 1 — the bubble stays open for the attempt that reports the real outcome", len(frames))
|
|
}
|
|
}
|
|
|
|
// The guard on the arrangement itself: exactly one party announces a failure.
|
|
// Subscribing in Outbound as well — which is how #7952 shipped upstream, and
|
|
// what a merge from main re-introduces — produces the bubble's ending AND a
|
|
// plain message under it.
|
|
//
|
|
// REVERSE VERIFICATION: add `bus.Subscribe(protocol.EventTaskFailed,
|
|
// o.handleEvent)` back to Outbound.Register and this fails with two.
|
|
func TestAFailedRunProducesExactlyOneMessage(t *testing.T) {
|
|
t.Parallel()
|
|
rig := newBoundRoomRig(t)
|
|
// BOTH subscribers on one bus, the way router.go wires them. Without the
|
|
// Outbound registration this test cannot see the defect it exists for:
|
|
// the indicator alone will always produce exactly one message.
|
|
rig.out.Register(rig.bus)
|
|
rig.askedInTheRoom(t, "task-1")
|
|
rig.ran(t, "REQ-1", "task-1")
|
|
|
|
rig.failedWithReason(t, "task-1", "上下文超出模型限制")
|
|
|
|
// Counting subscribers rather than messages, because the message count
|
|
// depends on which handler the bus reaches first: with Outbound ahead of
|
|
// the indicator it takes the round and seals the bubble with its
|
|
// "nothing to say" copy, and the indicator then sends the failure under
|
|
// it; the other way round Outbound finds the round gone and says nothing.
|
|
// One of those orders is silent, so only the subscription itself is a
|
|
// stable statement of the rule.
|
|
if n := rig.bus.SubscriberCount(protocol.EventTaskFailed); n != 1 {
|
|
t.Fatalf("task:failed has %d subscribers, want exactly 1 — the run's ending has one owner, "+
|
|
"and a second announcer is a second message in somebody's chat", n)
|
|
}
|
|
sealed := 0
|
|
for _, f := range rig.conn.streamFrames(t) {
|
|
if f["finish"] == true {
|
|
sealed++
|
|
}
|
|
}
|
|
if total := sealed + len(pushedTexts(t, rig.conn)); total != 1 {
|
|
t.Fatalf("one failure produced %d message(s) in the chat (%d sealed bubbles, %d plain), want 1",
|
|
total, sealed, len(pushedTexts(t, rig.conn)))
|
|
}
|
|
}
|
|
|
|
// ---- every closer reads the same evidence ----
|
|
|
|
// The answer path learned that an unconfirmed seal must not be said again. The
|
|
// failure and cancellation closers share the rule and did not: writeClosing
|
|
// computed streamUnusable for its log line and then pushed a plain copy
|
|
// regardless.
|
|
//
|
|
// A closing frame whose write was entered may already be in the bubble. WeCom
|
|
// has no unsend, so a notice the asker might read twice is worse than one they
|
|
// can ask for again — the same trade the answer path makes, and there is only
|
|
// one reading of it now.
|
|
func TestAClosingNoticeWhoseOutcomeIsUnknownIsNotSaidAgain(t *testing.T) {
|
|
t.Parallel()
|
|
rig := newBoundRoomRig(t)
|
|
rig.conn.failClosingWrite = errors.New("broken pipe") // entered the write, no verdict
|
|
rig.ran(t, "REQ-UNK", "task-1")
|
|
rig.askedInTheRoom(t, "task-1")
|
|
|
|
rig.failed(t, "task-1", false)
|
|
|
|
if got := pushedTexts(t, rig.conn); len(got) != 0 {
|
|
t.Fatalf("the room was told %q as a plain message after a closing frame that may already "+
|
|
"be in the bubble — WeCom has no unsend, so this copy is permanent", got)
|
|
}
|
|
}
|
|
|
|
// The other half of the same rule: a STATED refusal is proof the words are not
|
|
// on screen, and that is what licenses saying them again. StreamFailed is the
|
|
// only "that run did not go through" WeCom ever produces, so a frame refused
|
|
// for good must not leave the asker with a spinner and no explanation.
|
|
func TestAClosingNoticeThatWasRefusedIsSaidAsAMessage(t *testing.T) {
|
|
t.Parallel()
|
|
rig := newBoundRoomRig(t)
|
|
rig.conn.refuseClosingCode = 846608 // this stream will never take a frame
|
|
rig.ran(t, "REQ-REF", "task-1")
|
|
rig.askedInTheRoom(t, "task-1")
|
|
|
|
rig.failed(t, "task-1", false)
|
|
|
|
got := pushedTexts(t, rig.conn)
|
|
if len(got) != 1 || got[0] != streamCopyFailed {
|
|
t.Fatalf("the room read %q after a closing frame the server refused outright, want [%q]",
|
|
got, streamCopyFailed)
|
|
}
|
|
}
|