mirror of
https://gh-proxy.org/https://github.com/klinsmaya/multica.git
synced 2026-11-08 01:16:45 +08:00
MUL-7227 test(perf): measure comment typing while agent runs stream (#8276)
* test(perf): measure comment typing while runs stream (MUL-7227) The regression that made typing a comment take tens of seconds was a summary animation replacing a DOM node per streamed message, and nothing we had could see it. The component suite runs in jsdom, which has no style recalculation, no layout and no main-thread contention, so several thousand passing tests said nothing about the thing users felt. #8260 pinned the DOM node reuse; this pins what the user actually waits for. One browser scenario, on the real page: app shell, shared CSS, the ProseMirror composer, inline runs, a production Next.js build. Only HTTP and the WebSocket are synthetic, so no API server, database, daemon, agent or account is involved. Nothing disables animations or trims the DOM — the cost being measured lives in exactly that machinery. Fixed workload, identical for both builds under comparison: 80 comments with prose and code, three running agents, 1000 seeded transcript messages each, 60 ticks of live messages 100ms apart, and 172 keystrokes at 25ms. Messages are scheduled from Node rather than the page, so a stalling build cannot quietly measure less work than the one it is being compared against. `typing_elapsed_ms` is what the user waits for: first keystroke to the editor holding the whole string, main-thread queueing included. Style recalculation, layout, task time and long tasks say where it went. An empty page types very fast, so a run whose messages never reached the UI, or whose composer did not end up holding every character, is reported as `invalid` or `timeout` and never as a duration — including when the scenario blows its budget, which is the case whose numbers matter most. Reports rather than gates. `scripts/perf-compare.mjs` builds and measures two refs in sequence on one machine, each installed against its own lockfile, with the spec, fixture and browser always taken from the running tree so the product is the only difference. The workflow is separate and not required: one sample per ref cannot separate a small regression from machine noise, and the noise floor has not been calibrated on CI hardware yet. Verified against the known regression: restoring the pre-#8260 summary animation moves style recalculation from ~130ms to 1577ms and typing from ~4.9s to 6.2s, against a spread of ~1% across three runs of the fixed build. Disconnecting the message injection reports `timeout: run 0 end marker`, and typing fewer characters reports `timeout: editor content` — neither can pass as fast. #8260's DOM reuse test is untouched and still green. Tests and run configuration only: no product change. Co-authored-by: multica-agent <[email protected]> * test(perf): always build for real when comparing two refs (MUL-7227) The comparison reports a build time, and turbo's cache made it read as ~1.7s when the real build is ~60s — understating what this costs CI by a factor of thirty. Worse, a run where one ref hit the cache and the other missed would put that difference in the timing split as if it belonged to the products. `--force` on both sides: two real builds, two comparable numbers. Co-authored-by: multica-agent <[email protected]> * test(perf): report cached builds instead of forcing real ones (MUL-7227) Forcing a real build on both sides cost minutes on every local run to remove an ambiguity that a single line of output removes instead. A restored build is byte-identical to the one that produced it, so the cache cannot move the numbers this script collects — only the build time it reports, which is now labelled when it came from the cache. It buys nothing on CI either: the runner is cold every time, so both builds are real whether or not they are forced. Locally a repeat comparison drops from roughly three minutes to under one. Co-authored-by: multica-agent <[email protected]> * test(perf): run the comparison on demand, and let it exit (MUL-7227) The comparison never exited on success. The frontend server is spawned detached, and its teardown hung off `process.on("exit")` — but a live child keeps the event loop running, so `exit` never fired and the child was never killed. The report was written and the process then waited on the server it was supposed to stop. On CI that ran until the job was cancelled at sixty minutes; locally it left the servers running after being killed. The failure path only terminated because it happened to call `process.exit(1)`. Teardown is explicit now. Each side stops its server, waits until the process has exited and the port has stopped answering, and removes its worktree before the next side starts — so the second build is also measured on a machine with nothing of the first one running. The script then exits explicitly, and the `exit` and signal handlers remain only as a last resort for cancellation. Checked on every path, with a harness that records the gap between the report and the exit and looks for anything left behind: success, a scenario failure with a server running, an unknown ref, and SIGTERM mid-run all exit promptly with no server, worktree or temp directory remaining. Run through the same harness, the previous script wrote its report, then was still running 247s later and left three servers and two worktrees behind. The workflow is manual only, capped at fifteen minutes. On CI one comparison took 340s — 292s of it the two production builds, 26s the two measurements — which is a lot to spend on every frontend PR for a report that gates nothing. The routine guard stays the component test from #8260, which pins the exact failure mode and runs with the normal suite; this is for changes that touch summary animation, long lists, transcript rendering or global styles. No change to the scenario or its metrics. Co-authored-by: multica-agent <[email protected]> * test(perf): only trust this run's report, and wait for the whole server (MUL-7227) Two ways the comparison could report something that was not true. A report left in the output directory by an earlier run was read back when this run's scenario failed before writing one, so two failed scenarios showed the previous run's timings, were marked a usable comparison, and exited 0. And a scenario that wrote `ok` and then failed was passed through too: the exit code was recorded as `spec_failed` and then ignored. Each side's report is now deleted before the scenario runs, the files this script writes are cleared up front, and a failing scenario process overrides an `ok` report — whatever failed after the numbers were written is exactly what nobody has looked at. And a server that stopped answering was taken for one that had stopped. Any fetch error counted as a closed port, a timeout included, and the server was dropped from the last-resort cleanup before anything had been confirmed. With the pnpm wrapper gone but a child still holding the port, the script exited 0 and left two servers running. Teardown now waits for the whole process group to exit — escalating to SIGKILL if it outlives SIGTERM — keeps the server registered until then, and treats the port as free only when a connection is refused, since a hung server still completes the handshake. `scripts/perf-compare.test.sh` drives the real script with a fake pnpm — no build, no browser, 9 seconds — through success, a scenario that writes nothing, a stale report, `ok` followed by failure, a child that ignores SIGTERM, and an unknown ref. Each of the three new cases fails against the previous script for the reason above. It runs in `frontend-build` like the other script tests; the browser comparison itself stays manual. `PERF_STOP_GRACE_MS` sets the SIGTERM grace (default 10s) so the test can exercise the SIGKILL path without waiting on it. Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: J <[email protected]> Co-authored-by: multica-agent <[email protected]>
This commit is contained in:
Bohan Jiang
co-authored by
multica-agent
J
multica-agent
J
parent
51803b3b3b
commit
6af2268bd9
@@ -99,6 +99,8 @@ jobs:
|
||||
- 'scripts/worktree-db.test.sh'
|
||||
- 'scripts/dev-env.sh'
|
||||
- 'scripts/dev-env.test.sh'
|
||||
- 'scripts/perf-compare.mjs'
|
||||
- 'scripts/perf-compare.test.sh'
|
||||
- 'scripts/selfhost-wait.sh'
|
||||
# selfhost-config.test.sh asserts on the installers' side of
|
||||
# the port contract, so a change to either must run it.
|
||||
@@ -295,6 +297,13 @@ jobs:
|
||||
if: ${{ needs.changes.outputs.frontend == 'true' }}
|
||||
run: bash scripts/dev-env.test.sh
|
||||
|
||||
# The browser comparison runs only on demand, but its verdict and its
|
||||
# cleanup are cheap to prove on every change: a fake pnpm, no build and
|
||||
# no browser, a few seconds.
|
||||
- name: Test UI performance comparison runner
|
||||
if: ${{ needs.changes.outputs.frontend == 'true' }}
|
||||
run: bash scripts/perf-compare.test.sh
|
||||
|
||||
- name: Verify reserved-slugs.ts is up to date
|
||||
if: ${{ needs.changes.outputs.frontend == 'true' }}
|
||||
# Re-runs the generator and fails on any drift from the
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
name: UI performance report
|
||||
|
||||
# Manual only. This builds the frontend twice — about five minutes of runner
|
||||
# time on CI — to produce one sample per ref, which is a report and not a
|
||||
# verdict. Running it on every frontend PR would spend that on reports nobody
|
||||
# asked for, and a job that is not required still takes a runner and a
|
||||
# concurrency slot.
|
||||
#
|
||||
# The routine guard is the component test #8260 added, which pins the exact
|
||||
# failure mode behind MUL-7227 and runs in the normal suite. Run this when a
|
||||
# change touches summary animation, long lists, transcript rendering or global
|
||||
# styles, and attach the report to the PR.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
base:
|
||||
description: "Base ref (branch, tag or SHA)"
|
||||
required: true
|
||||
head:
|
||||
description: "Head ref (branch, tag or SHA)"
|
||||
required: true
|
||||
|
||||
# Read-only and without secrets: it builds and runs whatever refs it is given.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ui-performance-${{ inputs.base }}-${{ inputs.head }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
compare:
|
||||
runs-on: ubuntu-latest
|
||||
# Measured on CI: ~340s for the comparison itself, of which ~292s is the
|
||||
# two builds and ~26s the two measurements, plus setup. Fifteen minutes
|
||||
# leaves room for build variance and still stops a run that has hung.
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
# Both refs must be reachable: the script builds each one in its own
|
||||
# worktree, and a shallow clone cannot produce them.
|
||||
fetch-depth: 0
|
||||
|
||||
- uses: pnpm/action-setup@v4
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 22
|
||||
cache: pnpm
|
||||
|
||||
- name: Install (runner tree — supplies the spec and the browser)
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Install Chromium
|
||||
run: pnpm exec playwright install --with-deps chromium
|
||||
|
||||
- name: Compare
|
||||
run: node scripts/perf-compare.mjs --base "${{ inputs.base }}" --head "${{ inputs.head }}" --out perf-report
|
||||
|
||||
- name: Publish to the job summary
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f perf-report/comparison.md ]; then
|
||||
cat perf-report/comparison.md >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
echo "The comparison produced no report; see the job log." >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
- name: Upload reports
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ui-performance-report
|
||||
path: perf-report/
|
||||
if-no-files-found: warn
|
||||
@@ -78,3 +78,4 @@ data/
|
||||
# Python (agent tooling)
|
||||
__pycache__/
|
||||
*.pyc
|
||||
perf-report/
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# UI performance scenario
|
||||
|
||||
One browser scenario: typing a comment on a long issue thread while three agent
|
||||
runs stream progress. It exists because of MUL-7227, where a summary animation
|
||||
replaced a DOM node per streamed message and typing a comment slowed to a crawl.
|
||||
Component tests could not see it — they run in jsdom, which has no style
|
||||
recalculation, no layout and no main-thread contention.
|
||||
|
||||
## What it runs against
|
||||
|
||||
The real Next.js production build, the real page, the real shell and CSS, the
|
||||
real ProseMirror composer. Only HTTP and the WebSocket are synthetic, so there
|
||||
is no API server, database, daemon, agent or account involved. Nothing disables
|
||||
animations or trims the DOM: the cost being measured lives in that machinery.
|
||||
|
||||
It covers the renderer shared by Web and Desktop. Electron's own main-process
|
||||
behaviour is not exercised here.
|
||||
|
||||
## Running it
|
||||
|
||||
Against a frontend you already have running:
|
||||
|
||||
```bash
|
||||
pnpm exec playwright test --config=playwright.perf.config.ts
|
||||
# PLAYWRIGHT_BASE_URL defaults to http://localhost:3000
|
||||
# PERF_REPORT_PATH=<file> also writes the JSON report
|
||||
```
|
||||
|
||||
Comparing two refs — this builds and measures each one, in sequence:
|
||||
|
||||
```bash
|
||||
node scripts/perf-compare.mjs --base origin/main --head HEAD --out perf-report
|
||||
```
|
||||
|
||||
The comparison needs Node, pnpm and Chromium. It writes `comparison.json` and
|
||||
`comparison.md`. Each side stops its server, waits for it to exit and removes
|
||||
its worktree before the next side starts, and the script exits on its own on
|
||||
success, on failure and on cancellation.
|
||||
|
||||
The ordinary e2e suite ignores this directory; it needs a production build and
|
||||
a machine that is not busy doing anything else.
|
||||
|
||||
## When to run it
|
||||
|
||||
Nothing runs this automatically. The routine guard is the component test from
|
||||
#8260, which pins the exact failure mode behind MUL-7227 — a streamed message
|
||||
replacing the summary's DOM node — and runs with the normal suite.
|
||||
|
||||
Run the comparison yourself, or through the **UI performance report** workflow
|
||||
(`workflow_dispatch`, base and head refs as inputs), when a change touches
|
||||
summary animation, long lists, transcript rendering or global styles — and
|
||||
attach the report to the PR. The cost of doing it by hand instead of on every
|
||||
PR is that a regression outside those areas will not be caught automatically.
|
||||
|
||||
## What it costs
|
||||
|
||||
On a GitHub-hosted `ubuntu-latest` runner the comparison took 340.5 s: 291.5 s
|
||||
for the two production builds, 26 s for the two measurements, the rest install
|
||||
and startup — plus checkout, dependency and Chromium setup around it. The job is
|
||||
capped at 15 minutes. Locally, a repeat comparison whose builds come out of the
|
||||
turbo cache takes under a minute; the report says when a build was restored.
|
||||
|
||||
## Reading the result
|
||||
|
||||
`typing_elapsed_ms` is the primary number: how long the user waits from the
|
||||
first keystroke until the editor holds the whole string, main-thread queueing
|
||||
included. `recalc_style_ms`, `layout_ms`, `task_ms` and the long-task counters
|
||||
say where the time went.
|
||||
|
||||
Two things are not the same as a fast run, and the scenario reports them as
|
||||
`invalid` or `timeout` rather than as a duration: messages that never reached
|
||||
the UI, and a composer that did not end up holding every character. An empty
|
||||
page types very quickly.
|
||||
|
||||
One sample per ref. Read a small difference as noise until a second run says
|
||||
otherwise. Locally the spread across three runs of one build was around 1%; on
|
||||
CI, comparing two refs with no product difference between them came out 2%
|
||||
apart on typing and on style recalculation. A threshold needs more CI samples
|
||||
than that before it can be trusted.
|
||||
|
||||
## Changing the fixture
|
||||
|
||||
`fixtures/typing-under-live-runs.ts` generates every byte the scenario feeds the
|
||||
product, and the report carries the fixture version and a digest of it. Bump
|
||||
`FIXTURE_VERSION` whenever the generated bytes change — two runs are only
|
||||
comparable when the digests match, which the comparison report states.
|
||||
|
||||
Do not add DOM to the page to keep the pressure up. A future virtualization
|
||||
change that legitimately renders fewer nodes should show up as fewer nodes.
|
||||
@@ -0,0 +1,193 @@
|
||||
/**
|
||||
* Fixture for the comment-typing performance scenario (MUL-7227).
|
||||
*
|
||||
* One shape, generated deterministically, so that two product builds under
|
||||
* comparison receive byte-identical data. Everything here is synthetic: no
|
||||
* account, no workspace data, no captured transcript. Bump FIXTURE_VERSION on
|
||||
* any change that alters the generated bytes — the report carries the version
|
||||
* and a digest so two runs can be shown to have measured the same work.
|
||||
*/
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
export const FIXTURE_VERSION = "1.0.0";
|
||||
|
||||
export const WORKSPACE_SLUG = "perf";
|
||||
export const WORKSPACE_ID = "11111111-1111-4111-8111-111111111111";
|
||||
export const USER_ID = "22222222-2222-4222-8222-222222222222";
|
||||
export const ISSUE_ID = "33333333-3333-4333-8333-333333333333";
|
||||
export const ISSUE_IDENTIFIER = "PERF-1";
|
||||
|
||||
/** 1 root comment + 79 replies, matching the thread that reproduced the stall. */
|
||||
export const ROOT_COMMENT_COUNT = 1;
|
||||
export const REPLY_COMMENT_COUNT = 79;
|
||||
/** Messages already in each run's transcript when the page opens. */
|
||||
export const INITIAL_MESSAGES_PER_RUN = 1000;
|
||||
/** Live stream: one tool_use/tool_result pair per run per tick. */
|
||||
export const TICK_COUNT = 60;
|
||||
export const TICK_INTERVAL_MS = 100;
|
||||
export const RUN_COUNT = 3;
|
||||
/** Fixed keystroke load. Not submitted — this measures typing, not posting. */
|
||||
export const TYPING_TEXT = "abcdefghijklmnopqrstuvwxyz".repeat(7).slice(0, 172);
|
||||
export const KEY_DELAY_MS = 25;
|
||||
|
||||
const uuid = (prefix: string, n: number) =>
|
||||
`${prefix}${String(n).padStart(4, "0")}-4444-4444-8444-444444444444`.slice(0, 36);
|
||||
|
||||
export const runIds = Array.from({ length: RUN_COUNT }, (_, i) => uuid("aaaa", i));
|
||||
export const agentIds = Array.from({ length: RUN_COUNT }, (_, i) => uuid("bbbb", i));
|
||||
export const commentIds = Array.from(
|
||||
{ length: ROOT_COMMENT_COUNT + REPLY_COMMENT_COUNT },
|
||||
(_, i) => uuid("cccc", i),
|
||||
);
|
||||
export const rootCommentId = commentIds[0]!;
|
||||
|
||||
const at = (offsetSeconds: number) =>
|
||||
new Date(Date.UTC(2026, 8, 1, 0, 0, 0) + offsetSeconds * 1000).toISOString();
|
||||
|
||||
/**
|
||||
* A comment body with the shape that makes this page expensive to lay out:
|
||||
* prose paragraphs plus a highlighted code block, as the reproduction had.
|
||||
*/
|
||||
function commentBody(index: number): string {
|
||||
const paragraphs = Array.from(
|
||||
{ length: 20 },
|
||||
(_, p) =>
|
||||
`Paragraph ${p + 1} of comment ${index}. The reviewer walks through the ` +
|
||||
`change, its blast radius, and what was measured before and after.`,
|
||||
);
|
||||
const code = [
|
||||
"```ts",
|
||||
`export function handler${index}(input: Record<string, unknown>) {`,
|
||||
" const entries = Object.entries(input);",
|
||||
" return entries.map(([key, value]) => ({ key, value }));",
|
||||
"}",
|
||||
"```",
|
||||
].join("\n");
|
||||
return `${paragraphs.join("\n\n")}\n\n${code}`;
|
||||
}
|
||||
|
||||
export function buildTimelineEntries() {
|
||||
return commentIds.map((id, index) => ({
|
||||
type: "comment" as const,
|
||||
id,
|
||||
actor_type: "user" as const,
|
||||
actor_id: USER_ID,
|
||||
created_at: at(index),
|
||||
updated_at: at(index),
|
||||
issue_id: ISSUE_ID,
|
||||
// Replies nest inside the root's card, which is the thread shape that
|
||||
// mounts every body at once.
|
||||
parent_id: index === 0 ? null : rootCommentId,
|
||||
content: commentBody(index),
|
||||
resolved_at: null,
|
||||
reactions: [],
|
||||
attachments: [],
|
||||
source_task_id: null,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs carry no trigger comment on purpose: a run anchored to a comment that
|
||||
* is not in the timeline is dropped silently, while an unanchored run renders
|
||||
* at the live end of the thread — directly above the composer, which is where
|
||||
* the reproduction had them.
|
||||
*/
|
||||
export function buildTaskRuns() {
|
||||
return runIds.map((id, index) => ({
|
||||
id,
|
||||
agent_id: agentIds[index]!,
|
||||
runtime_id: uuid("dddd", index),
|
||||
issue_id: ISSUE_ID,
|
||||
status: "running" as const,
|
||||
priority: 0,
|
||||
created_at: at(100 + index),
|
||||
dispatched_at: at(100 + index),
|
||||
started_at: at(100 + index),
|
||||
completed_at: null,
|
||||
result: null,
|
||||
error: null,
|
||||
}));
|
||||
}
|
||||
|
||||
/** Short, stable tool argument summaries; see `commandForTick`. */
|
||||
const COMMANDS = [
|
||||
"pnpm test packages/core",
|
||||
"pnpm lint packages/views",
|
||||
"pnpm typecheck",
|
||||
"rg --files packages/ui",
|
||||
"pnpm build --filter web",
|
||||
];
|
||||
|
||||
/**
|
||||
* Two consecutive ticks reuse one command, then it changes. The repeat is the
|
||||
* case the regression turned on — a new seq whose displayed text is identical
|
||||
* — and the change covers an ordinary update.
|
||||
*/
|
||||
export function commandForTick(tick: number): string {
|
||||
return COMMANDS[Math.floor(tick / 2) % COMMANDS.length]!;
|
||||
}
|
||||
|
||||
export const endMarkerFor = (runIndex: number) => `perf-final-${runIndex}`;
|
||||
|
||||
/** Seq layout: the initial transcript owns [0, INITIAL_MESSAGES_PER_RUN). */
|
||||
export const seqForTick = (tick: number, half: 0 | 1) =>
|
||||
INITIAL_MESSAGES_PER_RUN + tick * 2 + half;
|
||||
|
||||
export function buildInitialMessages(runIndex: number) {
|
||||
const taskId = runIds[runIndex]!;
|
||||
const rows: Record<string, unknown>[] = [];
|
||||
for (let seq = 0; seq < INITIAL_MESSAGES_PER_RUN; seq += 2) {
|
||||
const command = COMMANDS[(seq / 2) % COMMANDS.length]!;
|
||||
rows.push({
|
||||
task_id: taskId, issue_id: ISSUE_ID, seq, type: "tool_use",
|
||||
tool: "exec_command", input: { command }, created_at: at(200 + seq),
|
||||
});
|
||||
rows.push({
|
||||
task_id: taskId, issue_id: ISSUE_ID, seq: seq + 1, type: "tool_result",
|
||||
tool: "exec_command",
|
||||
output: `ok: ${command}\n`.repeat(8),
|
||||
output_truncated: false,
|
||||
created_at: at(200 + seq + 1),
|
||||
});
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
/** The pair pushed over the socket for one run on one tick. */
|
||||
export function buildTickMessages(runIndex: number, tick: number) {
|
||||
const taskId = runIds[runIndex]!;
|
||||
const last = tick === TICK_COUNT - 1;
|
||||
const command = last ? endMarkerFor(runIndex) : commandForTick(tick);
|
||||
return [
|
||||
{
|
||||
task_id: taskId, issue_id: ISSUE_ID, seq: seqForTick(tick, 0),
|
||||
type: "tool_use", tool: "exec_command", input: { command },
|
||||
created_at: at(400 + tick * 2),
|
||||
},
|
||||
{
|
||||
task_id: taskId, issue_id: ISSUE_ID, seq: seqForTick(tick, 1),
|
||||
type: "tool_result", tool: "exec_command",
|
||||
output: `ok: ${command}\n`.repeat(8), output_truncated: false,
|
||||
created_at: at(400 + tick * 2 + 1),
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
/** Every byte the scenario feeds the product, for the report's digest. */
|
||||
export function fixtureDigest(): { version: string; sha256: string; bytes: number } {
|
||||
const payload = JSON.stringify({
|
||||
version: FIXTURE_VERSION,
|
||||
timeline: buildTimelineEntries(),
|
||||
runs: buildTaskRuns(),
|
||||
initial: runIds.map((_, i) => buildInitialMessages(i)),
|
||||
ticks: runIds.map((_, i) =>
|
||||
Array.from({ length: TICK_COUNT }, (_, t) => buildTickMessages(i, t)),
|
||||
),
|
||||
typing: TYPING_TEXT,
|
||||
});
|
||||
return {
|
||||
version: FIXTURE_VERSION,
|
||||
sha256: createHash("sha256").update(payload).digest("hex").slice(0, 16),
|
||||
bytes: Buffer.byteLength(payload),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,427 @@
|
||||
/**
|
||||
* Comment typing under live agent runs (MUL-7227).
|
||||
*
|
||||
* #8172 keyed the run summary by message seq, so every streamed message
|
||||
* replaced a DOM node and the page paid a style recalculation for it. Typing a
|
||||
* comment on a long thread went from seconds to tens of seconds. #8260 restored
|
||||
* a stable node, and its unit test pins that the node is reused — but a jsdom
|
||||
* test cannot see style recalculation, layout, or main-thread contention. This
|
||||
* scenario measures what the user actually waits for.
|
||||
*
|
||||
* The whole page is real: app shell, sidebar, shared CSS, the ProseMirror
|
||||
* composer, inline runs. Only the HTTP and WebSocket boundaries are synthetic,
|
||||
* so no API, database, daemon or account is involved. Nothing here disables
|
||||
* animations or trims the DOM to reduce noise — the cost being measured lives
|
||||
* in exactly that machinery.
|
||||
*
|
||||
* The result is a report, not a gate: one sample per build cannot separate a
|
||||
* small regression from machine noise. The test still fails outright when the
|
||||
* scenario did not happen — messages that never reached the UI or a composer
|
||||
* that did not receive every keystroke must never read as "fast".
|
||||
*/
|
||||
import { expect, test, type Page, type WebSocketRoute } from "@playwright/test";
|
||||
import { writeFileSync, mkdirSync } from "node:fs";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import * as fx from "./fixtures/typing-under-live-runs";
|
||||
|
||||
/** Playwright's own cap. The scenario's validity cap is SCENARIO_LIMIT_MS. */
|
||||
test.setTimeout(300_000);
|
||||
/** Beyond this the sample is reported as a timeout rather than as a duration. */
|
||||
const SCENARIO_LIMIT_MS = 60_000;
|
||||
/** Loading, seeding and focusing share one budget, kept clear of the scenario's. */
|
||||
const SETUP_LIMIT_MS = 120_000;
|
||||
|
||||
test.use({
|
||||
viewport: { width: 1280, height: 720 },
|
||||
locale: "en-US",
|
||||
colorScheme: "light",
|
||||
// A worker could serve responses the route table never saw.
|
||||
serviceWorkers: "block",
|
||||
});
|
||||
|
||||
const json = (body: unknown) => ({
|
||||
status: 200,
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
|
||||
const workspace = {
|
||||
id: fx.WORKSPACE_ID, name: "Perf", slug: fx.WORKSPACE_SLUG,
|
||||
description: null, context: null,
|
||||
settings: { github_enabled: false }, repos: [],
|
||||
issue_prefix: "PERF", avatar_url: null,
|
||||
created_at: "2026-09-01T00:00:00Z", updated_at: "2026-09-01T00:00:00Z",
|
||||
};
|
||||
|
||||
const me = {
|
||||
id: fx.USER_ID, name: "Perf User", email: "[email protected]", avatar_url: null,
|
||||
language: null, onboarded_at: "2026-09-01T00:00:00Z",
|
||||
onboarding_questionnaire: { source: "other" },
|
||||
created_at: "2026-09-01T00:00:00Z", updated_at: "2026-09-01T00:00:00Z",
|
||||
};
|
||||
|
||||
const issue = {
|
||||
id: fx.ISSUE_ID, workspace_id: fx.WORKSPACE_ID, number: 1,
|
||||
identifier: fx.ISSUE_IDENTIFIER, title: "Typing under live runs",
|
||||
description: "Performance scenario fixture.", status: "in_progress",
|
||||
status_category: "in_progress", status_name: "In Progress", priority: "none",
|
||||
assignee_type: null, assignee_id: null, creator_type: "user", creator_id: fx.USER_ID,
|
||||
parent_issue_id: null, project_id: null, position: 0,
|
||||
start_date: null, due_date: null, stage: null, metadata: {}, properties: {},
|
||||
labels: [], reactions: [],
|
||||
created_at: "2026-09-01T00:00:00Z", updated_at: "2026-09-01T00:00:00Z",
|
||||
};
|
||||
|
||||
/**
|
||||
* Every request the page is allowed to make. Anything else invalidates the
|
||||
* sample: a business call answered by a catch-all would quietly change what is
|
||||
* being measured.
|
||||
*/
|
||||
function responseFor(pathname: string): unknown | undefined {
|
||||
const messages = /^\/api\/tasks\/([0-9a-f-]+)\/messages$/.exec(pathname);
|
||||
if (messages) {
|
||||
const index = fx.runIds.indexOf(messages[1]!);
|
||||
return index >= 0 ? fx.buildInitialMessages(index) : undefined;
|
||||
}
|
||||
switch (pathname) {
|
||||
case "/api/config": return { feature_flags: {}, allow_signup: true };
|
||||
case "/api/me": return me;
|
||||
case "/api/workspaces": return [workspace];
|
||||
case "/api/client-usage": return {};
|
||||
case "/api/agents": case "/api/runtimes": case "/api/squads":
|
||||
case "/api/agent-task-snapshot": case "/api/invitations": case "/api/inbox":
|
||||
case "/api/inbox/unread-summary": case "/api/pins": case "/api/chat/sessions":
|
||||
return [];
|
||||
case "/api/chat/pending-tasks/has-any": return { has_pending: false };
|
||||
case `/api/workspaces/${fx.WORKSPACE_ID}/members`: return [];
|
||||
case "/api/projects": return { projects: [], total: 0 };
|
||||
case "/api/issues": return { issues: [], total: 0 };
|
||||
case "/api/issue-statuses": return { statuses: [] };
|
||||
case "/api/properties": return { properties: [], total: 0 };
|
||||
case "/api/quick-actions": return { quick_actions: [] };
|
||||
case "/api/assignee-frequency": return [];
|
||||
// Typing debounces a trigger preview. It is part of the cost being
|
||||
// measured, so it is answered rather than blocked.
|
||||
case `/api/issues/${fx.ISSUE_ID}/comments/trigger-preview`:
|
||||
return { agents: [], blocked: [] };
|
||||
case "/api/issues/child-progress": return { progress: [] };
|
||||
case `/api/issues/${fx.ISSUE_IDENTIFIER}`:
|
||||
case `/api/issues/${fx.ISSUE_ID}`: return issue;
|
||||
case `/api/issues/${fx.ISSUE_ID}/timeline`: return fx.buildTimelineEntries();
|
||||
case `/api/issues/${fx.ISSUE_ID}/task-runs`: return fx.buildTaskRuns();
|
||||
case `/api/issues/${fx.ISSUE_ID}/subscribers`:
|
||||
case `/api/issues/${fx.ISSUE_ID}/attachments`: return [];
|
||||
case `/api/issues/${fx.ISSUE_ID}/labels`: return { labels: [] };
|
||||
case `/api/issues/${fx.ISSUE_ID}/children`: return { issues: [] };
|
||||
case `/api/issues/${fx.ISSUE_ID}/pull-requests`: return { pull_requests: [] };
|
||||
default: return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
const nextFrame = (page: Page) =>
|
||||
page.evaluate(() => new Promise<void>((done) => {
|
||||
requestAnimationFrame(() => requestAnimationFrame(() => done()));
|
||||
}));
|
||||
|
||||
/** Writes the report wherever the run can see it, whatever the outcome. */
|
||||
function emit(report: Record<string, unknown>) {
|
||||
const out = process.env.PERF_REPORT_PATH;
|
||||
if (out) {
|
||||
mkdirSync(dirname(resolve(out)), { recursive: true });
|
||||
writeFileSync(resolve(out), JSON.stringify(report, null, 2));
|
||||
}
|
||||
test.info().annotations.push({ type: "perf", description: JSON.stringify(report) });
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`PERF_REPORT ${JSON.stringify(report)}`);
|
||||
}
|
||||
|
||||
test("comment typing stays responsive while runs stream", async ({ page, context, baseURL }) => {
|
||||
const origin = new URL(baseURL ?? "http://localhost:3000").origin;
|
||||
const unknownRequests: string[] = [];
|
||||
const pageErrors: string[] = [];
|
||||
const external: string[] = [];
|
||||
|
||||
page.on("pageerror", (error) => pageErrors.push(error.message));
|
||||
// Motion must be on: the regression this scenario exists for is skipped
|
||||
// under reduced motion, so measuring with it enabled would measure nothing.
|
||||
await page.emulateMedia({ colorScheme: "light", reducedMotion: "no-preference" });
|
||||
|
||||
await context.addCookies([
|
||||
{ name: "multica_logged_in", value: "1", url: origin },
|
||||
{ name: "last_workspace_slug", value: fx.WORKSPACE_SLUG, url: origin },
|
||||
]);
|
||||
// The floating chat mounts its own queries on every page; this scenario is
|
||||
// about the issue thread, and those requests are not part of it.
|
||||
await context.addInitScript(() => {
|
||||
window.localStorage.setItem("multica:chat:floatingChatEnabled", "false");
|
||||
});
|
||||
// Long tasks are read from the browser's own observer rather than a
|
||||
// threshold invented here. Records still queued at the end are drained.
|
||||
await context.addInitScript(() => {
|
||||
const store: { duration: number }[] = [];
|
||||
(window as unknown as { __perfLongTasks: typeof store }).__perfLongTasks = store;
|
||||
const observer = new PerformanceObserver((list) => {
|
||||
for (const entry of list.getEntries()) store.push({ duration: entry.duration });
|
||||
});
|
||||
observer.observe({ type: "longtask", buffered: true });
|
||||
(window as unknown as { __perfDrain: () => void }).__perfDrain = () => {
|
||||
for (const entry of observer.takeRecords()) store.push({ duration: entry.duration });
|
||||
};
|
||||
});
|
||||
|
||||
await context.route("**/*", async (route) => {
|
||||
const url = new URL(route.request().url());
|
||||
if (url.origin !== origin) {
|
||||
external.push(url.href);
|
||||
return route.abort();
|
||||
}
|
||||
if (!url.pathname.startsWith("/api/")) return route.continue();
|
||||
const body = responseFor(url.pathname);
|
||||
if (body === undefined) {
|
||||
unknownRequests.push(`${route.request().method()} ${url.pathname}${url.search}`);
|
||||
return route.fulfill({ status: 404, contentType: "application/json", body: "{}" });
|
||||
}
|
||||
return route.fulfill(json(body));
|
||||
});
|
||||
|
||||
let socket!: WebSocketRoute;
|
||||
let onSocket!: () => void;
|
||||
const socketReady = new Promise<void>((done) => { onSocket = done; });
|
||||
await context.routeWebSocket((url) => url.pathname === "/ws", (ws) => {
|
||||
socket = ws;
|
||||
// Cookie mode: the client sends nothing and treats open as authenticated.
|
||||
ws.onMessage(() => {});
|
||||
onSocket();
|
||||
});
|
||||
|
||||
const push = (runIndex: number, tick: number) => {
|
||||
for (const payload of fx.buildTickMessages(runIndex, tick)) {
|
||||
socket.send(JSON.stringify({
|
||||
type: "task:message", payload, actor_id: "", actor_type: "system",
|
||||
}));
|
||||
}
|
||||
};
|
||||
|
||||
const setupDeadline = Date.now() + SETUP_LIMIT_MS;
|
||||
const setupLeft = () => Math.max(1, setupDeadline - Date.now());
|
||||
const failSetup = (stage: string) => {
|
||||
emit({
|
||||
status: "invalid",
|
||||
invalid: [`setup did not complete: ${stage}`],
|
||||
fixture: fx.fixtureDigest(),
|
||||
page_errors: pageErrors,
|
||||
unmocked_requests: unknownRequests,
|
||||
typing_elapsed_ms: null,
|
||||
scenario_elapsed_ms: null,
|
||||
});
|
||||
};
|
||||
|
||||
// ---- 1. Load, and prove the fixture is actually on screen -----------------
|
||||
await page.goto(
|
||||
`${origin}/${fx.WORKSPACE_SLUG}/issues/${fx.ISSUE_IDENTIFIER}#comment-${fx.rootCommentId}`,
|
||||
{ waitUntil: "domcontentloaded", timeout: setupLeft() },
|
||||
);
|
||||
// A page that error-boundaries never opens a socket; without a bound this
|
||||
// wait would hang past every budget below it.
|
||||
await Promise.race([
|
||||
socketReady,
|
||||
new Promise<void>((_, reject) =>
|
||||
setTimeout(() => reject(new Error("the app never opened a WebSocket")), setupLeft()),
|
||||
),
|
||||
]).catch((error: Error) => {
|
||||
failSetup(`${error.message}${pageErrors.length ? ` | ${pageErrors.join(" | ")}` : ""}`);
|
||||
throw error;
|
||||
});
|
||||
|
||||
const summaries = page.locator("[data-run-summary]");
|
||||
try {
|
||||
await expect(summaries).toHaveCount(fx.RUN_COUNT, { timeout: setupLeft() });
|
||||
} catch (error) { failSetup("run summaries never rendered"); throw error; }
|
||||
const bodies = page.locator("[data-comment-block]");
|
||||
try {
|
||||
await expect
|
||||
.poll(() => bodies.count(), { timeout: setupLeft() })
|
||||
.toBeGreaterThanOrEqual(fx.ROOT_COMMENT_COUNT + fx.REPLY_COMMENT_COUNT);
|
||||
} catch (error) { failSetup("comment bodies never rendered"); throw error; }
|
||||
// Activity detail stays collapsed: the row shows a summary, not a step list.
|
||||
await expect(page.locator("[data-run-summary] >> nth=0")).toBeVisible();
|
||||
await page.evaluate(() => document.fonts?.ready);
|
||||
await nextFrame(page);
|
||||
|
||||
// ---- 2. Preparation: prove messages reach the real render path ------------
|
||||
for (let runIndex = 0; runIndex < fx.RUN_COUNT; runIndex += 1) push(runIndex, 0);
|
||||
try {
|
||||
for (let runIndex = 0; runIndex < fx.RUN_COUNT; runIndex += 1) {
|
||||
await expect(summaries.nth(runIndex))
|
||||
.toHaveText(fx.commandForTick(0), { timeout: setupLeft() });
|
||||
}
|
||||
} catch (error) { failSetup("streamed message never reached a summary"); throw error; }
|
||||
|
||||
// ---- 3. Focus the composer -----------------------------------------------
|
||||
const shell = page.getByTestId("comment-composer-shell");
|
||||
await expect(shell).toBeVisible();
|
||||
// Resolve the composer's container before activating it: the shell is removed
|
||||
// once the real editor mounts, and the issue description is a second editable
|
||||
// editor on this page — keystrokes must be proven to land in the composer.
|
||||
const composer = await page
|
||||
.locator('div:has(> [data-testid="comment-composer-shell"])')
|
||||
.first()
|
||||
.elementHandle();
|
||||
if (!composer) throw new Error("comment composer container not found");
|
||||
await shell.click();
|
||||
const editor = await composer.waitForSelector('.ProseMirror[contenteditable="true"]', {
|
||||
timeout: setupLeft(),
|
||||
});
|
||||
expect(await composer.$$('.ProseMirror[contenteditable="true"]')).toHaveLength(1);
|
||||
await editor.click();
|
||||
expect(await editor.evaluate((node) => node === document.activeElement)).toBe(true);
|
||||
expect((await editor.innerText()).trim()).toBe("");
|
||||
|
||||
const cdp = await context.newCDPSession(page);
|
||||
await cdp.send("Performance.enable");
|
||||
const readMetrics = async () => {
|
||||
const { metrics } = await cdp.send("Performance.getMetrics");
|
||||
return Object.fromEntries(metrics.map((m) => [m.name, m.value]));
|
||||
};
|
||||
const before = await readMetrics();
|
||||
await page.evaluate(() => {
|
||||
(window as unknown as { __perfLongTasks: unknown[] }).__perfLongTasks.length = 0;
|
||||
});
|
||||
|
||||
// ---- 4. Stream and type at the same time ---------------------------------
|
||||
// The schedule lives in Node: a stalling page must not be allowed to slow the
|
||||
// message rate down and measure less work than the other build.
|
||||
const startedAt = Date.now();
|
||||
const sentAt: number[] = [];
|
||||
const streaming = (async () => {
|
||||
for (let tick = 1; tick < fx.TICK_COUNT; tick += 1) {
|
||||
const due = startedAt + tick * fx.TICK_INTERVAL_MS;
|
||||
const wait = due - Date.now();
|
||||
if (wait > 0) await new Promise((done) => setTimeout(done, wait));
|
||||
for (let runIndex = 0; runIndex < fx.RUN_COUNT; runIndex += 1) push(runIndex, tick);
|
||||
sentAt.push(Date.now() - startedAt);
|
||||
}
|
||||
})();
|
||||
const typing = page.keyboard.type(fx.TYPING_TEXT, { delay: fx.KEY_DELAY_MS });
|
||||
// A build slow enough to blow the budget must still produce a report saying
|
||||
// so. Letting Playwright's own timeout kill the run would leave the reader
|
||||
// with no measurement at all, which is the one outcome a regression must not
|
||||
// be allowed to produce.
|
||||
streaming.catch(() => {});
|
||||
typing.catch(() => {});
|
||||
const remainingMs = () => Math.max(0, startedAt + SCENARIO_LIMIT_MS - Date.now());
|
||||
let timedOutAt: string | null = null;
|
||||
async function beforeDeadline<T>(work: Promise<T>, label: string): Promise<boolean> {
|
||||
const budget = remainingMs();
|
||||
if (budget <= 0) { timedOutAt ??= label; return false; }
|
||||
let timer: NodeJS.Timeout | undefined;
|
||||
const expired = new Promise<false>((done) => {
|
||||
timer = setTimeout(() => done(false), budget);
|
||||
});
|
||||
const finished = await Promise.race([work.then(() => true, () => false), expired]);
|
||||
clearTimeout(timer);
|
||||
if (!finished) timedOutAt ??= label;
|
||||
return finished;
|
||||
}
|
||||
|
||||
let typingElapsedMs: number | null = null;
|
||||
if (await beforeDeadline(typing, "keystrokes")) {
|
||||
// The wait is over only once the editor holds every character.
|
||||
const verified = await beforeDeadline(
|
||||
expect
|
||||
.poll(async () => (await editor.innerText()).trim(), { timeout: remainingMs() })
|
||||
.toBe(fx.TYPING_TEXT),
|
||||
"editor content",
|
||||
);
|
||||
if (verified) {
|
||||
await beforeDeadline(nextFrame(page), "paint");
|
||||
typingElapsedMs = Date.now() - startedAt;
|
||||
}
|
||||
}
|
||||
|
||||
let scenarioElapsedMs: number | null = null;
|
||||
if (!timedOutAt && await beforeDeadline(streaming, "stream")) {
|
||||
let settled = true;
|
||||
for (let runIndex = 0; runIndex < fx.RUN_COUNT && settled; runIndex += 1) {
|
||||
settled = await beforeDeadline(
|
||||
expect(summaries.nth(runIndex))
|
||||
.toHaveText(fx.endMarkerFor(runIndex), { timeout: remainingMs() }),
|
||||
`run ${runIndex} end marker`,
|
||||
);
|
||||
}
|
||||
if (settled) {
|
||||
await beforeDeadline(nextFrame(page), "final paint");
|
||||
scenarioElapsedMs = Date.now() - startedAt;
|
||||
}
|
||||
}
|
||||
const abortedAfterMs = timedOutAt ? Date.now() - startedAt : null;
|
||||
|
||||
// ---- 5. Collect --------------------------------------------------------
|
||||
// A jammed renderer cannot answer `evaluate` either, so collection gets its
|
||||
// own bounded budget and reports nothing rather than hanging.
|
||||
const COLLECT_BUDGET_MS = 20_000;
|
||||
const collect = async <T>(work: Promise<T>, fallback: T): Promise<T> => {
|
||||
let timer: NodeJS.Timeout | undefined;
|
||||
const expired = new Promise<T>((done) => {
|
||||
timer = setTimeout(() => done(fallback), COLLECT_BUDGET_MS);
|
||||
});
|
||||
const value = await Promise.race([work.catch(() => fallback), expired]);
|
||||
clearTimeout(timer);
|
||||
return value;
|
||||
};
|
||||
await collect(
|
||||
page.evaluate(() => (window as unknown as { __perfDrain: () => void }).__perfDrain()),
|
||||
undefined,
|
||||
);
|
||||
const longTasks = await collect(
|
||||
page.evaluate(
|
||||
() => (window as unknown as { __perfLongTasks: { duration: number }[] }).__perfLongTasks,
|
||||
),
|
||||
[] as { duration: number }[],
|
||||
);
|
||||
const after = await collect(readMetrics(), {} as Record<string, number>);
|
||||
const metricsRead = Object.keys(after).length > 0;
|
||||
const deltaMs = (name: string) =>
|
||||
metricsRead ? Math.round(((after[name] ?? 0) - (before[name] ?? 0)) * 1000) : null;
|
||||
const domNodes = metricsRead ? Math.round(after.Nodes ?? 0) : null;
|
||||
|
||||
const invalid: string[] = [];
|
||||
if (unknownRequests.length) invalid.push(`unmocked requests: ${unknownRequests.join(", ")}`);
|
||||
if (external.length) invalid.push(`external requests: ${external.join(", ")}`);
|
||||
if (pageErrors.length) invalid.push(`page errors: ${pageErrors.join(" | ")}`);
|
||||
if (!timedOutAt && sentAt.length !== fx.TICK_COUNT - 1) {
|
||||
invalid.push(`sent ${sentAt.length} ticks`);
|
||||
}
|
||||
if (!timedOutAt && !metricsRead) invalid.push("performance metrics unavailable");
|
||||
const status = timedOutAt ? "timeout" : invalid.length ? "invalid" : "ok";
|
||||
|
||||
const report = {
|
||||
status, invalid,
|
||||
timed_out_at: timedOutAt,
|
||||
aborted_after_ms: abortedAfterMs,
|
||||
fixture: fx.fixtureDigest(),
|
||||
typing_elapsed_ms: typingElapsedMs,
|
||||
scenario_elapsed_ms: scenarioElapsedMs,
|
||||
recalc_style_ms: deltaMs("RecalcStyleDuration"),
|
||||
layout_ms: deltaMs("LayoutDuration"),
|
||||
task_ms: deltaMs("TaskDuration"),
|
||||
long_task_count: longTasks.length,
|
||||
long_task_total_ms: Math.round(longTasks.reduce((sum, t) => sum + t.duration, 0)),
|
||||
long_task_max_ms: Math.round(Math.max(0, ...longTasks.map((t) => t.duration))),
|
||||
long_tasks_observed: longTasks.length > 0 || metricsRead,
|
||||
dom_nodes: domNodes,
|
||||
ticks_sent: sentAt.length + 1,
|
||||
stream_span_ms: sentAt.length ? sentAt[sentAt.length - 1]! : 0,
|
||||
typed_chars: fx.TYPING_TEXT.length,
|
||||
node_version: process.version,
|
||||
browser_version: page.context().browser()?.version() ?? "unknown",
|
||||
};
|
||||
|
||||
emit(report);
|
||||
|
||||
expect(invalid, invalid.join(" | ")).toEqual([]);
|
||||
expect(
|
||||
timedOutAt,
|
||||
`scenario exceeded ${SCENARIO_LIMIT_MS}ms while waiting for ${timedOutAt}`,
|
||||
).toBeNull();
|
||||
});
|
||||
@@ -3,6 +3,9 @@ import { defineConfig } from "@playwright/test";
|
||||
|
||||
export default defineConfig({
|
||||
testDir: "./e2e",
|
||||
// The performance scenario has its own config: one worker, no retries, and a
|
||||
// running production build. It must not be swept up by the ordinary suite.
|
||||
testIgnore: "**/perf/**",
|
||||
timeout: 60000,
|
||||
workers: 1,
|
||||
retries: 0,
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import { defineConfig } from "@playwright/test";
|
||||
|
||||
/**
|
||||
* The performance scenario runs on its own: one Chromium worker, one test, no
|
||||
* retries. A retry would hide the run that mattered, and a second worker would
|
||||
* make the two builds under comparison contend for the same machine.
|
||||
*
|
||||
* The default e2e config ignores this directory, so `pnpm exec playwright test`
|
||||
* never picks it up by accident.
|
||||
*/
|
||||
export default defineConfig({
|
||||
testDir: "./e2e/perf",
|
||||
timeout: 180_000,
|
||||
workers: 1,
|
||||
retries: 0,
|
||||
reporter: [["list"]],
|
||||
use: {
|
||||
baseURL: process.env.PLAYWRIGHT_BASE_URL ?? "http://localhost:3000",
|
||||
headless: true,
|
||||
},
|
||||
projects: [{ name: "chromium", use: { browserName: "chromium" } }],
|
||||
});
|
||||
Executable
+352
@@ -0,0 +1,352 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Run the comment-typing performance scenario against two refs and report the
|
||||
* difference (MUL-7227).
|
||||
*
|
||||
* Each ref is checked out, installed and built on its own, then measured once,
|
||||
* sequentially, on this machine — one build must never be measured while the
|
||||
* other is compiling. The spec, the fixture and the browser always come from
|
||||
* the working tree this script runs in, so the two products are the only thing
|
||||
* that differs; the base ref does not need to contain the test at all.
|
||||
*
|
||||
* One sample per ref is a report, not a verdict. A difference near the noise
|
||||
* floor means "run it again by hand", not "regression".
|
||||
*
|
||||
* node scripts/perf-compare.mjs --base <ref> --head <ref> [--out <dir>]
|
||||
*/
|
||||
import { execFileSync, spawn } from "node:child_process";
|
||||
import { mkdtempSync, mkdirSync, rmSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { connect, createServer } from "node:net";
|
||||
|
||||
const repoRoot = execFileSync("git", ["rev-parse", "--show-toplevel"]).toString().trim();
|
||||
const args = process.argv.slice(2);
|
||||
const flag = (name, fallback) => {
|
||||
const at = args.indexOf(`--${name}`);
|
||||
return at >= 0 && args[at + 1] ? args[at + 1] : fallback;
|
||||
};
|
||||
const baseRef = flag("base");
|
||||
const headRef = flag("head", "HEAD");
|
||||
const outDir = resolve(flag("out", join(repoRoot, "perf-report")));
|
||||
if (!baseRef) {
|
||||
console.error("usage: node scripts/perf-compare.mjs --base <ref> [--head <ref>] [--out <dir>]");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const run = (cmd, cmdArgs, opts = {}) =>
|
||||
execFileSync(cmd, cmdArgs, { stdio: "inherit", ...opts });
|
||||
|
||||
const delay = (ms) => new Promise((done) => setTimeout(done, ms));
|
||||
|
||||
/**
|
||||
* Servers and checkouts still alive. Each side stops its own before the next
|
||||
* one starts; these sets exist only for the last-resort handlers below.
|
||||
*/
|
||||
const liveServers = new Set();
|
||||
const liveCheckouts = new Set();
|
||||
|
||||
const killGroup = (child, signal) => {
|
||||
// Detached, so the server leads its own process group: signalling the group
|
||||
// reaches `next-server` as well as the pnpm wrapper that started it.
|
||||
try { process.kill(-child.pid, signal); } catch { /* already gone */ }
|
||||
};
|
||||
|
||||
function removeCheckout(checkout) {
|
||||
liveCheckouts.delete(checkout);
|
||||
try { run("git", ["worktree", "remove", "--force", checkout], { cwd: repoRoot, stdio: "ignore" }); }
|
||||
catch { rmSync(checkout, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
/** True while any process in the group is still alive. */
|
||||
const groupAlive = (pgid) => {
|
||||
try {
|
||||
process.kill(-pgid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
// ESRCH: no process left in the group. EPERM would mean one exists that
|
||||
// we may not signal — alive, as far as teardown is concerned.
|
||||
return error.code === "EPERM";
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Whether anything still holds the port. Only a refused connection counts as
|
||||
* closed: a server that has stopped answering still accepts the handshake, and
|
||||
* a timeout proves nothing either way.
|
||||
*/
|
||||
const portBound = (port) =>
|
||||
new Promise((done) => {
|
||||
const socket = connect({ port, host: "127.0.0.1" });
|
||||
const settle = (bound) => { socket.destroy(); done(bound); };
|
||||
socket.setTimeout(1_000, () => settle(true));
|
||||
socket.once("connect", () => settle(true));
|
||||
socket.once("error", (error) => settle(error.code !== "ECONNREFUSED"));
|
||||
});
|
||||
|
||||
async function groupExitWithin(pgid, ms) {
|
||||
const deadline = Date.now() + ms;
|
||||
while (groupAlive(pgid)) {
|
||||
if (Date.now() > deadline) return false;
|
||||
await delay(100);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop a server and wait until every process it started is gone.
|
||||
*
|
||||
* The pnpm wrapper exiting proves nothing about `next-server` beneath it, so
|
||||
* this waits on the whole process group, escalating to SIGKILL if the group
|
||||
* outlives SIGTERM. The server stays registered for the last-resort handlers
|
||||
* until the group is confirmed gone: dropping it any earlier is what let a
|
||||
* surviving child outlive the script. Then the port must refuse connections —
|
||||
* not merely stop answering.
|
||||
*
|
||||
* Teardown is explicit rather than hung off `process.on("exit")`: a live child
|
||||
* keeps the event loop running, so `exit` would never fire and the child would
|
||||
* never be killed — on CI, until the job was cancelled at sixty minutes.
|
||||
*/
|
||||
/**
|
||||
* How long a server gets to exit on SIGTERM before it is killed. Next exits in
|
||||
* well under a second; the margin is for a loaded machine. Overridable so the
|
||||
* lifecycle test can prove the SIGKILL path without waiting on it.
|
||||
*/
|
||||
const STOP_GRACE_MS = Number(process.env.PERF_STOP_GRACE_MS) || 10_000;
|
||||
|
||||
async function stopServer(child, port) {
|
||||
const pgid = child.pid;
|
||||
killGroup(child, "SIGTERM");
|
||||
if (!(await groupExitWithin(pgid, STOP_GRACE_MS))) {
|
||||
killGroup(child, "SIGKILL");
|
||||
if (!(await groupExitWithin(pgid, 5_000))) {
|
||||
throw new Error(`server process group ${pgid} survived SIGKILL`);
|
||||
}
|
||||
}
|
||||
liveServers.delete(child);
|
||||
|
||||
const deadline = Date.now() + 5_000;
|
||||
while (await portBound(port)) {
|
||||
if (Date.now() > deadline) {
|
||||
throw new Error(`port ${port} still bound after its server's process group exited`);
|
||||
}
|
||||
await delay(200);
|
||||
}
|
||||
}
|
||||
|
||||
// Last resort only, for a signal (a cancelled CI job) or an exit that happens
|
||||
// before a side finished tearing itself down. Both handlers run synchronously
|
||||
// and cannot await, which is exactly why the normal path does not use them.
|
||||
const emergencyStop = () => {
|
||||
for (const child of liveServers) killGroup(child, "SIGKILL");
|
||||
liveServers.clear();
|
||||
for (const checkout of [...liveCheckouts]) removeCheckout(checkout);
|
||||
};
|
||||
process.on("exit", emergencyStop);
|
||||
for (const signal of ["SIGINT", "SIGTERM"]) {
|
||||
process.on(signal, () => { emergencyStop(); process.exit(130); });
|
||||
}
|
||||
|
||||
const freePort = () =>
|
||||
new Promise((done, fail) => {
|
||||
const probe = createServer();
|
||||
probe.on("error", fail);
|
||||
probe.listen(0, "127.0.0.1", () => {
|
||||
const { port } = probe.address();
|
||||
probe.close(() => done(port));
|
||||
});
|
||||
});
|
||||
|
||||
const waitForServer = async (port, timeoutMs = 120_000) => {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/`);
|
||||
if (response.status > 0) return;
|
||||
} catch { /* not up yet */ }
|
||||
await new Promise((done) => setTimeout(done, 500));
|
||||
}
|
||||
throw new Error(`frontend on :${port} did not become reachable`);
|
||||
};
|
||||
|
||||
const seconds = (from) => Math.round((Date.now() - from) / 100) / 10;
|
||||
|
||||
async function measure(ref, label) {
|
||||
const sha = execFileSync("git", ["rev-parse", ref], { cwd: repoRoot }).toString().trim();
|
||||
const checkout = mkdtempSync(join(tmpdir(), `perf-${label}-`));
|
||||
liveCheckouts.add(checkout);
|
||||
let server;
|
||||
let port;
|
||||
// Everything this side started is torn down before it returns or throws, so
|
||||
// the next side is measured on a machine with nothing of this one running.
|
||||
try {
|
||||
console.log(`\n=== ${label}: ${ref} (${sha.slice(0, 9)}) ===`);
|
||||
run("git", ["worktree", "add", "--detach", checkout, sha], { cwd: repoRoot });
|
||||
|
||||
// Each product installs against its own lockfile: a build measured with the
|
||||
// other ref's dependency tree is not that ref.
|
||||
const installStart = Date.now();
|
||||
run("pnpm", ["install", "--frozen-lockfile"], { cwd: checkout });
|
||||
const installS = seconds(installStart);
|
||||
|
||||
// REMOTE_API_URL is a runtime setting. Passing it to the build breaks
|
||||
// prerendering, and turbo filters it out of the build env anyway.
|
||||
//
|
||||
// The cache is left on. A restored build is byte-identical to the one that
|
||||
// produced it, so it cannot move the numbers this script collects — only the
|
||||
// build time it reports. Saying which side was cached costs nothing; forcing
|
||||
// two real builds to avoid the ambiguity costs minutes on every local run,
|
||||
// and buys nothing on CI, where the runner is always cold anyway.
|
||||
const buildStart = Date.now();
|
||||
const buildLog = execFileSync(
|
||||
"pnpm",
|
||||
["exec", "turbo", "build", "--filter=@multica/web"],
|
||||
{ cwd: checkout, encoding: "utf8", stdio: ["ignore", "pipe", "inherit"] },
|
||||
);
|
||||
process.stdout.write(buildLog);
|
||||
const buildS = seconds(buildStart);
|
||||
const buildCached = /cache hit/.test(buildLog);
|
||||
|
||||
port = await freePort();
|
||||
const startStart = Date.now();
|
||||
server = spawn("pnpm", ["--filter", "@multica/web", "start"], {
|
||||
cwd: checkout,
|
||||
env: { ...process.env, PORT: String(port), REMOTE_API_URL: "http://127.0.0.1:1" },
|
||||
stdio: "ignore",
|
||||
detached: true,
|
||||
});
|
||||
liveServers.add(server);
|
||||
await waitForServer(port);
|
||||
const startS = seconds(startStart);
|
||||
|
||||
// The spec, fixture and browser come from this working tree, not the ref's.
|
||||
const reportPath = join(outDir, `${label}.json`);
|
||||
// Only a report this run wrote may be read back. A file left by an earlier
|
||||
// run into the same directory would otherwise stand in for a scenario that
|
||||
// failed before writing one — stale numbers, presented as today's.
|
||||
rmSync(reportPath, { force: true });
|
||||
const measureStart = Date.now();
|
||||
let scenarioExit = 0;
|
||||
try {
|
||||
run("pnpm", ["exec", "playwright", "test", "--config=playwright.perf.config.ts"], {
|
||||
cwd: repoRoot,
|
||||
env: {
|
||||
...process.env,
|
||||
PLAYWRIGHT_BASE_URL: `http://127.0.0.1:${port}`,
|
||||
PERF_REPORT_PATH: reportPath,
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
scenarioExit = typeof error.status === "number" ? error.status : 1;
|
||||
}
|
||||
const measureS = seconds(measureStart);
|
||||
|
||||
let report;
|
||||
try {
|
||||
report = JSON.parse(readFileSync(reportPath, "utf8"));
|
||||
} catch {
|
||||
report = { status: "invalid", invalid: ["the scenario produced no readable report"] };
|
||||
}
|
||||
// The scenario process has the last word. A report that says `ok` from a
|
||||
// run whose process then failed is not a sample: whatever failed after the
|
||||
// numbers were written is exactly what nobody has looked at.
|
||||
if (scenarioExit !== 0 && report.status === "ok") {
|
||||
report.status = "invalid";
|
||||
report.invalid = [
|
||||
...(report.invalid ?? []),
|
||||
`the scenario reported ok but its process exited ${scenarioExit}`,
|
||||
];
|
||||
}
|
||||
const failed = scenarioExit !== 0;
|
||||
return {
|
||||
...report, ref, sha, spec_failed: failed, build_cached: buildCached,
|
||||
timings_s: { install: installS, build: buildS, start: startS, measure: measureS },
|
||||
};
|
||||
} finally {
|
||||
if (server) await stopServer(server, port);
|
||||
removeCheckout(checkout);
|
||||
}
|
||||
}
|
||||
|
||||
const METRICS = [
|
||||
["typing_elapsed_ms", "Typing elapsed (primary)"],
|
||||
["scenario_elapsed_ms", "Scenario elapsed"],
|
||||
["recalc_style_ms", "Style recalculation"],
|
||||
["layout_ms", "Layout"],
|
||||
["task_ms", "Main-thread task time"],
|
||||
["long_task_count", "Long tasks"],
|
||||
["long_task_total_ms", "Long task total"],
|
||||
["long_task_max_ms", "Long task max"],
|
||||
];
|
||||
|
||||
function markdown(base, head) {
|
||||
const usable = base.status === "ok" && head.status === "ok";
|
||||
const rows = METRICS.map(([key, label]) => {
|
||||
const a = base[key];
|
||||
const b = head[key];
|
||||
if (typeof a !== "number" || typeof b !== "number") return `| ${label} | ${a ?? "—"} | ${b ?? "—"} | — | — |`;
|
||||
const delta = b - a;
|
||||
const ratio = a === 0 ? "N/A" : `${((b / a - 1) * 100).toFixed(1)}%`;
|
||||
return `| ${label} | ${a} | ${b} | ${delta >= 0 ? "+" : ""}${delta} | ${ratio} |`;
|
||||
});
|
||||
const timing = (r) =>
|
||||
`install ${r.timings_s.install}s · build ${r.timings_s.build}s${r.build_cached ? " (restored from cache — not a real build)" : ""} · start ${r.timings_s.start}s · measure ${r.timings_s.measure}s`;
|
||||
return [
|
||||
"## Comment typing under live runs (MUL-7227)",
|
||||
"",
|
||||
usable
|
||||
? "One sample per ref. This is a report, not a merge gate — read a small difference as noise until a second run says otherwise."
|
||||
: `**Not a usable comparison.** base: \`${base.status}\`, head: \`${head.status}\`.`,
|
||||
"",
|
||||
"| Metric | base | head | Δ | ratio |",
|
||||
"| --- | ---: | ---: | ---: | ---: |",
|
||||
...rows,
|
||||
"",
|
||||
`- base \`${base.ref}\` (${base.sha.slice(0, 9)}) — ${base.status}${base.invalid?.length ? `: ${base.invalid.join("; ")}` : ""}`,
|
||||
`- head \`${head.ref}\` (${head.sha.slice(0, 9)}) — ${head.status}${head.invalid?.length ? `: ${head.invalid.join("; ")}` : ""}`,
|
||||
`- fixture \`${head.fixture?.version}\` sha256 \`${head.fixture?.sha256}\` (${head.fixture?.bytes} bytes); same digest on base: ${base.fixture?.sha256 === head.fixture?.sha256}`,
|
||||
`- DOM nodes: base ${base.dom_nodes ?? "—"}, head ${head.dom_nodes ?? "—"}; ticks sent: base ${base.ticks_sent ?? "—"}, head ${head.ticks_sent ?? "—"}`,
|
||||
`- node ${head.node_version ?? "—"}, chromium ${head.browser_version ?? "—"}`,
|
||||
`- base timings: ${timing(base)}`,
|
||||
`- head timings: ${timing(head)}`,
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
mkdirSync(outDir, { recursive: true });
|
||||
// The files this script writes, cleared up front: a comparison that aborts
|
||||
// early must not leave the previous run's summary looking like its own.
|
||||
for (const name of ["base.json", "head.json", "comparison.json", "comparison.md"]) {
|
||||
rmSync(join(outDir, name), { force: true });
|
||||
}
|
||||
const totalStart = Date.now();
|
||||
let exitCode = 1;
|
||||
let base;
|
||||
let head;
|
||||
try {
|
||||
base = await measure(baseRef, "base");
|
||||
head = await measure(headRef, "head");
|
||||
const totalS = seconds(totalStart);
|
||||
|
||||
const summary = markdown(base, head);
|
||||
writeFileSync(join(outDir, "comparison.json"), JSON.stringify({ base, head, total_s: totalS }, null, 2));
|
||||
writeFileSync(join(outDir, "comparison.md"), `${summary}\n\n- total wall clock: ${totalS}s\n`);
|
||||
console.log(`\n${summary}\n\n- total wall clock: ${totalS}s`);
|
||||
console.log(`\nreports written to ${outDir}`);
|
||||
|
||||
// The comparison itself only fails when a sample is not usable. A slower
|
||||
// head is information for the reviewer, not a failure of this script.
|
||||
exitCode = base.status === "ok" && head.status === "ok" ? 0 : 1;
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
console.error(`\ncomparison aborted: ${message}`);
|
||||
// Keep whatever was measured: the artifact is uploaded on failure too.
|
||||
writeFileSync(
|
||||
join(outDir, "comparison.json"),
|
||||
JSON.stringify({ error: message, base: base ?? null, head: head ?? null }, null, 2),
|
||||
);
|
||||
}
|
||||
|
||||
// Explicit, whatever happened above. Each side has already stopped its server
|
||||
// and removed its checkout; exiting here makes sure nothing left over — a
|
||||
// keep-alive socket, a stray timer — can hold the process open again.
|
||||
process.exit(exitCode);
|
||||
Executable
+162
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env bash
|
||||
# Proves perf-compare.mjs reports honestly and leaves nothing running.
|
||||
#
|
||||
# The comparison is only worth running if its verdict can be trusted and it
|
||||
# cleans up after itself, and both have failed before: a report left by an
|
||||
# earlier run was read back as today's result, a scenario that reported `ok`
|
||||
# and then failed still exited 0, a server child that stopped answering was
|
||||
# taken for a closed port and outlived the script, and teardown once waited
|
||||
# forever on the server it was meant to stop.
|
||||
#
|
||||
# This drives the real script with a fake `pnpm` on PATH — no build, no
|
||||
# browser, a few seconds — through each of those cases.
|
||||
set -euo pipefail
|
||||
|
||||
root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
compare="$root_dir/scripts/perf-compare.mjs"
|
||||
work="$(mktemp -d)"
|
||||
repo="$work/repo"
|
||||
bin="$work/bin"
|
||||
server_log="$work/servers.jsonl"
|
||||
mkdir -p "$repo" "$bin" "$work/tmp"
|
||||
touch "$server_log"
|
||||
|
||||
# Only ever signal processes this test's own fake servers recorded.
|
||||
kill_recorded() {
|
||||
node -e '
|
||||
const fs = require("fs");
|
||||
for (const line of fs.readFileSync(process.argv[1], "utf8").split("\n")) {
|
||||
if (!line) continue;
|
||||
try { process.kill(JSON.parse(line).pid, "SIGKILL"); } catch {}
|
||||
}' "$server_log" || true
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
kill_recorded
|
||||
rm -rf "$work"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
git -C "$repo" init -q
|
||||
echo fixture >"$repo/fixture.txt"
|
||||
git -C "$repo" add fixture.txt
|
||||
git -C "$repo" -c user.name=test -c user.email=[email protected] commit -qm fixture
|
||||
|
||||
# Stands in for pnpm: install and build succeed, `start` runs a tiny server,
|
||||
# and `playwright` behaves as PERF_PLAYWRIGHT says.
|
||||
cat >"$bin/pnpm" <<'NODEEOF'
|
||||
#!/usr/bin/env node
|
||||
const { createServer } = require("node:http");
|
||||
const { spawn } = require("node:child_process");
|
||||
const { appendFileSync, writeFileSync } = require("node:fs");
|
||||
const args = process.argv.slice(2);
|
||||
const record = (pid) =>
|
||||
appendFileSync(process.env.PERF_TEST_SERVER_LOG, `${JSON.stringify({ pid })}\n`);
|
||||
|
||||
if (args[0] === "install") process.exit(0);
|
||||
if (args[0] === "exec" && args[1] === "turbo") { console.log("fake build"); process.exit(0); }
|
||||
|
||||
if (args[0] === "--filter" && args[2] === "start") {
|
||||
const port = Number(process.env.PORT);
|
||||
if (process.env.PERF_SERVER === "stubborn") {
|
||||
// A child in the wrapper's process group that ignores SIGTERM and stops
|
||||
// answering while keeping the port — the wrapper itself exits promptly.
|
||||
const child = spawn(process.execPath, ["-e", `
|
||||
const { createServer } = require("node:http");
|
||||
let stopping = false;
|
||||
process.on("SIGTERM", () => { stopping = true; });
|
||||
createServer((req, res) => { if (!stopping) res.end("ok"); })
|
||||
.listen(${port}, "127.0.0.1");
|
||||
`], { stdio: "ignore" });
|
||||
record(child.pid);
|
||||
process.on("SIGTERM", () => process.exit(0));
|
||||
setInterval(() => {}, 1 << 30);
|
||||
} else {
|
||||
record(process.pid);
|
||||
createServer((req, res) => res.end("ok")).listen(port, "127.0.0.1");
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (args[0] === "exec" && args[1] === "playwright") {
|
||||
const report = {
|
||||
status: "ok", invalid: [], typing_elapsed_ms: 123,
|
||||
fixture: { version: "test", sha256: "fresh", bytes: 0 },
|
||||
};
|
||||
const mode = process.env.PERF_PLAYWRIGHT;
|
||||
if (mode !== "fail-no-report") writeFileSync(process.env.PERF_REPORT_PATH, JSON.stringify(report));
|
||||
process.exit(mode === "ok" ? 0 : 1);
|
||||
}
|
||||
process.exit(2);
|
||||
NODEEOF
|
||||
chmod +x "$bin/pnpm"
|
||||
|
||||
fail() { echo "FAIL [$case]: $*" >&2; cat "$work/out.log" >&2 || true; exit 1; }
|
||||
|
||||
# run_case <name> <playwright mode> <server mode> [base ref]
|
||||
run_case() {
|
||||
case="$1"
|
||||
local out="$work/out-$1"
|
||||
mkdir -p "$out"
|
||||
if [ -n "${SEED_STALE:-}" ]; then
|
||||
for side in base head; do
|
||||
echo '{"status":"ok","invalid":[],"typing_elapsed_ms":999,"fixture":{"sha256":"stale"}}' >"$out/$side.json"
|
||||
done
|
||||
echo "stale summary" >"$out/comparison.md"
|
||||
fi
|
||||
: >"$server_log"
|
||||
set +e
|
||||
(cd "$repo" && PATH="$bin:$PATH" TMPDIR="$work/tmp" PERF_STOP_GRACE_MS=300 \
|
||||
PERF_PLAYWRIGHT="$2" PERF_SERVER="$3" PERF_TEST_SERVER_LOG="$server_log" \
|
||||
node "$compare" --base "${4:-HEAD}" --head HEAD --out "$out") >"$work/out.log" 2>&1
|
||||
code=$?
|
||||
set -e
|
||||
comparison="$out/comparison.json"
|
||||
[ -f "$comparison" ] || fail "no comparison.json written"
|
||||
live="$(node -e '
|
||||
const fs = require("fs");
|
||||
let n = 0;
|
||||
for (const line of fs.readFileSync(process.argv[1], "utf8").split("\n")) {
|
||||
if (!line) continue;
|
||||
try { process.kill(JSON.parse(line).pid, 0); n++; } catch {}
|
||||
}
|
||||
console.log(n);' "$server_log")"
|
||||
[ "$live" = 0 ] || fail "$live server process(es) still running"
|
||||
[ -z "$(ls -A "$work/tmp")" ] || fail "temporary checkouts left behind"
|
||||
[ "$(git -C "$repo" worktree list | wc -l | tr -d ' ')" = 1 ] || fail "worktrees left behind"
|
||||
}
|
||||
|
||||
field() { node -e 'const r = require(process.argv[1]); console.log(eval("r." + process.argv[2]) ?? "")' "$comparison" "$1"; }
|
||||
expect_exit() { [ "$code" = "$1" ] || fail "expected exit $1, got $code"; }
|
||||
expect_status() { for side in base head; do [ "$(field "$side.status")" = "$1" ] || fail "$side status: expected $1, got $(field "$side.status")"; done; }
|
||||
|
||||
run_case success ok normal
|
||||
expect_exit 0
|
||||
expect_status ok
|
||||
|
||||
run_case failure-without-report fail-no-report normal
|
||||
expect_exit 1
|
||||
expect_status invalid
|
||||
|
||||
SEED_STALE=1 run_case stale-report fail-no-report normal
|
||||
expect_exit 1
|
||||
expect_status invalid
|
||||
[ "$(field base.typing_elapsed_ms)" != 999 ] || fail "the previous run's report was read as this run's"
|
||||
[ ! -f "$work/out-stale-report/comparison.md" ] || grep -vq "stale summary" "$work/out-stale-report/comparison.md" \
|
||||
|| fail "the previous run's summary survived"
|
||||
|
||||
run_case failure-after-ok-report fail-after-report normal
|
||||
expect_exit 1
|
||||
expect_status invalid
|
||||
grep -q "process exited" "$comparison" || fail "the invalid verdict does not say the process failed"
|
||||
|
||||
run_case server-ignores-sigterm ok stubborn
|
||||
expect_exit 0
|
||||
expect_status ok
|
||||
|
||||
SEED_STALE=1 run_case unknown-ref ok normal does-not-exist
|
||||
expect_exit 1
|
||||
[ ! -f "$work/out-unknown-ref/comparison.md" ] || fail "an aborted run left the previous summary in place"
|
||||
grep -q "comparison aborted\|error" "$comparison" || fail "the aborted run does not record why"
|
||||
|
||||
echo "perf-compare: all lifecycle and reporting cases passed"
|
||||
Reference in New Issue
Block a user