Dashboard E2E Flake (#578)

On this page

Overview

tests/e2e/specs/dashboard.spec.ts and the other dashboard-surface specs (supervisor / analyst / customize) flake intermittently under cargo xtask validate (pre-push): page.goto('/') times out waiting for the browser load event (~30–47s), then renders sub-second on retry.

NOTE
An earlier diagnosis (2026-05-26) attributed this to a synchronous audit-emit COMMIT stalling on a Postgres checkpoint WAL fsync. That was corrected by an external review on 2026-06-06 — see the full investigation in GitLab issue #578. The host is fast NVMe under near-zero load, so it is not an I/O / fsync problem. Two distinct causes are now understood; this page reflects the corrected understanding.

Primary cause — wrong Playwright wait-condition (a test bug)

A Jun-6 failure artifact showed page.goto('/') timing out waiting for the load event while the captured page snapshot already showed the fully-rendered dashboard. Bare page.goto(url) defaults to waitUntil: 'load'; every portal page opens a long-lived SSE EventSource (base.html) plus deferred subresources, so the load event can hang for tens of seconds after the DOM is rendered and interactive. The server is not stalling — the test is waiting on the wrong lifecycle event.

Fix (landed)

tests/e2e/lib/helpers.ts::gotoPortal(page, url) navigates with waitUntil: 'domcontentloaded' (HTML parsed) instead of load. Callers still assert on concrete panel locators, so a genuinely stalled render still fails — this does not blind the suite to a real server-side regression. Applied to the four dashboard surface specs, and (#1307, 2026-08-03) to accessibility.spec.ts’s worker targets (`visit(), caseDetail(), worker-application-process), which were the last worker specs still on bare page.goto/networkidle. Use gotoPortal (not bare page.goto) for any new WORKER portal-page navigation that opens the SSE stream. #1410 migrated every live bare page.goto in the suite (specs + shared helpers; each spec file’s FIRST navigation carries the contention-sized FIRST_LOAD_NAV_TIMEOUT_MS budget — first-load statics were observed at 4.8–17 s under a disk-I/O storm). NOTE: the public applicant portal is Dioxus/WASM, where domcontentloaded can precede hydration — those audits keep their hydration-aware wait (the PUBLIC_PORTAL_PAGES set is intentionally NOT on gotoPortal). Since #1284 the portal stamps data-hydrated="1" on <html> post-hydration and awaitHydrated(page) (lib/portal.ts) gates every first wasm-handled interaction on it — the click-retry loops it replaced are gone; new portal specs use the helper, never a retry-until-heard loop.

Server-side SSR stall — un-budgeted upstream fan-out (#1306, FIXED 2026-08-05)

A 2026-08-03 multi-agent root-cause (over a battery where worker /, /cases, and /cases/{id} all timed out) identified the real server-side cause: the canopy-web SSR page handlers block the HTTP response on un-budgeted upstream calls. The internal client only enforces its retry OVERALL deadline when a per-call budget is set; the dashboard panels are budgeted, but fetch_hero, /cases my_queue, and the case-detail serial GETs + section fan-out are not — so a single browned-out upstream holds the whole document ~15s (3×5s retries). This defeats even domcontentloaded (stylesheets block the deferred scripts that gate it). The redesign — an aggregate request deadline that preserves partials, distinct timeout/partial/error states, and telemetry — LANDED via #1306 (closed 2026-08-05). The section below stays as the telltale signature for any future regression of the same shape.

NOTE
the earlier pool-contention hypothesis (a 10-connection convoy) is superseded — the runtime pool is 10 and was NOT the bottleneck; the un-budgeted fan-out is. (crates/canopy-composition/src/loader.rs::spawn_render_audit correctly moved the render-telemetry COMMIT off the response future.)

Telltale signature

Distinguish the two causes by whether the DOM actually rendered:

  • Wrong wait-condition (#1307/#578, fixed): page.goto: Timeout … waiting until "load" (or networkidle) while the captured artifact shows the page DOM already rendered. /livez returns 200 in <5ms; the diff under test does not touch the render path. → the test waited on the wrong lifecycle event.

  • Server-side SSR stall (#1306, fixed — historical signature): page.goto times out even on domcontentloaded and the captured artifact shows a missing/partial DOM (e.g. only a few of ~20 panels); canopy-web logs show a truncated upstream fan-out with no "handler complete" for the window. → an upstream browned out and the un-budgeted SSR call held the document. Not a test bug.

Related — composition_documents cross-run leak (a DISTINCT, real bug — fixed)

A separate mechanism produced genuinely-wrong dashboard panel counts (e.g. worker dashboard "Expected 12, Received 9"): dashboard-customize.spec.ts wrote user_delta_v1 rows but cleaned up only in beforeEach; combined with composition_documents never being truncated between e2e runs, a leftover row leaked into the next run’s dashboard.spec.ts. Fixed by an afterEach wipe plus an unconditional composition_documents TRUNCATE on every e2e run (xtask::cmd::seed::reset_composition_documents). Distinguish from the wait-condition flake by the failure: a toHaveCount mismatch with a fully-rendered-but-subset DOM, vs. a page.goto: Timeout … waiting until "load".

Related

  • Stale JWKS recovery — a different flake with an overlapping empty-dashboard symptom; distinguished by the multi-service 401 cluster in the logs.

Edit this page · default