Plan: Test-Seed Harness Refactor (Issue #450)

On this page

Status

Step Description Status

1

Add --seed-output flag to canopy-seed binary. Default seed is now rand::random::<u64>(); resolved seed is written to the path passed via --seed-output (typically test-results/seed/last.txt) AND stamped at the top of the generated seed.ts manifest as a top-of-file /** Generated by canopy-seed with seed=N */ comment + an exported SEED: number constant. Echo Seed: N (replay with --seed N). on completion.

Done (2026-05-13)

2

Single source-of-truth seed call. xtask seed keeps its existing CLI shape (back-compat) but writes last.txt via the new flag. xtask e2e delegates to xtask::cmd::seed::run(…​) every run — devstack container resets between xtask seed and xtask e2e runs would otherwise leave the DB empty while last.txt still claimed a matching state; re-seeding is fast enough to be unconditional. Both commands share DEFAULT_HOUSEHOLDS = 50. Env-supplied CANOPY_SEED / CANOPY_HOUSEHOLDS override the defaults; otherwise the captured-seed values are reused so two consecutive xtask e2e runs stay byte-stable. Deviation from original plan: the initial design had xtask e2e skip re-seeding when last.txt matched env, but exploratory testing surfaced the container-reset hole (captured-seed receipt outliving the DB it referenced). Always-reseed is the correct invariant.

Done (2026-05-13)

3

Predicate-based Playwright fixture helpers. New tests/e2e/lib/fixtures.ts exposes pickFirstApprovedDetermination(program: "snap"|"tanf"|…​), pickAssessmentFor(personId), pickAuthorizationFor(determinationId), etc. — all backed by live API queries (caseworker-authenticated, hits the running devstack). Replace UUID-positional access in specs/*.spec.ts (e.g. seed.wicDeterminations.wicDet0.personId) with predicate calls. The auto-generated seed.ts becomes a thin metadata file (seed value, household count, named fixture availability map) rather than a positional UUID index. Manifest/DB drift is architecturally impossible because the test never references manifest-internal positional structure.

Done (2026-05-13)

4

Documentation. Update .claude/docs/testing.md with the --seed replay flow, the new predicate-fixture API, and the last.txt debugging recipe. Update plans archive note pointing at the symptom — this is the architectural fix to the 2026-05-12 flake.

Done (2026-05-13)

5

Verification. cargo xtask seed && cargo xtask e2e (in either order) produces identical DB state and passes 128/128 e2e specs. Running with --seed flag reproduces a known failure mode byte-for-byte. Random-by-default verified by running e2e twice without --seed and confirming the seeds differ.

Done (2026-05-13)

Issue: #450
Branch: feat/test-seed-harness-refactor
Labels: type::chore, priority::medium, service::web, service::xtask, service::seed, workflow::ready

Context

The 2026-05-12 push cycle for #392 (worker-portal program-action handlers) and the following #448 batches surfaced a recurring flake: Playwright assertions referencing seed.<table>.<row0>.<id> fields against seed.ts would silently match a manifest UUID that no longer existed in the database, producing assert!(panel).toContainText('provider-001')-style failures that looked like real product bugs.

The root cause was a two-entry-point seed pipeline:

  • cargo xtask seed --households 50 --seed 42 regenerated both the SQL files AND the tests/e2e/lib/seed.ts manifest, then loaded the SQL into the running devstack DBs. This was the manual debugging path.

  • cargo xtask e2e ALSO invoked the canopy-seed binary internally with its own defaults (--households 9, no seed) BEFORE running Playwright — regenerating the manifest but NOT reloading the DB. The manifest UUIDs would then point at rows that didn’t exist in the DB (because xtask seed --households 50 had loaded 50 households' worth of rows under a different UUID series).

Workaround during the session was cargo xtask seed --households 50 --seed 42 && cargo xtask e2e --no-refresh. Documented in the feedback_seed_args_match memory.

This plan replaces the dual-entry-point with a single source of truth, captures the resolved seed at every generation, and migrates Playwright fixtures off manifest-positional UUID access so the failure class is architecturally impossible.

Code references

  • xtask/src/cmd/seed.rs:70 — current xtask seed entry point; spawns cargo run -p canopy-seed — --households {N} --seed? {N} --manifest tests/e2e/lib/seed.ts.

  • xtask/src/cmd/e2e.rs:80-110 — current xtask e2e entry point; also spawns canopy-seed with its own defaults (--households 9 historically).

  • tools/canopy-seed/src/main.rs:60-72 — --manifest flag handling; no seed-capture-to-file today.

  • tools/canopy-seed/src/manifest.rs — manifest renderer.

  • tests/e2e/lib/seed.ts — auto-generated positional UUID index; consumed by specs via seed.<table>.<row>.<field>.

  • tests/e2e/specs/caps.spec.ts:14 — example positional access; this assertion was rewritten during #396 (caps-provider-registry) because the literal "provider-001" UUID-equivalent no longer existed after the FK retype.

  • ADR-016 — no schema relevance, but the test-seed harness is the development analog of the production forward-only discipline (the seed pipeline must produce the same DB state given the same inputs).

Scope

In scope:

  • All three layers from the issue body (Layer 1 single source-of-truth, Layer 2 random-by-default + replay capture, Layer 3 predicate-based Playwright fixtures).

  • canopy-seed binary: new --seed-output flag, random default, seed stamping in the manifest.

  • xtask: seed.rs writes last.txt; e2e.rs reads last.txt and only regenerates when necessary.

  • Playwright fixtures: new tests/e2e/lib/fixtures.ts predicate helpers; migrate every specs/*.spec.ts reference to manifest-positional UUIDs.

  • .claude/docs/testing.md update.

Out of scope:

  • Database-side "snapshot/restore" for fast test reset (would require pg_dump/pg_restore round-trips per test class — a separate plan).

  • Cross-platform seed reproducibility (only Linux/Docker; macOS isn’t a target).

Dependencies

  • No prerequisite plans on disk.

  • The Playwright auth/caseworker.json storageState must work for the predicate-query helpers (it already does — every existing spec uses it).

  • Devstack must be up + healthy before xtask seed / xtask e2e (unchanged precondition).

Design

Layer 1: single source-of-truth seed call (Step 2)

xtask e2e stops invoking canopy-seed directly. Instead:

  1. Reads test-results/seed/last.txt for the last-resolved seed + household count.

  2. If --seed / --households flags are supplied AND differ from last.txt, calls xtask::cmd::seed::run(…​) programmatically.

  3. Runs Playwright.

xtask seed keeps its existing CLI but always writes last.txt. The --households default is reconciled from the historical mismatch (9 in xtask seed, 50 in xtask e2e) onto a single shared constant. Pick 50 — the existing seed manifest already uses it for the SNAP UAT data and the per-program lifecycle rows from #398.

Layer 2: random seed + replay capture (Step 1)

canopy-seed gets a new --seed-output PATH flag. When set:

  1. Resolves the seed: args.seed.unwrap_or_else(|| rand::random::<u64>()).

  2. Writes <seed>\n<households>\n<jurisdiction>\n to PATH atomically (write to PATH.tmp, then rename).

  3. Stamps the manifest’s preamble: /** Generated by canopy-seed with seed=N households=M jurisdiction=X */ + export const SEED: number = N;.

  4. Echoes Seed: N (replay with --seed N --households M). to stderr.

xtask seed passes --seed-output test-results/seed/last.txt by default; xtask e2e reads from that path.

Layer 3: predicate-based fixtures (Step 3)

New tests/e2e/lib/fixtures.ts:

import type { Page } from '@playwright/test';

export interface ApprovedDetermination {
  id: string;
  householdId: string;
  personId: string;
  program: string;
}

/**
 * Find an approved determination for the named program via a live
 * `GET /v1/determinations?household_id=<any>` against the program's
 * service. Returns null when no approved rows exist in the seed (caller
 * uses `test.skip(!result, ...)` to skip cleanly).
 */
export async function pickFirstApprovedDetermination(
  page: Page,
  program: 'snap' | 'tanf' | 'medicaid' | 'caps' | 'wic',
): Promise<ApprovedDetermination | null> { /* ... */ }

export async function pickAssessmentFor(page: Page, personId: string): Promise<{ id: string } | null> { /* ... */ }
export async function pickAuthorizationFor(page: Page, determinationId: string): Promise<{ id: string } | null> { /* ... */ }

Specs migrate from:

import { SEED } from '../lib/seed';
const wicDet = SEED.wicDeterminations.wicDet0;
await page.goto(`/cases/${wicDet.householdId}?program=wic`);

to:

import { pickFirstApprovedDetermination } from '../lib/fixtures';
const wic = await pickFirstApprovedDetermination(page, 'wic');
if (!wic) {
  test.skip(true, 'no approved WIC determination in seed');
}
await page.goto(`/cases/${wic.householdId}?program=wic`);

The auto-generated seed.ts becomes a thin metadata file (seed value, household count, named fixture availability map). Pre-existing specs that reference manifest-internal positional structure (SEED.wicDeterminations.wicDet0) get rewritten.

Files Touched

File Change

tools/canopy-seed/src/main.rs

Add --seed-output PATH flag; randomise seed by default; atomic-write the seed-output file; stamp the manifest preamble + SEED constant.

tools/canopy-seed/src/manifest.rs

Manifest preamble updated to emit export const SEED: number = N; alongside the existing positional UUID index. Backwards-compatible — existing consumers ignore the new constant.

xtask/src/cmd/seed.rs

Pass --seed-output test-results/seed/last.txt; emit a Seed: N (replay with --seed N). notice on completion.

xtask/src/cmd/e2e.rs

Stop invoking canopy-seed directly. Read last.txt; only re-seed when --seed / --households flags differ from the captured values.

tests/e2e/lib/fixtures.ts (new)

Predicate-query helpers backed by live API calls.

tests/e2e/specs/*.spec.ts

Migrate every positional UUID access (seed.wicDeterminations.wicDet0) to await pickFirstApprovedDetermination(…​) etc.

tests/e2e/lib/seed.ts

Becomes a thin metadata file — seed value + household count + named-fixture availability map.

.claude/docs/testing.md

New section on the --seed replay flow + predicate-fixture API + last.txt debugging recipe.

CHANGELOG.adoc

Single entry covering all three layers.

Verification

  1. cargo xtask seed --households 50 --seed 42 && cat test-results/seed/last.txt — produces 42\n50\ngeorgia\n (replay receipt).

  2. cargo xtask e2e (no flags) — uses `last.txt’s captured seed; passes 128/128 specs.

  3. cargo xtask e2e --seed 12345 — different seed; reseeds the DB; passes 128/128 specs (or surfaces real failures if any exist for that seed — Layer 3 makes specs predicate-driven so positional drift can’t cause flakes).

  4. Running cargo xtask seed twice without --seed produces two different seeds in last.txt (random by default).

  5. cargo xtask validate green.

  6. Manual: cat tests/e2e/lib/seed.ts | head -3 shows the seed stamp.

Documentation Updates

  • .claude/docs/testing.md — --seed replay flow + predicate-fixture API + last.txt debugging recipe.

  • CHANGELOG.adoc — Tier A bundle entry covering all three layers.

  • Update the feedback_seed_args_match memory — note that the workaround is no longer needed post-#450.

  • Plan archive: move to plans/archive/ post-merge.

Edit this page · default