Testing (Canopy)

Every lane selects on ONE classification: a lib/bin test that needs live infrastructure (EphemeralSchema, infrastructure_available(), PG/AMQP/devstack HTTP) lives in a module whose path contains infra_tests; everything else is pure. The filterset constants live in xtask/src/lanes.rs; the blocking cargo xtask test-lanes-lint enforces the classification statically (a syn walk over lib/bin sources) and --verify-partition set-verifies the split against cargo nextest list in every validate battery (unit ⊎ integration == the full workspace list, disjoint — plus every serialized-group filterset still matching, so a test-fn rename can never silently unserialize a group).

The MR-CI gap is explicit and deliberate: no merge-request CI job runs the infra set against live services (repo policy — the pre-push validate battery is the sole functional-correctness gate; CI carries security/supply-chain/ release plus the static lane checks). Before #1377 the same tests silently SKIPPED in MR CI while the --workspace container duplicated every unit test on main; now the boundary is visible instead of vacuous.

A bare, unfiltered cargo xtask e2e — exactly what the pre-push hook runs — IS the blocking E2E battery, and it carries two standing devstack requirements: the full compose profile (every program service; the default profile since #1386) and the test-clock build (CANOPY_CARGO_FEATURES=canopy-api/test-clock cargo xtask dev reload, sticky). The multi-life-event journey project is a blocking battery lane; it runs strictly LAST via Playwright project dependencies, because its specs advance/reset the fleet’s logical clocks and would contaminate concurrently-running date-sensitive specs.

The lane is fail-loud: after a green Playwright exit the battery verifies that test-results/e2e/results.xml was written by this run, that the journey project executed at least one test with zero skips (every journey skip-guard marks an unhealthy battery state — non-clock build, missing profile, missing seed), and that at least one walkthrough artifact was written during the run (the #1409 phantom-pass corroboration). The test-clock probe runs before the suite so a production-shaped stack fails in seconds. Targeted runs are exempt from the battery gate — any trailing Playwright filter or --devstack-profile snap-only marks the run as targeted — but still carry the #1409 reality watchdog: whenever a run’s JUnit report claims executed journey tests, at least one walkthrough artifact must have been written during the run (every journey spec writes them, 14/14), or the run fails loudly. This is the answer to the documented runner-level false-green, where Playwright emitted per-test ✓ lines and a green summary for journey tests that never ran their bodies:

The demo-review capture sweep’s 20 case-detail section-readiness assertions stay in the default battery as the ordinary functional suite tests/e2e/specs/case-detail-sections.spec.ts (light scheme, caseworker lane, exactly once); dark-scheme correctness coverage lives in the accessibility project’s per-target dark entries.

Every nextest profile carries slow-timeout = { period, terminate-after } — a genuinely hung test is killed and reported failed instead of blocking a battery forever (a starved-pool hang once ran 15 hours inside a pre-push battery with only SLOW warnings). Bounds, audited against the slowest legitimate tests in the 2026-08-26 battery (integration max 13.3s, unit max 11.2s):

Manual proof (2026-08-26): a deliberately-hung loop {{ sleep }} test under the default profile was terminated at 180.0s, reported TIMEOUT + run failed. A test that legitimately needs more than the kill bound is a design smell — split it or give it an explicit per-test override with a written rationale, never a global loosening.

cargo xtask {validate,test,e2e} wrap their test phases (nextest, doctest, Playwright) in a best-effort 1 Hz SysMonitor (xtask::sysmon, Linux /proc). When a phase ends it prints a one-shot summary line next to any FAIL lines:

Every sample also streams (timestamped, RFC3339-ms) to test-results/resource-samples.jsonl for post-hoc correlation against the nextest JUnit/timings. The signal is purely diagnostic — low load during a failure is evidence of a real bug; pegged CPU / IO-PSI spikes during a cluster of timeouts point at a perf/timeout flake. It never changes test behaviour, and is an inert no-op off Linux.

MultiReplicaFixture::spawn(service, n).await brings up N independent processes of a service binary:

Each replica gets a free 127.0.0.1 port + a unique CANOPY_MQ_REPLICA_ID + the shared CANOPY_MQ_QUEUE_PREFIX injected as env. Caller must pre-build the binary via cargo build --bin <service> (or just cargo build) before the fixture runs — spawn does NOT build. Honors CARGO_TARGET_DIR if set (CI runners typically set it).

EphemeralSchema::new_for_<service>(base_url).await creates a randomly-named Postgres schema (test_<16-hex-of-UUIDv4>), opens a pool scoped via ?options=-c search_path=<schema>, runtime-loads and runs the service’s migrations, and returns:

Migrations are runtime-loaded (#1380): run_service_migrations resolves services/canopy-<service>/migrations from the workspace tree and feeds sqlx::migrate::Migrator::new, so a migration edit is picked up by the next test run with no rebuild — the former compile-time sqlx::migrate! embed went stale until someone re-touched db.rs (the retired #1242 ritual: a false-green window running old SQL). A missing dir errors loudly with the resolved path. The define_ephemeral_schema_for!(service) macro generates the 17 one-line constructors in db_constructors.rs (canopy-verification included since #1380 — the old "no migrations directory" claim had been false since 2026-05-16).

Infra gating (#1376): infrastructure_available() AND-probes postgres and the rules canary (a partial stack is not available), and CANOPY_TEST_INFRA=required — set by validate, cargo xtask test’s infra lanes, and the in-network container — turns an unavailable stack into an immediate loud panic instead of a skip: battery lanes can never false-green DB-backed tests. Bare local runs stay skip-friendly. DEPENDENCY-level skips (#1396) — a setup helper returning `None after infrastructure_available() already passed (Keycloak token mint failed, a peer service unhealthy) — must go through canopy_test_lib::skip_or_panic(detail), which applies the same required-mode contract; a bare eprintln!("skipping …"); return; vacuously green-passes battery lanes (the six notices worker tests did exactly that for as long as they existed).

Every test schema is run-scoped: test_<run16>_<hex12> inside an xtask battery (CANOPY_TEST_RUN_ID, minted per battery, carried into the in-network container by compose interpolation) and COMMENT-stamped canopy-test run=<id> created=<epoch> at creation. Connections carry application_name = canopy-test:<service>:<schema> (pools) and canopy-test-admin:<purpose> (create/cleanup/sweep) — visible in pg_stat_activity and, with the devstack’s connection logging (#1379), in the rotated container logs.

Cleanup is layered, weakest to strongest:

Constructor failures never leak: from the instant CREATE SCHEMA commits, a scoped-pool connect failure or migration-replay failure triggers an awaited compensating DROP (the scoped pool closes first — sqlx can return before releasing its per-database migration advisory lock).

Other runs' schemas are NEVER swept automatically without positive inactivity evidence: stamp age > 6h AND no backend referencing the schema in application_name. The battery lease cannot prove another run dead (bare cargo nextest, other worktrees, in-network validate, other developers), and a sweep landing in a foreign run’s migration-wait window would redirect its unqualified DDL into public. Unmarked schemas are reported, never auto-dropped; cargo xtask dev sweep-schemas [--older-than N] [--include-unmarked] is the explicit, lock-held maintenance path.

Read by canopy_mq::subscriber::queue_prefix() (returns empty string when unset — identity behavior). Applied uniformly inside:

MultiReplicaFixture::spawn generates a fresh prefix per fixture so concurrent fixtures see fully-isolated queue namespaces against a shared RabbitMQ broker. Topic-exchange routing keys are unaffected — publishers continue to publish by routing key regardless of which prefixed queues happen to be bound.

4 invariant tests at crates/canopy-test-lib/tests/multi_replica_test.rs, all `#[ignore]’d, opt-in via:

Tests:

The e2e-reachable sibling of EvilLayer: every service built with the canopy-api/test-fault cargo feature serves an unauthenticated /test/fault control surface (GET reads, POST sets, DELETE clears) whose spec — { "latency_ms": 30000 } and/or { "status": 500 } — is applied by a middleware on the service’s /v1 router only: /livez, /readyz, and /test/* stay healthy, so an injected 30-second hold degrades pages without tripping container health checks. The same compile-stripped doctrine as /test/clock (ADR-033 §5): the route, the state, and the middleware do not exist in a release build — off by default, nothing to guard in production, and a devstack without the feature makes the fault specs test.skip cleanly (no impact on normal batteries).

Playwright drives it through tests/e2e/lib/fault.ts (withFaults(ctx, { applications: { latency_ms: 30_000 } }, body) — always clears every faultable service in a finally, the withAdvancedClocks contract). The fault project runs strictly LAST (after the journey lane) because the spec is process-global on the devstack. Build the stack with CANOPY_CARGO_FEATURES=canopy-api/test-clock,canopy-api/test-fault cargo xtask dev refresh.

The specs/fault-injection.spec.ts lane is the controlled-failure proof the ssr-aggregate-deadline plan deferred here: one upstream held at +30s lands the worker dashboard under the 15s nav budget with the degraded arms rendered, and the all-sources specs pin my-queue’s two static error copies (time-class vs plain) — assertions the healthy-stack dashboard spec deliberately refuses to carry.

SpanCapture’s thread-local constraint means events fired inside devstack containers (canopy-auth’s `JwksProvider refresh task, canopy-mq’s OutboxDrainer running inside services) are unobservable from the test process — the chaos tests were "fixture landed" not "invariant proven" (closed #469 + #470). The chaos module sidesteps this by spawning production components IN the test process pointed at controlled endpoints, so spans fire on the same current_thread runtime as the test.

Helpers: spawn_jwks_provider_for_chaos(EvilLayer) (in-process, no devstack) and spawn_outbox_drainer_for_chaos(pool, broker_url) (devstack-required, mark tests #[ignore] per the existing chaos pattern). See Shared Crates Reference for the per-function contract. ADR-020 documents the strategy decision.

Adding a new contract: follow the step-by-step runbook at the chaos-observability-contract runbook (canonical references for the target: annotation, helper authoring, the #[tokio::test(flavor = "current_thread")] requirement, and the deterministic-20x verification pattern). Landed contracts as worked examples: jwks_rotation (#481) in-process, outbox_catches_up (#482) devstack-gated.

Metric assertions (MetricCapture) are stubbed — wire via opentelemetry_sdk::metrics::InMemoryMetricExporter if a chaos test demands counter assertions.

Chaos tests and integration tests that exercise transient-blip behavior can opt into canopy_api::retry::RetryPolicy via TestClient::with_retry (or, for typed clients, the per-client pass-through such as EligibilityClient::with_retry):

When a policy is set, get / post_json / delete dispatch through canopy_api::retry::retry_request (3 attempts default; 100ms→30s backoff; ±25% jitter). POST auto-generates a single Idempotency-Key: {uuid v7} header outside the retry loop so the server-side cache replays the first response on retry. TestClient’s return types are unchanged — `get/post_json/delete still return TestResponse directly; transport errors still panic (same posture as the one-shot path); retry-exhaustion with an HTTP status synthesizes a TestResponse carrying the last status seen so callers reading .status get a real number. PATCH/PUT are intentionally NOT retry-wrapped (server-side idempotency middleware caches POST only). See crates/canopy-api/src/retry.rs for the contract and the retry-middleware design plan for the rationale (#462).