Plan: battery wave 2 — lane partition, topology helpers, run-scoped cleanup, honest coverage (#1377/#1381/#1379/#1382, epic &76)
On this page
Status
| Step | Description | Status |
|---|---|---|
0 |
Commit this plan + nav link; amend #1379’s kill-9 AC on the issue (cross-run auto-sweep is unsafe as written). |
Done (2026-08-08) — plan 7ab950c1; AC amendment note 3662592196 |
1 |
MR-1 |
Done (2026-08-08) — !1092 merged (edc2b5ad; force-merged past the fleet-wide runner disk-full #1397 per the documented procedure, pre-push battery green). Partition live-verified 5109 = 3203 ⊎ 1906; poisoned-env tripwire 3203/3203; both-modes evidence on the MR. #1377 closed with evidence; #1381 stays open pending #1393/#1394/#1396. #1391 filed en route. |
2 |
MR-2 |
Done (2026-08-08) — !1093 merged (939bffab). Battery green including the first live end-to-end |
3 |
MR-3 |
Done (2026-08-08) — !1094 merged (86dad825), battery green first try. Floor honestly reset: 42.1566 % measured in the pinned CI image (188 234 lines / 79 353 covered) → |
4 |
Delivery tail: file the bootstrap-vs-schema-migrations follow-up issue; close issues with evidence; epic &76 ticks; plan → Archive. |
Done (2026-08-08) — #1392 (migrations split) + #1399 (bootstrap-extraction debt) filed; #1377/#1379/#1382/#1395/#1398 closed with implementation + merge SHAs; #1381 held open pending #1393/#1394/#1396 per the honesty condition; #1397 (fleet CI disk) filed with the green→red flip evidence; standards-page coverage drift escalated upstream (claude-quickstart#27); epic &76 ticked; plan archived. |
Epic: &76
Issues: #1377, #1381, #1379, #1382 (all T1 — Correctness)
Branches: fix/battery-topology-lanes → fix/battery-schema-lifecycle → fix/battery-honest-coverage (stacked)
Three stacked MRs rather than one: a single ~126-file MR carrying topology changes, lane semantics, destructive cleanup, and a coverage rebaseline is unreviewable and unbisectable (external review round 2; the earlier one-MR batching grant was permission, not mandate).
Context (verified by exploration)
-
#1377:
Dockerfile.integration’s ENTRYPOINT `cargo nextest run --workspacereruns every workspace test. Infra-backed tests live in lib/bin targets (~62 src files, no usable naming convention, private-item access blocks moving them totests/targets). Renewals' devstack-driving lib tests (noEphemeralSchema) belong in the set too. -
#1381: ~124 postgres + ~25 AMQP hand-rolled URL sites across ~126 files in 8 shapes; program DBs (snap/tanf/medicaid/caps/wic) have no URL channel in-network at all;
TestConfigis the canonical seam (db.rs:63documents a never-built*_db_urlfield). -
#1379:
EphemeralSchemaDrop is a detachedtokio::spawncancelled at runtime shutdown; the schema leaks on migration and scoped-pool-connect failures before the RAII guard exists;sweep_orphanshas zero callers; 259/484cleanup()call sites discard errors with.ok(); no run-id plumbing exists. -
#1382:
cargo xtask coverageruns infra-less, every DB-backed test skips vacuously, and the 41.0 floor defends that number labeled as workspace coverage.
External review dispositions (round 2, folded into this design)
| Finding | Resolution |
|---|---|
Older-runs sweep can destroy live schemas; a dropped search_path entry mid-migration-wait redirects unqualified DDL into |
Automatic sweep = current run only, plus an evidence-gated older pass (schema-COMMENT age > 6h AND no referencing backend). Unmarked schemas are manual-only. #1379’s kill-9 AC openly amended on the issue. |
Sweep skipped when tests fail |
Lanes capture their Result; the sweep always runs; the original failure is returned as primary. Forced-failing-test verification. |
Lifecycle repair starts too late |
Armed guard immediately after |
Drop teardown connection storm |
Teardown pools |
sqlx per-database migration advisory lock × 16-thread infra arm |
Validate arms unify onto the same partition as every other lane: pure-unit@16 (zero infra), one complete infra lane@4. Migration locking untouched; bootstrap-vs-schema migration separation filed as a follow-up issue. |
Stale topology at render time; two divergent env renderers; |
One pure renderer |
Program DB fallback tiers can silently select the wrong physical database |
Programs are tier-1-or-panic (fail-closed, loud). Generic derivation only for services genuinely colocated on shared PG. |
validate-in-network omitted; program |
Its injection set becomes renderer-derived (invariant: injects every |
Shared-db overrides applied too narrowly |
Folded into the renderer — every env-rendering path (dev start/reload/restart, e2e, test, validate, refresh) gets identical topology inputs. |
Filterset placeholders not executable; complement-by-construction (candidate B) unsafe under proc-macro kinds |
Exact literal filtersets below; candidate A chosen now; xtask consts + a Dockerfile-parity lint check. |
Classification lint cannot enforce the criterion; out-of-line |
Lint is a necessary-signal check (module-path-aware, including out-of-line files); |
URL lint too coarse (file-level allowlist hides violations) |
syn string-literal parsing; per-literal |
Blocking checks absent from MR CI |
Static checks run in the MR cargo-test job; the partition check stays in pre-push validate per the repo’s CI policy (pre-push is the sole functional gate); the resulting MR-CI infra-test gap is documented explicitly in testing.adoc. |
|
All six PG services gain |
Run id too short / untyped / lost across env refresh |
16-hex typed |
URL-concat connection tagging fragile |
|
|
The reliable surface is the sweep report (a nonzero current-run swept-count means cleanup failures) plus captured output on test failure; AC reworded accordingly. |
Sweep API cannot represent its failure modes |
|
|
Every test schema becomes |
nextest list comparison underspecified |
JSON output, |
Coverage profiles inherit `default’s junit path (stale-report clobbering) |
Coverage profiles get an explicit distinct junit path |
Coverage false-greens; baseline artifact not commit-ready |
Coverage sets |
Integration coverage rots silently |
Documented as an informational developer command; |
"Both modes green" contradicted deferring newly-lit failures |
Both-modes green is required to close #1381; newly-exposed pre-existing defects get |
Design
1. Classification (#1377)
Criterion: any lib/bin test needing live infrastructure (EphemeralSchema, infrastructure_available(), PG/AMQP/devstack HTTP). Convention: the test sits in a module whose path contains an infra_tests segment.
| File state | Transform |
|---|---|
All tests infra |
|
Mixed |
Infra tests move to a sibling |
Several infra modules ( |
Nest under one |
Out-of-line |
Rename declaration and file to |
|
→ |
No test fn is renamed (three serialized-group filtersets are fn-name-based).
2. Lane partition (#1377) — one partition, everywhere
Exact expressions, defined once as consts in xtask/src/lanes.rs; the Dockerfile literal is lint-checked for parity:
INFRA = test(/(^|::)infra_tests::/) # regex: catches crate-root modules
UNIT = !test(/(^|::)infra_tests::/) # used with --lib --bins
INTEG = kind(test) | test(/(^|::)infra_tests::/) # candidate A, chosen now
VUNIT = !kind(test) & !test(/(^|::)infra_tests::/) # validate pure-unit arm
| Lane | Change |
|---|---|
|
|
|
Drop |
Container ENTRYPOINT ( |
|
Validate unit arm |
|
Validate infra arm |
|
CI cargo-test job |
|
|
|
The partition (UNIT-on-lib/bins ⊎ INTEG == full list) is enforced by lint check 3 on every validate. Follow-up issue filed at delivery: separate DB-global bootstrap migrations from schema-private migrations (prerequisite for any future migration-lock tuning; out of scope here).
3. Topology (#1381)
API (crates/canopy-test-lib/src/topology.rs, SPDX header, re-exported from lib.rs):
pub fn service_database_url(service: &str) -> String // programs: tier-1-or-panic
pub fn admin_database_url() -> String // db `canopy`
pub fn rabbitmq_url() -> String // always /%2f vhost
pub fn rabbitmq_url_for(user: &str, password: &str) -> String
Precedence (each tier unit-pinned; load_ports_env_file becomes pub(crate)):
| Service class | Resolution |
|---|---|
Programs (snap/tanf/medicaid/caps/wic) |
|
Non-programs (colocated on shared PG by design) |
|
Admin |
|
AMQP |
|
Single renderer (xtask/src/docker.rs): render_test_env(topology: &Topology) → Vec<(String, String)> — Topology { shared_db: bool, ports: … } passed explicitly by callers (dev start/reload/restart from CLI flags; others read the marker once, before rendering). It subsumes today’s write_ports_env + build_env_for_ports + SHARED_DB_ENV (all three currently divergent). The file writer serializes the rendering plus CANOPY_PORTS_ENV_VERSION=2; reconcile regenerates on version mismatch or rendering diff. Emits per-program CANOPY_TEST__<PROG>_DATABASE_URL in both topologies.
In-network: compose canopy-integration gains five program lines ${CANOPY_<PROG>_DB_URL:-postgres://canopy:canopy@postgres-<prog>:5432/canopy_<prog>} plus service_healthy depends_on entries for all five program PGs; validate_in_network.rs’s injection set becomes renderer-derived (invariant: it injects every `CANOPY_TEST__* key the renderer emits, with in-network values) so host-.ports.env tier-1 values can never leak into the container.
Replacement sweep (grep-driven, per shape): A → admin_database_url(); B → service_database_url("<svc>"); C/D/E → service_database_url("<prog>") (shape C’s skip-if-unset behavior disappears — flagged in the MR); F/G → rabbitmq_url() / rabbitmq_url_for(); xtask/tests/chain_genesis_test.rs + cmd/chain_genesis.rs migrate to the helpers too. Per-batch gate: CANOPY_TEST_INFRA=required cargo nextest run -p <pkgs> with devstack up (bare -p runs would false-green by skipping).
4. Schema lifecycle (#1379)
Run id: 16-hex typed TestRunId minted after acquire_battery_locks; carried by the renderer (survives every env refresh); compose passthrough CANOPY_TEST_RUN_ID: "${CANOPY_TEST_RUN_ID:-}"; test-lib validates the ambient value against ^[0-9a-f]{16}$, else treats it as absent (loud eprintln).
Naming: shared run_scoped_schema_name(label) → String = <label>_<run16|local16>_<hex12> — used by EphemeralSchema (test), migration_phase_test (test_migseam), and inbox_parking_test (migrates). At creation, stamp COMMENT ON SCHEMA … IS 'canopy-test run=<id> created=<epoch from now()>'.
Connections: PgConnectOptions::from_str(base_url) + .application_name() + .options([("search_path", …)]) — no URL concatenation (preserves query params/TLS/IPv6/percent-encoding). application_name = canopy-test:<service>:<schema> (≤ 59 bytes; real assert!). Admin/teardown/sweep connections tagged canopy-test-admin:<purpose>. Pool width untouched (#1207 pin).
Lifecycle: an armed guard is constructed immediately after CREATE SCHEMA; any later failure (scoped-pool connect, migration replay) triggers an awaited compensating DROP (scoped pool closed first — sqlx can return before releasing its session advisory lock), the original error primary with the cleanup error attached; disarmed only on success. cleanup() reordered: mem::forget only after the DROP succeeds; on error the guard drops normally (best-effort + sweep backstop). Drop teardown: max_connections(1), explicit close, global Semaphore(4), Handle::try_current guard; the detached mechanism otherwise unchanged; the false "periodic devstack refresh" comment and stale "12-hex v7" doc fixed.
Sweep (sweep_schemas in db.rs returning Result<SweepStats, SweepError>; orchestrator xtask/src/schema_sweep.rs):
-
URLs come from the renderer output (never
topology::*— its.ports.envload isOnce-cached per process), canonicalized, instances deduped. -
Instances: shared PG ∪ dedicated program PGs (per topology). Databases: an exact allowlist from the known-service const (∩
pg_databasewheredatallowconn), owner-checked. -
Current run (always — lanes capture their Result, the sweep runs, the original failure is returned as primary): drop schemas matching the run component (
LIKE '%_<run16>\_%' ESCAPE '\'on validated[a-z0-9_]names, identifier-quoted,DROP SCHEMA IF EXISTS … CASCADE, re-enumerate after), with statement/lock/connect timeouts, sequential per instance. One retry pass (~2s) for failures — a schema gone on retry is success — then bail loud. -
Older pass (evidence-gated): only schemas whose COMMENT parses AND age > 6h AND no backend in that DB references the schema in
application_name— positive inactivity evidence. Unmarked/foreign-comment schemas: reported, never auto-dropped. -
Per-instance/DB report table in battery output; a nonzero current-run count is the reliable cleanup-failure surface (nextest captures test stdout on success, so `cleanup()’s eprintln alone is not).
-
Manual
cargo xtask dev sweep-schemas [--older-than <dur>] [--include-unmarked]— takes the battery locks;--include-unmarkedis the explicit acknowledged-risk path.
AC amendment (posted on #1379): "kill -9 → zero schemas after the next battery" is unsafe as written (cross-run auto-sweep can destroy live runs; the migration-wait window even redirects DDL to public). Amended: killed-run schemas are removed by the next battery once evidence-gated (>6h + no backends) or immediately via dev sweep-schemas.
PG observability (same MR): all six PG services get -c log_connections=on -c log_disconnections=on -c log_lock_waits=on -c log_line_prefix='%m [%p] app=%a db=%d ' plus docker json-file log rotation (max-size/max-file), so application_name yields historical logs, not just live pg_stat_activity.
5. cargo xtask test-lanes-lint (blocking; static parts also in the MR CI cargo-test job)
-
URL literals: syn string-literal parse for
postgres:///amqp://credential-bearing literals; per-literal// canopy-lint: allow-url-literal (<reason>)markers (parser tests, redaction fixtures, deliberately-unreachable pool tests, renderer internals); no file-level blanket exemptions. -
Classification: module-path-aware walk (inline mods + out-of-line files by filesystem layout): infra identifiers (
EphemeralSchema, qualifiednew_for_*,infrastructure_available, topology helpers) inside[cfg(test)]code must have aninfra_testspath segment;test_supportmodules exempt but must contain zero[test]/#[tokio::test]items. This is a necessary-signal check; sufficiency is therequired-mode dynamic backstop (CI + the poisoned local proof). -
Partition + groups (after
nextest-build):nextest list --message-format json, matching records only (filter-match.status == "matches"), composite (binary-id, test-name) identity,--ignore-default-filterfor the raw inventory; assert unit ⊎ integ == full and disjoint; iterate every override filterset in every profile of nextest.toml, assert each matches ≥ 1 test. New validate gates (test-lanes-lint,schema-sweep) getvalidate_report::STAGESentries.
6. Honest coverage (#1382)
-
coverage.rs:cargo llvm-cov nextest --workspace --lib --bins -E 'UNIT' --profile coverage --summary-only --fail-under-lines <t>withCANOPY_TEST_INFRA=required(a stowaway infra test fails instead of skipping vacuously). Labeled unit coverage everywhere (docs currently claim workspace-wide — corrected). -
Profiles
coverage(8 threads) /coverage-integration(4 threads, slow-timeout 120s,terminate-after 2) with an explicit junit pathtest-results/coverage/results.xml(profiles inheritdefault’s junit — omission would clobber `test-results/unit/). -
Floor: measured in the CI image with pinned cargo-llvm-cov + cargo-nextest versions (pinned in the job); the committed artifact is a normalized
coverage-baseline.toml(scope, line totals, tool/toolchain versions, date) — the 155 KB.coverage-baseline.jsonstays gitignored.DEFAULT_THRESHOLD= CI-measured − 1.5; CHANGELOG documents the one-time reset. -
coverage --integration: mirrors thetest --integrationbootstrap (locks → ensure_ready → required → run id →-E 'INTEG'→ sweep). Documented as an informational developer command (no CI invocation, no floor) — explicit, not silent.
Commits
MR-1 fix/battery-topology-lanes (Relates to #1381 until both-modes green, then close; Closes #1377)
| # | Subject |
|---|---|
1 |
|
2 |
|
3–9 |
|
10 |
|
11 |
|
12–16 |
|
17 |
|
18 |
|
MR-2 fix/battery-schema-lifecycle (Closes #1379)
| # | Subject |
|---|---|
19 |
|
20 |
|
21 |
|
MR-3 fix/battery-honest-coverage (Closes #1382)
| # | Subject |
|---|---|
22 |
|
23 |
|
Verification (per MR battery, plus these)
| # | Check |
|---|---|
1 |
Partition set-verify (JSON, matching-only, composite IDs): unit ⊎ integ == full, disjoint — pasted into MR-1; permanent via lint check 3 |
2 |
Both in-network modes ( |
3 |
Poisoned local proof: unit lane with all |
4 |
First shared-db startup and isolated↔shared transitions render correct URLs (no stale-marker window); an in-sync pre-wave |
5 |
Forced-failing-test battery: the sweep still runs, the original failure is reported as primary |
6 |
Lifecycle unit tests: scoped-pool connect failure → schema dropped; migration failure → schema dropped, advisory lock released, original error preserved; |
7 |
Two concurrent run ids cannot touch each other (current-run-only + evidence gate); invalid ambient run id rejected; quoting/ |
8 |
Post-battery: current-run prefix = zero rows across every allowlisted DB on every instance (sweep |
9 |
|
10 |
Coverage floor reproduced in the CI image (pinned tools) before commit; badge regex verified in the MR pipeline; |
11 |
Connection-count and wall-clock deltas before/after (from PG logs + validate-report stage timings) — posted in closing comments |
Risks
-
Cross-run deletion is the top hazard — bounded by current-run-only auto-sweep + evidence gates; the residual is the manual
--include-unmarkedpath, which is explicit and lock-held. -
Classification completeness is load-bearing (CI cargo-test reds on stragglers) — syn lint + poisoned proof +
requiredbackstop. -
The validate infra arm at 4 threads runs more tests than today’s integration arm — the wall-clock delta is measured in MR-1’s battery; the advisory-lock serialization means the 16-thread arm was largely illusory parallelism for DB tests anyway.
-
Renderer consolidation touches every devstack path — verification #4 (parity + transitions) covers; e2e bring-up is exercised by MR-1’s battery.
-
MR CI still runs no live infra tests (repo policy: pre-push validate is the functional gate) — now documented rather than implicit.
Errata
-
SweepErrortyped enum not built (post-completion audit, 2026-08-08). The review-disposition table promisedResult<SweepStats, SweepError>with instance/db/schema/phase fields. Delivered shape:sweep_schemasreturnsResult<SweepStats, sqlx::Error>(a typed passthrough — canopy-test-lib’s only failure source here IS sqlx) and the xtask orchestrator attaches instance/db/schema context viaanyhow::Context, which is the coding-conventions split (thiserror-class enums for library error taxonomies, anyhow + context in application code). A bespoke enum would wrap one variant for one caller; the failure-mode information the review demanded is all present in the sweep report + error chains. Recorded as a deviation, resolved in favor of the conventions.