sweep_orphans Schema-Drop Race

On this page

Overview

canopy_test_lib::db::sweep_orphans(base_url, prefix) issues DROP SCHEMA <name> CASCADE for every schema whose name matches LIKE '{prefix}%'. Called with a workspace-shared prefix (test_) from a test that runs alongside other tests using EphemeralSchema, it will drop a sibling test’s live schema mid-run. This runbook documents the hazard and the safe-usage rule.

Symptom

A test in another crate fails non-deterministically with either:

  • relation "<table>" does not exist (Postgres 42P01) — its schema was dropped out from under it; or

  • a wall-clock blowout ("took 10.78s — per-program timeout not isolating") — lock contention during the drop.

Observed 2026-05-25: services/canopy-eligibility/tests/orchestrator_dispatch_test.rs::slow_program_does_not_block_combined_result failed both ways.

Root cause

sweep_orphans runs each DROP SCHEMA inside a SET LOCAL lock_timeout = '500ms' transaction (since the #520/#521/#523 MR), so a schema held by an active query raises lock_timeout rather than blocking. But lock_timeout only helps when the foreign session is mid-query — an idle pooled connection whose search_path points at the schema holds no namespace-level lock, so DROP SCHEMA CASCADE succeeds against it immediately and its next query `42P01`s.

The trigger 2026-05-25: crates/canopy-test-lib/tests/db_cleanup_test.rs called sweep_orphans(&url) with the shared test_ prefix. nextest runs canopy-test-lib and canopy-eligibility as separate OS processes against the same devstack Postgres, so the cleanup test’s sweep clobbered the eligibility test’s live test_<uuid> schema.

Safe-usage rule

  • Production / xtask usage: only call sweep_orphans(url, "test_") from a single-process context (e.g. cargo xtask dev refresh before/after the suite), never alongside parallel test binaries.

  • Test fixtures: mint your own unique prefix and sweep only that:

    let prefix = format!("sweepfixture_{}_", uuid_no_hyphens());
// create schemas under `prefix` via raw SQL ...
sweep_orphans(&url, &prefix);   // scoped — cannot touch another crate's test_<uuid>

    See crates/canopy-test-lib/tests/db_cleanup_test.rs::sweep_orphans_drops_matching_schemas_and_is_idempotent for the pattern.

Diagnosis pattern that worked

Five contextless agents dispatched in parallel without hypotheses: two converged on the sweep_orphans race (the true proximate cause); three chased the orchestrator’s retry policy and EphemeralSchema::Drop (real concerns, but not the proximate cause of these specific failures). The convergent answer is the trustworthy one.

Related

Edit this page · default