Plan: cross-process chaos observability harness (#480) + contested-environment parity (epic &80)

On this page
NOTE

Scope expansion (2026-08-25). Phase 1 (the #480 in-process chaos harness, epic &50) shipped 2026-05-18/19; its Status table below now records that (fixing the drift filed as #1588). Phase 2 — reaching parity with CRAIG’s contested-environment program — is the new scope, tracked under epic &80. See Phase 2.

Context — the larger picture

Phase-1-era diagnosis (2026-05). Line anchors in the phase-1 sections below are as of 54b0fbd1 (2026-05-19) — grep by name against today’s tree.

Three of the four chaos tests then in flight (jwks_rotation and outbox_catches_up in crates/canopy-test-lib/tests/evil_proxy_test.rs, plus the multi_replica_test.rs suite) named invariants the SpanCapture primitive could not observe. SpanCapture::install_scoped at crates/canopy-test-lib/src/observability.rs:120-130 uses tracing::subscriber::set_default, which is thread-local in the test process. Production code emitting events inside devstack containers (canopy-auth’s JwksProvider refresh task, canopy-mq’s OutboxDrainer running inside each service) is invisible to the test process’s subscriber — the chaos tests today are "fixture landed" rather than "invariant proven" (the diagnosis that drove #469 + #470, both now closed as duplicates under epic &50).

#462 (retry middleware, merged via MR !332) closed the in-process retry contract. The remaining three contracts (#481 JWKS, #482 outbox, multi-replica work) needed a harness primitive that lets them run production components in the test process so SpanCapture can see their events — all three have since shipped (see "Out of scope" below).

Strategy decision (ADR-020)

Three candidates evaluated:

  1. In-process production fixtures — instantiate JwksProvider and OutboxDrainer directly in the test process pointed at EvilLayer-wrapped endpoints. SpanCapture observes spans because they fire on the same current_thread runtime as the test.

  2. OTEL export — devstack has no trace receiver today (canopy-common/src/telemetry.rs:42 documents this); container-to-test routing complexity; heavy dep tree (opentelemetry-proto). Rejected.

  3. Log scraping via Docker API — brittle (log-shape coupling), eventually-consistent polling, container-name coupling, new bollard dep. Rejected.

Decision: in-process production fixtures. Zero new infrastructure, zero new container plumbing, zero production-code changes. Production constructors are already test-friendly. ADR-020 documents the decision + the thread-local-subscriber constraint.

Plan-file location

This canonical plan at docs/modules/ROOT/pages/plans/cross-process-chaos-observability-harness.adoc is the durable artifact per ADR-013. Linked from docs/modules/ROOT/nav.adoc under ** Infrastructure. Implementation begins only after this commit + ADR-020 + nav entries land on the feature branch (Step 1).

Status — phase 1: the harness (#480, shipped)

Step Description Status

1

Land canonical plan + ADR-020 + nav entry. This commit. REWRITE this canonical .adoc to match the reviewed design (replaces an earlier stale draft on the feature branch). Write docs/modules/ROOT/pages/adrs/adr-020-cross-process-chaos-observability.adoc. Add 2 entries to docs/modules/ROOT/nav.adoc: one ADR row after ADR-019, one plan row under ** Infrastructure after the canopy-api retry middleware entry. AsciiDoc passthroughs #[...] around any Rust attribute references so they don’t collide with AsciiDoc …​ mark syntax.

Done (2026-05-18) — e99b1b2f

2

crates/canopy-test-lib/Cargo.toml (~2 lines). Add to [dependencies]: canopy-auth = { workspace = true }, canopy-mq = { workspace = true }. No serial_test, no lapin (helper’s Result return type uses anyhow::Error via .map_err(anyhow::Error::from); anyhow is already a dep). Cycle check: cargo build -p canopy-test-lib clean (canopy-auth + canopy-mq dev-depend on canopy-test-lib; dev-deps don’t propagate to normal-dep cycle detection).

Done (2026-05-19) — 54b0fbd1

3

crates/canopy-test-lib/src/mock.rs::spawn_mock_jwks (NEW, ~30 LOC). Modeled on spawn_mock_persons at mock.rs:165-209. Serves a static canonical Keycloak realm JWKS document at /protocol/openid-connect/certs — a single RS256 public key payload baked into the helper as a string literal. No keypair generation, no JWT issuance. Returns MockHandle. The fixed key shape is enough for JwksProvider::refresh() to parse and cache; consumers that need to issue JWTs against the mock add their own signing helpers in #481.

Done (2026-05-19) — 54b0fbd1

4

crates/canopy-test-lib/src/chaos/mod.rs (NEW, ~80 LOC). Module entry. Re-exports spawn_jwks_provider_for_chaos, spawn_outbox_drainer_for_chaos, ChaosJwksHandle, ChaosOutboxHandle. Module rustdoc spells out the thread-local-subscriber constraint with a #[tokio::test(flavor = "current_thread")] requirement + canonical use shape.

Done (2026-05-19) — 54b0fbd1

5

crates/canopy-test-lib/src/chaos/jwks.rs (NEW, ~120 LOC inc. tests). See "Harness API surface" below for the full signature. No background refresh task — JwksProvider::start_refresh_task is fire-and-forget (jwks.rs:91-102 discards the JoinHandle); tests drive provider.refresh().await manually for deterministic timing.

Done (2026-05-19) — 54b0fbd1

6

crates/canopy-test-lib/src/chaos/outbox.rs (NEW, ~100 LOC inc. tests). See "Harness API surface" below. Rustdoc documents: OutboxDrainer has no Drop / abort (outbox_drainer.rs:174 — private join handles); spawned tasks live until the test runtime drops. ConnectionManager::new connects immediately — pass a REAL broker URL. Transient-failure injection requires AMQP-transparent EvilLayer (evil_proxy is JSON-only) — tracked in #482. The info!("outbox drainer started") event fires immediately on spawn (BEFORE the first tick sleep) — observable in SpanCapture without any tick env-var manipulation.

Done (2026-05-19) — 54b0fbd1

7

crates/canopy-test-lib/src/lib.rs — pub mod chaos; next to existing pub mod observability; + re-exports.

Done (2026-05-19) — 54b0fbd1

8

4 unit tests across chaos::jwks::tests + chaos::outbox::tests (all #[tokio::test(flavor = "current_thread")]):

  • jwks_helper_returns_provider_usable_for_refresh (in-process, no devstack): EvilLayer::new() zero-failure passthrough; assert handle.provider.refresh().await returns Ok(()).

  • jwks_helper_emits_jwks_refreshed_event_under_in_process_subscriber (in-process, no devstack): install SpanCapture::install_scoped, drive provider.refresh().await, assert SpanCapture saw "JWKS refreshed" event (existing info! at jwks.rs:81). Pins the architectural invariant.

  • jwks_helper_returns_err_when_upstream_fails (in-process, no devstack): EvilLayer::new().with_failure_rate(1.0), provider.refresh().await returns Err. No tracing assertion (refresh() doesn’t emit on failure — #481’s scope).

  • outbox_helper_emits_drainer_started_event_under_in_process_subscriber (devstack-gated, #[ignore]): if !infrastructure_available().await { return; }; private fn amqp_url() reads CANOPY_TEST__RABBITMQ_URL (pattern from outbox_drainer_test.rs:22-30); private fn pg_url() reads CANOPY_PORT_POSTGRES_5432 and constructs postgres://canopy:canopy@localhost:{port}/canopy_persons (pattern from outbox_drainer_test.rs:35,52); install SpanCapture; call helper; sleep 50ms; assert SpanCapture saw "outbox drainer started" at outbox_drainer.rs:213. Marked #[ignore = "chaos: requires devstack + opt-in via cargo nextest run --run-ignored only"] matching existing chaos pattern.

Done (2026-05-19) — 54b0fbd1

9

CHANGELOG.adoc — entry under === Added. Template in "CHANGELOG entry template" below.

Done (2026-05-19) — 54b0fbd1

10

Docs: (a) Shared Crates — new "canopy-test-lib chaos helpers" subsection under canopy-test-lib. (b) Testing — paragraph near the existing SpanCapture-thread-local note.

Done (2026-05-19) — 54b0fbd1 (landed in the then-canonical .claude/docs/; content migrated to the Antora pages cited here)

11

Precommit Q1-Q8 + validate + push + MR. cargo fmt --all + cargo clippy --all-targets --workspace --locked — -D warnings clean per Coding Conventions (zero #[allow(clippy::*)]). cargo xtask validate clean. Push flow operator’s choice.

Done (2026-05-19) — 54b0fbd1 merged; #480 closed; epic &50 closed via #483 (runbook, 1a41ab99)

Issue: https://gitlab.com/gadhs/application/eligibility/canopy/-/issues/480
Epic: &50 — Chaos observability contracts
Branch: feature/cross-process-chaos-harness (per Git Workflow — type::feature uses feature/ prefix)
Labels: priority::medium, service::shared-crates, program::infrastructure, type::feature, workflow::done

Harness API surface

chaos/jwks.rs

pub struct ChaosJwksHandle {
    pub provider: canopy_auth::JwksProvider,
    pub mock_url: String,
    _mock_handle: crate::mock::MockHandle,        // Drop aborts axum
    _evil_handle: crate::evil::EvilProxyHandle,   // Drop aborts proxy
}

pub async fn spawn_jwks_provider_for_chaos(
    evil_layer: crate::evil::EvilLayer,
) -> ChaosJwksHandle {
    let mock = crate::mock::spawn_mock_jwks().await;
    let upstream_url = format!("http://{}", mock.addr);
    let evil = crate::evil::evil_proxy(&upstream_url, evil_layer);
    let discovery = canopy_auth::OidcDiscovery {
        issuer: "https://chaos-harness.test/realms/canopy".to_string(),
        jwks_uri: format!("{}/protocol/openid-connect/certs", evil.url),
        ..Default::default()
    };
    let provider = canopy_auth::JwksProvider::from_discovery(&discovery)
        .expect("from_discovery infallible for caller-built URL");
    ChaosJwksHandle {
        provider,
        mock_url: evil.url.clone(),
        _mock_handle: mock,
        _evil_handle: evil,
    }
}

The issuer is set to a stable mock value (https://chaos-harness.test/realms/canopy). Returning a provider with issuer: "" would be a footgun — JwksProvider::validate_token checks the JWT’s iss claim against this value. Tests that only exercise refresh() won’t notice, but a hypothetical consumer using validate_token would get confusing failures.

chaos/outbox.rs

pub struct ChaosOutboxHandle {
    _drainer: canopy_mq::OutboxDrainer,
}

pub async fn spawn_outbox_drainer_for_chaos(
    pool: sqlx::PgPool,
    broker_url: &str,
) -> Result<ChaosOutboxHandle, anyhow::Error> {
    let manager = canopy_mq::ConnectionManager::new(broker_url)
        .await
        .map_err(anyhow::Error::from)?;
    let drainer = canopy_mq::OutboxDrainer::spawn(pool, manager);
    Ok(ChaosOutboxHandle { _drainer: drainer })
}

Critical files

  • /home/bitskrieg/code/canopy/docs/modules/ROOT/pages/plans/cross-process-chaos-observability-harness.adoc (this file, REWRITE in Step 1)

  • /home/bitskrieg/code/canopy/docs/modules/ROOT/pages/adrs/adr-020-cross-process-chaos-observability.adoc (NEW, Step 1)

  • /home/bitskrieg/code/canopy/docs/modules/ROOT/nav.adoc (+ 2 entries, Step 1)

  • /home/bitskrieg/code/canopy/crates/canopy-test-lib/Cargo.toml (+ canopy-auth, canopy-mq normal-deps)

  • /home/bitskrieg/code/canopy/crates/canopy-test-lib/src/chaos/mod.rs (NEW)

  • /home/bitskrieg/code/canopy/crates/canopy-test-lib/src/chaos/jwks.rs (NEW)

  • /home/bitskrieg/code/canopy/crates/canopy-test-lib/src/chaos/outbox.rs (NEW)

  • /home/bitskrieg/code/canopy/crates/canopy-test-lib/src/mock.rs (+ spawn_mock_jwks)

  • /home/bitskrieg/code/canopy/crates/canopy-test-lib/src/lib.rs (+ pub mod chaos;)

  • /home/bitskrieg/code/canopy/CHANGELOG.adoc

  • Shared Crates (docs/modules/ROOT/pages/shared-crates.adoc)

  • Testing (docs/modules/ROOT/pages/testing.adoc)

CHANGELOG entry template

* *Cross-process chaos observability harness — `canopy_test_lib::chaos`
  module (\#480 + ADR-020).* NEW module + 2 helper functions that spawn
  production `JwksProvider` and `OutboxDrainer` in the test process
  pointed at `EvilLayer`-wrapped endpoints. Unblocks the three chaos
  contracts (\#481 JWKS, \#482 outbox, multi-replica work) that were
  previously architecturally blind: `SpanCapture::install_scoped` uses
  `tracing::subscriber::set_default` which is thread-local in the test
  process and could not observe events fired inside devstack containers.
  In-process fixtures sidestep the cross-process problem entirely while
  exercising the same production constructors.
  `spawn_jwks_provider_for_chaos(EvilLayer) -> ChaosJwksHandle` spawns
  a canonical-shape JWKS mock under the supplied `EvilLayer`, constructs
  `JwksProvider::from_discovery` with a stable mock issuer.
  `spawn_outbox_drainer_for_chaos(pool, broker_url) -> Result<ChaosOutboxHandle, anyhow::Error>`
  spawns `OutboxDrainer` against the supplied broker URL — chaos tests
  point at a real broker (devstack) and observe the `info!("outbox drainer started")`
  emit at `outbox_drainer.rs:213` immediately on spawn. Transient-failure
  injection requires AMQP-transparent EvilLayer (tracked in \#482).
  4 unit tests: 3 in-process JWKS tests + 1 devstack-gated outbox test
  marked `+\#[ignore]+` per existing chaos pattern. ADR-020 documents
  the strategy decision + the thread-local-subscriber constraint. Zero
  production-code changes. `shared-crates.adoc` +
  `testing.adoc` updated. Closes \#480.

Verification

  1. cargo nextest run -p canopy-test-lib chaos::jwks — 3 JWKS unit tests pass (in-process only).

  2. cargo nextest run -p canopy-test-lib --run-ignored only chaos::outbox — outbox test passes when devstack up; skips cleanly when down.

  3. cargo nextest run -p canopy-test-lib — full crate green.

  4. cargo build -p canopy-test-lib — no cycle.

  5. cargo fmt --all — --check + cargo clippy --all-targets --workspace --locked — -D warnings clean (zero #[allow]).

  6. cargo xtask validate clean.

Project-specific gotchas

  • SPDX header on every new .rs file line 1.

  • #![warn(missing_docs)] on canopy-test-lib (lib.rs:3) — every pub symbol in chaos:: needs ///.

  • Clippy -D warnings — zero #[allow(clippy::*)] carve-outs (memory feedback_no_clippy_papering).

  • nextest only: never cargo test (memory feedback_nextest_only).

  • No Q1-Q8 in commit messages (memory feedback_no_q1q8_in_commit).

  • Commit title ≤72 chars, prefix ^(feat|fix|chore|refactor|docs|test|ci):.

  • Thread-local subscriber constraint — chaos tests using the harness MUST use #[tokio::test(flavor = "current_thread")]. Documented in module rustdoc + ADR-020 + testing.md.

Documentation updates

  • CHANGELOG.adoc — entry under === Added.

  • Shared Crates — new canopy-test-lib chaos helpers subsection.

  • Testing — chaos-helpers paragraph + thread-local-subscriber callout.

Out of scope (separate issues / future MRs — all since shipped)

  • JWKS chaos contract rewrite — #481, closed (target: "jwks" emit sites landed in canopy-auth/src/jwks.rs).

  • Outbox chaos contract rewrite — #482, closed (target: "outbox" emit sites landed in canopy-mq/src/outbox_drainer.rs). Its AMQP-transparent fault-injection residue is now phase 2’s U3/U4.

  • Durable docs + runbook — #483, closed (1a41ab99; the chaos runbook).

  • Multi-replica chaos rewrite — superseded: owned by phase 2 (U14, epic &80) below.

Reuses existing patterns

  • canopy_test_lib::mock::spawn_router (mock.rs:51) + spawn_mock_persons (mock.rs:165-209) — axum mock shape, MockHandle abort-on-Drop.

  • canopy_test_lib::evil::{EvilLayer, evil_proxy, EvilProxyHandle} (evil.rs) — fault-injection layer + handle that aborts on Drop.

  • canopy_test_lib::observability::SpanCapture::install_scoped (observability.rs:120).

  • canopy_test_lib::infrastructure_available (infrastructure.rs:14) — devstack-presence gate.

  • canopy_auth::JwksProvider::from_discovery (jwks.rs:50) + OidcDiscovery (discovery.rs:70, derive(Default)).

  • canopy_mq::ConnectionManager::new (connection.rs:49, async, returns Result<_, lapin::Error>).

  • canopy_mq::OutboxDrainer::spawn(PgPool, ConnectionManager) (outbox_drainer.rs:183).

  • CANOPY_TEST__RABBITMQ_URL + private fn amqp_url() pattern (crates/canopy-mq/tests/outbox_drainer_test.rs:22-30).

  • CANOPY_PORT_POSTGRES_5432 + private fn pg_url() pattern (outbox_drainer_test.rs:35,52).

Phase 2 — contested-environment parity (epic &80)

WARNING

DRAFT — PENDING MAINTAINER PLAN REVIEW (2026-08-25). This phase-2 section was drafted and its children (#1589–#1605) filed without the maintainer’s plan review — agent-side reviewer rounds only, a process error the maintainer flagged. Every child carries planning::needs-plan and epic &80 carries the matching banner; NO unit may start until the maintainer reviews/amends this section and lifts the labels. Treat the unit table, weights, and dependency spine below as a proposal, not a ratified spec.

Why

CRAIG (sibling project, gadhs/application/ccwis/craig) ran a 25-unit contested-environment program (their epic &83, closed 2026-08-21): adversity — severed connections, black-holed sockets, fault schedules, crash residue, bursts — became a first-class deterministic test input, backed by a per-surface registry and a blocking ratchet. Yield: ~32 findings, 15 product-code defects, clustered in exactly four families — broker liveness (their worst: an AMQP path accepting TCP but never answering pinned a consume supervisor forever), unbounded awaits, auth-plane error handling (5 defects from wiremock alone), and write-path degraded UX (every mutating form lost all typed data on a backend blip). Their burst/ordering floods and pool-contention rig found zero product defects.

A four-reviewer parity assessment (2026-08-25; architecture, tooling/style, coverage/findings, canopy inventory + synthesis) mapped that program against canopy. Canopy is at or above parity on several capabilities (below) but has eight real gaps, and canopy’s two known unbounded awaits (#1320 write verbs, #1587 auth-exchange leg) sit squarely in CRAIG’s highest-yield defect family. This phase closes the gaps; the doctrine and enforcement spine keep them closed.

What canopy already has (do not redo)

  • Race/barrier harnesses — ADR-038 finalize race matrix (finalize_acceptance_test.rs), watch-channel-driven idempotency lease-steal suite (idempotency_concurrency_test.rs), 16-racer breaker probe, advisory / window-fence election tests, multi-replica fixture. At parity with CRAIG’s concurrent.rs; no unit filed.

  • Duplicate-delivery convergence — pinned across eligibility, renewals, security, notices, scheduler fencing. CRAIG’s dup storms found zero defects.

  • Lane partitioning — set-verified disjoint lanes (test-lanes-lint --verify-partition); ahead of CRAIG’s run-ignored monolith. Phase-2 gating lands as a new verified lane, never --run-ignored=all.

  • L7 fault injection — EvilLayer (latency jitter, failure rate, connection drop, payload tamper — tamper is ahead: TCP toxics cannot express it), per-service fail_on_step/commit_then_fail_on_step mocks, the shipped phase-1 chaos module.

  • Compile-stripped fault surface — /test/fault + the test-fault feature (#1325): absence-by-construction, a stronger model than CRAIG’s default-off feature fields. Read-path degraded-page e2e already proven (tests/e2e/specs/fault-injection.spec.ts).

Gaps (ranked by expected yield)

  1. AMQP transport severance/black-hole — no true-severance tooling (EvilLayer returns 502s rather than dropping sockets); the lapin no-reconnect trap is documented-and-accepted (xtask/src/cmd/e2e.rs:288). CRAIG’s worst defects were inexpressible in-process.

  2. Timeout legs on the known unbounded awaits — #1320 (write verbs), #1587 (auth exchange). Highest yield-per-weight; needs no new infra.

  3. Auth-plane contested contract — JWKS/discovery/introspection under adversity; the phase-1 chaos harness is a ready substrate.

  4. Write-path degraded-UX browser legs — form-data preservation, outage-renders-as-outage, redirect honesty under IdP outage.

  5. Silent-skip — 3 of the 4 evil_proxy_test.rs chaos tests, all 4 multi-replica tests, and all 3 reconnect tests are #[ignore] opt-in, structurally never running (only jwks_rotation runs in the default battery): the exact pattern CRAIG deleted. Multiplies every other gap’s yield.

  6. Crash-residue reap — sweeps/workers have races tested but not crash-window residue (a crashed run stranded as in-flight forever).

  7. Pool exhaustion + typed DB degradation posture — lowest defect yield, but canopy lacks the production posture (phase-aware acquire/statement/ commit error taxonomy, Retry-After) entirely.

  8. CLI lost-response recovery (verified during this review — CRAIG’s C18 class is real here) — tools/canopy-cli issues POST/PUT/DELETE across applications/persons/renewals with zero idempotency-key handling; a rerun after a lost response may double-apply.

Doctrine (ratified in U1)

  • Deterministic adversity: force the condition (barrier, fault schedule, severed proxy), never load-at-scale hoping to hit it.

  • Two-path rule: a graceful protocol close and a TCP severance are different failures; every MQ surface needs both. Record the lever rationale in-code at the call site.

  • Lever taxonomy: L1 in-process (EvilLayer, the in-tree mock.rs routers, per-service mocks, crash armers) · L2 transport (toxiproxy) · fault surface (compile-stripped features). L7 tampering stays EvilLayer; severance is L2’s job. No wiremock: canopy’s hand-rolled mock routers already cover the shape, per the hand-rolled-over-wrapper divergence.

  • Never make the battery easier: timeout budgets are fixture ceilings, never assertion widening; zero serialization/envelope hunks unless semantically required and called out; breach needs a plan amendment.

  • Seed-and-replay: every randomized draw env-pinned, failure output prints a copy-pasteable replay command.

  • Arbitration: GREEN (fix the fixture, never easier) / RED (real defect
    seeded repro) / CANNOT-RUN (fault-layer owner). Transient/environmental is never a terminal disposition. A zero-fault green battery is a FAILURE once accounting (U13) lands.

  • Non-vacuous oracles: reconnect must land on a connection whose name differs from the pre-sever capture; barriers derive from the app’s own constants, never round numbers.

  • Honest claims: in-code residue lists name what a gate cannot prove; "machine-enforced absence", never "impossible"; closeout counts separate product defects from test-infra/tooling/residue.

Deliberate divergences from CRAIG

  • Keep canopy’s set-verified lane partition (fault = a new verified lane).

  • Keep the compile-stripped fault-surface model; the cargo-tree gate (U16) is belt-and-suspenders, not the primary control.

  • Typed thiserror errors everywhere in NEW phase-2 code, test-lib included (no anyhow at pub boundaries). Phase-1’s shipped spawn_outbox_drainer_for_chaos anyhow boundary is grandfathered; retrofit only if a unit touches that file.

  • Skip the 6-axis tag lint — coverage forcing comes from the registry ratchet (U12), which is machine-decidable.

  • Skip the Postgres port-lease allocator until contention is measured (the integration lane is 4-thread; nextest test-groups serialize).

Units

Weights are implementation complexity (1/2/3/5/8). Issue refs are appended to each Unit cell (as #N) when children are filed under epic &80.

Unit Scope + acceptance criteria W Status

U1 (#1589)

Doctrine (docs). ADR-020 Amendment 1 — the contested-environment doctrine above, verbatim; Testing gains the doctrine section + the never-easier checklist as a blocking review item; CHANGELOG.

2

Not started

U2 (#1590)

EvilLayer determinism retrofit. Replace the unseeded rand::rng() draws (evil.rs:207,232) with a seeded StdRng; seed from CANOPY_FAULT_SEED or generated-then-printed; failure paths print a copy-pasteable nextest replay command; self-test pins same-seed ⇒ same fault schedule. Also correct the stale evil.rs module rustdoc: drop_connection_after surfaces as a 502
marker header (evil.rs:222-227), not a transport-level socket drop as the doc claims.

2

Not started

U3 (#1591)

Toxiproxy fault layer in the devstack. Opt-in compose fault profile; digest-pinned image; loopback-only port range; standing mirrors for RabbitMQ, Postgres, Keycloak; hand-rolled typed control client in canopy-test-lib (typed toxic structs with unit-suffixed fields; RAII guard — async destroy(), Drop detect-only, SILENT-OK teardown markers); cargo xtask fault-preflight with --required (fail, never skip); compose-drift pin test (Rust port constants vs the compose publish strings). No port-lease allocator (see divergences).

5

Not started

U4 (#1592)

MQ two-path contested legs. Per consumer surface: graceful mgmt-close AND proxy-disable severance; establishment black-hole leg (TCP accepted, never answered) against consume supervisors; outbox-drainer severance catch-up (backlog slope ≤ 0 after heal); publisher-channel invalid-state confrontation of the documented lapin trap (e2e.rs:288) — expected to spawn fix: issues, filed separately. Reconnect oracle: post-heal connection name differs from pre-sever capture; barriers from canopy-mq’s own backoff constants.

8

Not started

U5 (#1593)

DB + object-store L2 legs. Acquire/statement/commit-phase severance on canopy-db (per-phase behavior pinned); ambiguous-commit pin (severed between COMMIT send and ack ⇒ outcome-unknown, no double-apply); canopy-store timeout + ambiguous-put legs.

3

Not started

U6 (#1594)

Deadline audit + timeout legs for S2S write verbs. Inventory every S2S write call’s deadline posture (the #1320 class); an EvilLayer latency-hold leg per call pinning it bounded — or a filed fix: issue per unbounded one (the fix itself stays #1320-side); the #1587 interim-contact exchange leg explicitly covered. In-process only; no U3 prerequisite.

5

Not started

U7 (#1595)

Auth-plane contested contract. JWKS refresh under non-200 (a proxied 503 must not wipe the live key cache); discovery outage classified as outage, not not-configured; introspection/validation clock-skew leeway pinned against token exp/nbf; JWKS stale-serve bound; RFC 8693 exchange deadline (#1587). Substrate: spawn_jwks_provider_for_chaos + the in-tree mock.rs routers, extended with non-200 / never-respond shapes as needed (no wiremock — see divergences); prior art #481.

3

Not started

U8 (#1596)

Degraded-UX write-path e2e legs. Mutating-form data preservation on backend failure; outage renders as outage (never 404) on detail + write handlers; download/login redirect honesty under IdP/backend outage. Status/latency legs ride /test/fault in the existing fault Playwright project; true-severance legs ride U3 standing mirrors. Read-path legs exist (#1325) — do not redo.

5

Not started

U9 (#1597)

Crash-residue armer + reap legs. A crash armer in canopy-test-lib (CRAIG’s PgFaultArmer shape): the test arms a statement-count cadence tracked through a non-transactional PG sequence on the target DSN, and the armer kills the session when the count fires — so the worker under test dies mid-write and leaves realistic residue (a row stranded in its in-flight state). A DSN denylist guard refuses to arm anything that is not a test database; a typed async finish() returns an armed-vs-fired report so an unfired arm fails the test rather than passing vacuously. Then: stranded-state reap legs for enact_sweep, scan_worker, the renewals scheduler, and the chain drainers — a crashed run must be distinguishable from in-flight and reaped under the sweep lease.

5

Not started

U10 (#1598)

Pool-contention rig. Pinned contender matrix vs canopy-db’s acquire_timeout — a hard-coded 5s const (lib.rs:179; DbPoolOpts has no knob, same posture as CRAIG’s const). The rig builds its own PgPoolOptions mirroring that constant, with a compile-time-adjacent pin (rig hold-time > the 5s bound) so a canopy-db constant change breaks the rig loudly; any production knob is U11’s decision, not this rig’s. Structural winners/timeouts split asserted; JSON report artifact; report-only.

3

Not started

U11 (#1599)

DB degradation posture ADR + typed taxonomy. Phase-aware mapping — acquire timeout ⇒ 503 + Retry-After, statement timeout ⇒ 504, commit-outcome-unknown typed; route-class ceilings; fleet adoption (pre-1.0 breaking OK with CHANGELOG); then promote U10 to latency-shape enforcement.

5

Not started

U12 (#1600)

contested-surfaces registry + AST census + per-class ratchet. Typed [[surface]] schema (class, AST-resolvable anchor, oracle prose, typed legs, status); census over subscribe / outbox-worker / sweep / S2S-client seams reusing test-lanes-lint’s AST machinery; PROMOTED vs REPORT_ONLY partition with promotion criteria in code + a partition unit test; the surface schema carries a per-consumer ordering-contract declaration field (commutative / revision-gated / buffered / strictly-ordered); --bless mints report-only stubs only; nextest exact-name inventory check; wired into cargo xtask validate.

8

Not started

U13 (#1601)

Executed-fault accounting + program gate. Fault-record primitive (countdown + recorder); reset/verify stamp pair around the battery; armed ⇒ fired; per-class fired floors starting mq-class only (U10’s rig emits fault records and adds the pool floor when it lands — U13 sits before U10 in the suggested order, so the floor set must not assume the rig); honest in-code residue list naming unrecorded classes.

5

Not started

U14 (#1602)

Fault lane + silent-skip deletion. Promote the 10 #[ignore]-gated tests (3 of 4 in evil_proxy_test.rs — jwks_rotation already runs in the default battery; all 4 multi-replica; all 3 reconnect) into a preflight-gated REQUIRED fault lane (a new verified lane in lanes.rs); CANNOT-RUN is a failure, never a skip; the #[ignore] markers deleted or converted; fix the stale evil_proxy_test.rs header comment claiming all its tests are ignored.

3

Not started

U15 (#1603)

Arbitration runbook (docs). GREEN/RED/CANNOT-RUN ladder; transient-never-terminal; capacity attribution only via a recorded GREEN; seeded-replay instructions. Extends the chaos runbook.

2

Not started

U16 (#1604)

Release-artifact gate. A cargo-tree assertion in cargo xtask validate that test-fault (and any future fault feature) appears in NO Dockerfile-built binary’s normal-dep graph. Claim: "machine-enforced absence".

1

Not started

U17 (#1605)

CLI lost-response/idempotency legs. Characterize canopy-cli’s recovery contract for its mutating verbs (verified: zero idempotency handling in tools/canopy-cli/src); mock-router lost-response legs — extend mock.rs with a commit-then-never-reply shape (the reply future holds forever after the mock records the write); pin either convergent rerun (server-derived key) or a disclosure requirement (request id printed pre-flight); defects ⇒ fix: issues.

2

Not started

Total ≈ 67 weight vs CRAIG’s ~97 — the discount is the standing inventory above. Plan-drift fix #1588 (weight 1) closes with the MR that lands this phase-2 section.

Dependency spine + suggested order

Hard blocks (encoded as GitLab blocked-by at filing): U3 → {U4, U5, U8, U14}; U12 → U13; U10 → U11.

Suggested order (yield-first): U1/U2 → U6 + U7 + U17 (no infra, highest yield-per-weight) → U3 → U4/U5 → U9 → U8 → U12 → U13/U14 → U10/U11 → U15/U16.

Skip list (evidence-backed)

  • Barrier/race harness unit — at parity (inventory above).

  • Burst/ordering flood legs — CRAIG harvested zero product defects; canopy duplicate-convergence already strong. The per-consumer ordering-contract declaration lands inside U12; the seed retrofit is U2.

  • Cluster docker-kill rig (their C20/L3) — deferred; the multi-replica fixture + RabbitMQ-restart tests cover most of the value. Revisit after U4 confronts the lapin trap.

Verification (phase 2)

  • Every unit: the full pre-push battery (the sole functional gate).

  • U3: cargo xtask fault-preflight --required green on a fault-profile devstack; compose-drift pin test green.

  • U12/U13: ratchet + program gate run inside cargo xtask validate; a fabricated uncovered surface / unfired armed fault fails the battery (negative test).

  • U14: the fault lane appears in the verified lane partition; deleting a lane test breaks the nextest inventory check.

  • Program closeout: defect census counted product vs test-infra vs residue (honest-claims doctrine).

Links

Epic: &80 (phase 2) — prior art &50 (phase 1, closed).
Children: #1589–#1605 (filed 2026-08-25; refs in the units table).
Related open issues: #1320 (write-verb bounds — U6 pins, #1320 fixes), #1587 (exchange deadline — U6/U7), #1271 (ELE fault harness, T2 — service-level sibling, relate to &80), #1588 (plan drift — closed by this MR).
Provenance: four-reviewer parity workflow, 2026-08-25 (session artifact; CRAIG evidence cited against gadhs/application/ccwis/craig at its 2026-08-25 state, incl. their contested-surfaces.toml, ADR-067/ADR-068, xtask contested.rs/program_gate.rs/fault_preflight.rs, and issues CRAIG-#1520–#1547).

Edit this page · default