ADR-020: Cross-Process Chaos Observability via In-Process Production Fixtures

On this page

Context

The chaos tests in crates/canopy-test-lib/tests/evil_proxy_test.rs assert that production code emits structured tracing::Event instances (target: "retry", target: "jwks", target: "outbox") under EvilProxy-induced fault injection. Three of the four current tests are architecturally blind: SpanCapture::install_scoped (crates/canopy-test-lib/src/observability.rs:120-130) uses tracing::subscriber::set_default, which is thread-local in the test process. Production code emitting events inside devstack containers (canopy-auth’s JwksProvider refresh task, canopy-mq’s OutboxDrainer running inside each service) cannot reach the test process’s subscriber. The tests today are "fixture landed" rather than "invariant proven" — diagnosis closed \#469 and \#470 as duplicates under epic \&50, and motivates this decision.

\#462 (retry middleware, merged 2026-05-18) closed the in-process retry contract — the typed TestClient runs IN the test process, so its retry events ARE observable. The remaining three contracts (\#481 JWKS, \#482 outbox, multi-replica work) need a strategy.

Options considered

Option 1: In-process production fixtures (selected)

Instantiate JwksProvider and OutboxDrainer directly in the test process pointed at EvilLayer-wrapped endpoints. SpanCapture observes events because they fire on the same current_thread runtime as the test.

  • Pros: zero new infrastructure, zero new container plumbing, zero NAT routing complexity, zero production-code changes. Production constructors are already test-friendly (JwksProvider::from_discovery, OutboxDrainer::spawn, ConnectionManager::new). Aligns with the existing chaos-test architecture (EvilLayer, current_thread tokio, SpanCapture).

  • Cons: tests do NOT exercise the exact devstack process / network shape — they exercise the same production code via the same constructor surface, but in the test process. Acceptable because the behaviors under test (retry semantics, refresh-task event emission, drainer lease lifecycle) are independent of the process boundary.

Option 2: OTEL export

Production services already wire OTLP via canopy_common::telemetry, but devstack does NOT run a trace receiver — canopy-common/src/telemetry.rs:42 documents the trace export is explicitly stubbed today (metrics-only via Prometheus). Tests would deploy an in-process OTLP collector and scrape its span buffers.

  • Rejected. Requires standing up a full OTLP receiver in tests (parsing protobufs, buffering spans); adds the heavy opentelemetry-proto dep tree; containers running INSIDE Docker need NAT routing to reach the test process’s collector port (especially fraught on Linux Docker Engine where host.docker.internal resolution is opt-in); and the existing metrics export path is enough scope creep to derail the chaos-test work entirely. The harness-primitive-only path (Option 1) gets the chaos contracts unblocked first; OTEL is a future possibility for production observability, not test plumbing.

Option 3: Log scraping via Docker API

Production services emit JSON-structured logs (canopy-common/src/telemetry.rs:164). Tests could poll Docker stdout via bollard, parse each log line as JSON, extract structured tracing fields, apply assertions.

  • Rejected. Brittle (log-shape changes silently break tests; the JSON schema isn’t part of a contract); eventually-consistent polling (docker logs is not real-time); container-name coupling (tests need to map devstack container names → service IDs); multi-replica dedup logic is non-trivial; adds the bollard async Docker client dep. Existing chaos test infrastructure does NOT use Docker API integration; introducing it for tracing assertions is disproportionate.

Decision

In-process production fixtures (Option 1). The harness primitive lives in crates/canopy-test-lib/src/chaos/ and exposes two helpers:

  • spawn_jwks_provider_for_chaos(EvilLayer) → ChaosJwksHandle — spawns a static-document JWKS mock under the supplied EvilLayer, constructs canopy_auth::JwksProvider::from_discovery with a stable mock issuer (https://chaos-harness.test/realms/canopy).

  • spawn_outbox_drainer_for_chaos(PgPool, broker_url) → Result<ChaosOutboxHandle, anyhow::Error> — constructs canopy_mq::ConnectionManager::new(broker_url).await then canopy_mq::OutboxDrainer::spawn(pool, manager). Requires a live broker (devstack), pointed-at via CANOPY_TEST__RABBITMQ_URL.

Chaos tests using either helper MUST run on #[tokio::test(flavor = "current_thread")] — the thread-local-subscriber constraint is the architectural reason this strategy works AT ALL. On a multi-threaded runtime, set_default does not reach work-stealing tasks on other threads, and the assertions silently fail.

Consequences

Positive

  1. Zero production-code changes. The harness uses existing public constructors. No new methods, no new traits, no production refactor.

  2. Architectural invariant testable. Tests assert that in-process production code IS observable by SpanCapture — pins the constraint so future regressions are caught at the harness level rather than in each consumer chaos test.

  3. Symmetric for both contracts. Both \#481 (JWKS) and \#482 (outbox) consume the same harness primitive with parallel structure, so the chaos-test pattern stays consistent across the epic.

  4. Devstack-gated tests stay opt-in. Outbox-side tests follow the existing #[ignore] chaos pattern (evil_proxy_test.rs:48) so default cargo nextest run doesn’t require RabbitMQ; opt-in via --run-ignored only.

Negative

  1. Tests do not exercise devstack networking. The JwksProvider and OutboxDrainer run in the test process, not in the canopy-auth or canopy-mq service containers. A bug specific to the devstack networking layer (e.g., DNS resolution, TLS termination at a proxy, container-restart sequencing) would not be caught by these tests. Acceptable — those concerns belong to devstack integration tests and e2e, not chaos observability tests.

  2. AMQP-transparent EvilLayer not in scope. evil_proxy is JSON-only; it cannot intercept the AMQP wire protocol. Transient-AMQP-failure chaos for the OutboxDrainer requires an AMQP-transparent proxy layer, which is tracked in \#482, not this ADR.

  3. JwksProvider::start_refresh_task cannot be aborted. The production method is fire-and-forget (canopy-auth/src/jwks.rs:91-102 spawns and discards the JoinHandle). Chaos tests drive provider.refresh().await manually for deterministic timing; the harness does not own the refresh task lifecycle. Honest documentation is the mitigation.

  4. OutboxDrainer cannot be cleanly aborted by harness. The struct’s join handles are private (outbox_drainer.rs:174-208); spawned drain + janitor tasks live until the test runtime drops (current_thread runtime aborts all spawned tasks on drop, so this is reliable at test scope exit).

Implementation

Tracked in plan: docs/modules/ROOT/pages/plans/cross-process-chaos-observability-harness.adoc (\#480).

Consumers:

  • \#481 — JWKS chaos contract rewrite. Adds target: "jwks" to the 4 emit sites in canopy-auth/src/jwks.rs (lines 81, 98, 212, 216) AND a failure-path warn! in refresh() itself so chaos tests can assert on it.

  • \#482 — Outbox chaos contract rewrite. Adds target: "outbox" to the 4 emit sites in canopy-mq/src/outbox_drainer.rs (lines 213, 228, 367, 405). Requires AMQP-transparent EvilLayer work for transient-failure paths.

  • \#483 — Durable docs + runbook.

References

Edit this page · default