Chaos Observability Contracts: Adding a New Contract

On this page

Overview

A chaos observability contract asserts that production code emits a specific structured tracing::Event under fault-injected conditions. The contract is enforced by a test in crates/canopy-test-lib/tests/evil_proxy_test.rs that:

  1. Spawns the production component in the test process via the canopy_test_lib::chaos harness.

  2. Wraps the upstream dependency with EvilLayer (or points at a real broker for protocols evil_proxy cannot wrap, such as AMQP).

  3. Asserts via SpanCapture that events with a specific target: fired.

Existing contracts (epic \&50):

Issue Target Production code

\#462

target: "retry"

crates/canopy-api/src/retry.rs

\#481

target: "jwks"

crates/canopy-auth/src/jwks.rs

\#482

target: "outbox"

crates/canopy-mq/src/outbox_drainer.rs

This runbook covers adding a new contract for production code that doesn’t yet have one.

See ADR-020 for the strategy decision (in-process fixtures vs OTEL vs log scraping) and the thread-local-subscriber constraint that motivated the harness primitive.

The thread-local subscriber lesson

SpanCapture::install_scoped uses tracing::subscriber::set_default, which is thread-local in the test process. Three implications worth internalising before writing a chaos test:

  1. Tests using SpanCapture MUST use #[tokio::test(flavor = "current_thread")]. On a multi-threaded runtime, work-stealing spawns tasks onto threads that do not see the test thread’s default subscriber. The assertions silently fail under nextest.

  2. Production components running in devstack containers are unobservable. Events emitted by JwksProvider running inside canopy-web, or OutboxDrainer running inside canopy-snap, fire on a different process than the test — no cross-process tracing capture today.

  3. The chaos harness side-steps the cross-process problem by spawning the production component IN the test process pointed at a controlled endpoint. Same constructor, same code, same emit sites — just executed on the test’s runtime so the thread-local subscriber sees the spans.

Adding a new chaos contract — step by step

1. Add a target: to the production emit sites

In the production code module (e.g. crates/canopy-mq/src/inbox_drainer.rs), add target: "<name>" to every tracing::info! / warn! / error! that names an invariant the chaos test should assert on. Use a stable name that an operator would grep:

info!(
    target: "inbox",
    drainer_id = %cfg.drainer_id,
    "inbox drainer started"
);

2. If the contract needs failure-path observability, ensure the failure path emits

? shortcut propagation is the most common source of un-emitted failures. The pattern that makes chaos tests work:

pub async fn refresh(&self) -> Result<(), anyhow::Error> {
    let result: Result<JwkSet, anyhow::Error> = async {
        let resp = self.client.get(&self.jwks_uri).send().await?;
        Ok(resp.json().await?)
    }.await;

    match result {
        Ok(jwks) => {
            info!(target: "jwks", "JWKS refreshed");
            *self.keys.write().await = Some(jwks);
            Ok(())
        }
        Err(e) => {
            warn!(target: "jwks", error = %e, "JWKS refresh failed");
            Err(e)
        }
    }
}

This is the pattern \#481 introduced for JwksProvider::refresh().

3. If the production component is not yet harness-spawnable, add a helper

canopy_test_lib::chaos currently provides:

  • spawn_jwks_provider_for_chaos(EvilLayer) → ChaosJwksHandle

  • spawn_outbox_drainer_for_chaos(pool, broker_url) → Result<ChaosOutboxHandle, anyhow::Error>

To add a new helper (e.g. spawn_inbox_drainer_for_chaos):

  1. Verify the production constructor’s signature: is it test-friendly (constructor-injectable, clone-friendly, async)? JwksProvider::from_discovery and OutboxDrainer::spawn were both ready out of the box. If your target component requires production-code changes to be testable, that’s a separate scope concern — discuss before adding.

  2. Model the helper on chaos/jwks.rs or chaos/outbox.rs:

    1. Take an EvilLayer (HTTP) or broker_url (AMQP / other non-HTTP) parameter.

    2. For HTTP: spawn a static-document mock (modeled on crate::mock::spawn_mock_jwks), wrap with evil_proxy, construct the production component pointed at the EvilLayer URL.

    3. For AMQP / other: take the URL directly; the chaos test passes a real devstack URL.

    4. Return a ChaosXHandle struct holding the production component + MockHandle + EvilProxyHandle as _-prefixed fields so their Drop impls run when the handle is dropped.

  3. Re-export from crates/canopy-test-lib/src/lib.rs next to the existing harness re-exports.

  4. Document the thread-local-subscriber constraint in the helper’s module rustdoc.

4. Write the chaos test

In crates/canopy-test-lib/tests/evil_proxy_test.rs, add a #[tokio::test(flavor = "current_thread")] that:

  1. Installs SpanCapture::install_scoped (capture must come before the helper spawns anything).

  2. Drives a fail-then-recover cycle. The two phases serve different invariants:

    1. Fail phase — exercises the helper’s error path and the production failure-path emit.

    2. Recover phase — exercises the success-path emit.

  3. Asserts capture.assert_span_emitted("<your-target-name>").

If the test requires devstack (e.g., a real broker), gate it with #[ignore = "chaos: requires devstack + opt-in via cargo nextest run --run-ignored only"] and check infrastructure_available().await early. If the test is fully in-process (mock + EvilLayer + spawn in-process), drop the #[ignore] gate — let it run in the regular nextest sweep.

5. Verify determinism

Run the test 20 times in a tight loop:

for i in $(seq 1 20); do
  cargo nextest run -p canopy-test-lib <test_name>
done

(Add --run-ignored only if the test is #[ignore]-gated.)

Any failure / flake means the test depends on something timing-, network-, or scheduler-sensitive. Investigate — do not retry-until-green (memory feedback_no_flake_dismissal). The four contracts in epic \&50 each verified 20/20 PASS at single-digit ms before merging.

6. CHANGELOG + commit

CHANGELOG.adoc entry under === Added. Include:

  • Which target: was added and where (file:line for each emit site).

  • Which fail-then-recover cycle the chaos test drives.

  • The deterministic-run count (20/20 PASS) + per-test wall-clock.

  • Which transient-failure paths are out of scope (e.g., AMQP-transparent EvilLayer requirement; see the \#482 entry for the canonical wording).

Example diff: \#481 JWKS chaos contract (merged 2026-05-18, MR \!334)

Production:

- info!(jwks_uri = %self.jwks_uri, key_count = jwks.keys.len(), "JWKS refreshed");
+ info!(target: "jwks", jwks_uri = %self.jwks_uri, key_count = jwks.keys.len(), "JWKS refreshed");

Plus the match-based refactor of refresh() so the failure path also emits with target: "jwks".

Test (consumes the \#480 harness):

#[tokio::test(flavor = "current_thread")]
async fn jwks_rotation() {
    let (capture, _guard) = SpanCapture::install_scoped();

    // Fail phase
    let fail_handle = spawn_jwks_provider_for_chaos(
        EvilLayer::new().with_failure_rate(1.0)
    ).await;
    for _ in 0..3 {
        let res = fail_handle.provider.refresh().await;
        assert!(res.is_err());
    }
    drop(fail_handle);

    // Recover phase
    let ok_handle = spawn_jwks_provider_for_chaos(EvilLayer::new()).await;
    for _ in 0..3 {
        ok_handle.provider.refresh().await.expect("happy-path");
    }

    capture.assert_span_emitted("jwks");
}

20/20 PASS at 6-9 ms each. Fully in-process — no #[ignore] gate.

Common pitfalls

  • Asserting on a target that doesn’t exist yet. The chaos test fails with a diagnostic dump of all captured signals. Either add the target: in the same MR, or file the production-code change first and gate the test against it. Don’t merge a test that’s structurally impossible to satisfy — that’s the trap epic \&50 was filed to close.

  • Multi-threaded tokio runtime. #[tokio::test] defaults to multi-thread. #[tokio::test(flavor = "current_thread")] is required for SpanCapture to work.

  • Drop order in helper handles. If a helper holds a MockHandle AND an EvilProxyHandle, the proxy must drop before the mock (the proxy is forwarding to the mock; reversing the drop order produces a hanging Drop). Tokio’s default field-declaration drop order handles this — declare _evil_handle AFTER _mock_handle so the proxy drops first.

  • ?-shortcut hides the failure-path emit. If refresh().await returns Err but emits nothing, your chaos test’s failure-phase assertion will catch nothing. Pattern in step 2 above is the canonical fix.

  • Asserting on devstack-container events. Production code running inside a devstack container is not observable by SpanCapture in the test process — even when the test exercises the container via HTTP. The harness exists because of this. If your contract requires asserting on events from canopy-* services running in containers, you cannot use SpanCapture directly; the harness in-process spawn is the workaround.

  • Transient-AMQP-failure injection. evil_proxy is JSON-only; it cannot intercept AMQP. Chaos contracts that need transient-AMQP-failure paths (e.g., \#482’s "drainer recovers from publish failures") require an AMQP-transparent EvilLayer — currently unscoped. The basic "drainer fires target: "outbox" on spawn" contract is sufficient and works through the standard harness.

References

Edit this page · default