Plan: retry observability contract (#462 narrowed)

On this page

Context — the larger picture

The canopy-test-lib port’s Phase C (MR !318, refs #436) landed four chaos tests in crates/canopy-test-lib/tests/evil_proxy_test.rs. Each chaos test points a typed client at an EvilLayer proxy and asserts a tracing::Event target fires in production code.

Architectural discovery during this planning cycle: SpanCapture::install_scoped (crates/canopy-test-lib/src/observability.rs:120-130) uses tracing::subscriber::set_default, which is thread-local in the test process. Three of the four chaos tests assert on events emitted by code running in different processes (devstack containers for canopy-auth’s JWKS refresh, canopy-mq’s outbox drainer inside services). SpanCapture cannot see those events. Only the retry chaos test (inbox_dedup_at_100_percent_failure at evil_proxy_test.rs:39-82) drives in-process code paths — the typed client / TestClient executes IN the test process, so any retry middleware wired there IS visible to SpanCapture.

Revised structure (split into a GitLab epic + shippable children, per GitLab Workflow requirements):

  • EPIC (new GitLab group-level epic): titled "Chaos observability contracts: cross-process capture + retry/JWKS/outbox". Filed via glab api groups/gadhs%2Fapplication%2Feligibility/epics -X POST -f title=…​. Children linked via epic_id API field per gitlab-workflow.md "Issue-epic linking" — NOT description-only cross-references.

  • #462 (renamed via glab issue update 462 --title "…​"): "Retry observability contract". Linked to the new epic via epic_id. This plan implements #462 in its narrowed form.

  • Child A: "Cross-process chaos observability harness" — decide and build the test strategy (in-process production fixtures vs OTEL export vs log-capture).

  • Child B: "JWKS chaos contract" — rewrite jwks_rotation at evil_proxy_test.rs:140-176 to drive JwksProvider behavior via the chosen harness + add target: "jwks" to the 4 emit sites in crates/canopy-auth/src/jwks.rs (81, 98, 212, 216).

  • Child C: "Outbox chaos contract" — rewrite outbox_catches_up at evil_proxy_test.rs:184-220 to create an actual outbox failure-then-recovery path + add target: "outbox" to the 4 drain-retry emit sites in crates/canopy-mq/src/outbox_drainer.rs (213, 228, 367, 405).

  • Child D: "Durable chaos docs + runbook" — update Testing (SpanCapture-is-thread-local lesson) + Shared Crates.

This plan is scoped exclusively to the narrowed #462 (retry only). The epic + other children get filed in Step 2 (after Step 1 lands this canonical plan file so the epic description can include a working clickable plan URL).

Why a retry layer is justified beyond closing the chaos test:

  • The orchestrator’s program-service dispatch (services/canopy-eligibility/src/orchestrator.rs:600-618) is one-shot — a transient 503 from canopy-snap fails the entire SNAP determination.

  • canopy-test-lib::TestClient is one-shot — devstack-flake-induced test failures bubble through to E2E assertions. A bounded retry kills a class of test flakes.

  • The server-side idempotency middleware (crates/canopy-api/src/idempotency.rs:343-364) was built specifically to make client-side retries safe — caching first responses keyed on Idempotency-Key header — but no client today emits the key or retries.

Status

Step Description Status

1

First implementation action — create the canonical plan file. Copy this file into docs/modules/ROOT/pages/plans/canopy-api-retry-middleware.adoc. Add an entry to docs/modules/ROOT/nav.adoc under Infrastructure: * canopy-api retry middleware (#462). Commit + push as the FIRST commit on the implementation branch — this commit creates the URL the GitLab epic description (Step 2) will link to. No code changes precede this commit.

In progress

2

GitLab admin (after Step 1 lands the canonical plan file). Step 1 must complete first so the epic description below can include a working clickable plan URL. (a) Create the epic at the group level via glab api groups/gadhs%2Fapplication%2Feligibility/epics -X POST with description following gitlab-workflow.md "Epic description format" — Summary, Plan link (clickable URL to this .adoc), Task list with - [ ] #N title (weight: W) for each child. Weights: #462 retry (3), child A cross-process harness (5), child B JWKS (3), child C outbox (3), child D docs (2). (b) Rename #462 to "Retry observability contract" and link via epic_id. (c) File children A/B/C/D with full Issue Standards descriptions, link each via epic_id. (d) Update epic description’s task list with actual child IIDs once filed.

Not started

3

crates/canopy-api/src/retry.rs (NEW, ~300 LOC inc. tests) — provides RetryPolicy, RetryError, RetryDecision, RetryRejection, classify, RetryRequest, and the entry point retry_request(policy, request, make_req). Hand-rolled exponential-backoff loop matching canopy_mq::connection::ConnectionManager::reconnect at crates/canopy-mq/src/connection.rs:93-132. Constants: BACKOFF_INITIAL_MS: u64 = 100, BACKOFF_MAX_MS: u64 = 30_000. SPDX header line 1. RetryPolicy fields private with with_* setters asserting invariants. RetryRequest fields private with typed constructors get() / head() / delete() / post_with_idempotency_key() / try_new(method, has_key). Caller-declared safety (descriptor records the caller’s assertion; retry_request does NOT inject the header itself). Ok(Response) returned for terminal success AND terminal non-retryable HTTP (body preserved). Per-attempt timeout via tokio::time::timeout. Sleep capped to remaining overall budget. classify takes Option<&reqwest::Error> (no stringification before decision). RetryError::Network wraps the typed reqwest::Error. Every pub symbol has a /// doc.

Not started

4

crates/canopy-api/src/lib.rs — add pub mod retry; + re-exports. crates/canopy-api/Cargo.toml — add BOTH rand and thiserror (workspace pins: rand = "0.9", thiserror = "2" per root Cargo.toml). Cycle check via cargo metadata before adding canopy-api as a normal dep of canopy-test-lib in Step 6.

Not started

5

crates/canopy-test-lib/src/client.rs::TestClient — add retry_policy: Option<canopy_api::retry::RetryPolicy> field + with_retry builder. Modify get/post_json/delete to dispatch through retry_request when policy is set. Preserve existing return types (TestResponse, not Result). Synthesize TestResponse for retry-exhaustion-with-status; panic for retry-exhaustion-with-network-error (mirrors existing client.rs:250). Idempotency-Key generated OUTSIDE the loop. PATCH/PUT not retry-wrapped in this MR (server-side cache covers POST only). Closure replicates TestClient::auth precedence — service_api_key over auth_token (client.rs:233-241).

Not started

6

crates/canopy-test-lib/Cargo.toml — add canopy-api = { workspace = true }. cargo build -p canopy-test-lib clean.

Not started

7

crates/canopy-test-lib/src/clients/eligibility.rs — add with_retry(self, policy) pass-through.

Not started

8

crates/canopy-test-lib/tests/evil_proxy_test.rs:61 — single-line edit to enable retry on the chaos test’s EligibilityClient.

Not started

9

6 unit tests in retry.rs::tests covering classification, API-boundary safety, per-attempt timeout, backoff curve match, ±25% jitter. Default #[tokio::test] multi-thread runtime; in-process axum mock per crates/canopy-test-lib/src/mock.rs::spawn_router pattern.

Not started

10

services/canopy-eligibility/src/orchestrator.rs:580-618 — wrap dispatch in retry_request. Generate det_id = uuid::Uuid::now_v7() above the spawn; pass as Idempotency-Key. overall_timeout = config.timeout. Existing match arms unchanged.

Not started

11

CHANGELOG.adoc entry under === Fixed. Template in "CHANGELOG entry template" below.

Not started

12

Docs: Shared Crates (new canopy-api retry subsection) + Testing (TestClient::with_retry paragraph).

Not started

13

Precommit Q1-Q8 + validate + push. cargo fmt --all + cargo clippy --workspace --tests — -D warnings clean (zero #[allow(clippy::*)]). cargo xtask validate 1734/1734 + e2e + docker, exit 0.

Not started

Issue: https://gitlab.com/gadhs/application/eligibility/canopy/-/issues/462 (renamed in Step 2)
Epic: TBD — created in Step 2
Branch: feature/canopy-api-retry-middleware (per Git Workflowtype::feature uses feature/ not fix/)
Labels: priority::medium, service::shared-crates, program::infrastructure, type::feature, workflow::ready

Design

retry.rs API surface

// SPDX-License-Identifier: AGPL-3.0-or-later

//! Bounded exponential-backoff retry for outbound reqwest calls.
//!
//! Coordinates with the server-side idempotency middleware
//! (`canopy_api::idempotency_middleware`) so retried POSTs replay
//! safely. Emits `tracing::info!(target: "retry", ...)` before each
//! retry attempt — operators grep this target in canopy_logs to
//! correlate transient downstream blips with the runbook.

use std::time::Duration;

const BACKOFF_INITIAL_MS: u64 = 100;
const BACKOFF_MAX_MS: u64 = 30_000;

/// Tunables for retry behavior. Fields are private; construct via
/// [`RetryPolicy::default_http`] and refine via the `with_*` setters,
/// which assert invariants (no zero `max_attempts`, no NaN/negative
/// `jitter`).
#[derive(Debug, Clone)]
pub struct RetryPolicy {
    max_attempts: u32,
    initial_backoff: Duration,
    max_backoff: Duration,
    overall_timeout: Option<Duration>,
    jitter: f64,
}

impl RetryPolicy {
    /// HTTP-typical defaults: 3 attempts, 100ms → 30s backoff, ±25%
    /// jitter, no overall_timeout.
    pub const fn default_http() -> Self {
        Self {
            max_attempts: 3,
            initial_backoff: Duration::from_millis(BACKOFF_INITIAL_MS),
            max_backoff: Duration::from_millis(BACKOFF_MAX_MS),
            overall_timeout: None,
            jitter: 0.25,
        }
    }

    /// Override the attempt cap. **Panics** if `n == 0`.
    pub fn with_max_attempts(mut self, n: u32) -> Self {
        assert!(n > 0, "RetryPolicy::with_max_attempts: n must be > 0; got 0");
        self.max_attempts = n;
        self
    }

    /// Bound total wall-clock across all attempts.
    pub fn with_overall_timeout(mut self, d: Duration) -> Self {
        self.overall_timeout = Some(d);
        self
    }

    /// Override backoff jitter as a fraction (e.g. 0.25 = ±25%).
    /// **Panics** if `j` is NaN, negative, or >= 1.0.
    pub fn with_jitter(mut self, j: f64) -> Self {
        assert!(
            j.is_finite() && j >= 0.0 && j < 1.0,
            "RetryPolicy::with_jitter: j must be finite, in [0.0, 1.0); got {j}"
        );
        self.jitter = j;
        self
    }

    /// Read-only access to `max_attempts` for tests / metrics.
    pub fn max_attempts(&self) -> u32 { self.max_attempts }
}

/// Description of the request being retried. Fields are private so
/// callers cannot construct an unsafe descriptor (e.g., POST without an
/// idempotency key). Use the typed constructors.
#[derive(Debug, Clone)]
pub struct RetryRequest {
    method: reqwest::Method,
    has_idempotency_key: bool,
}

/// Reason a `RetryRequest` cannot be constructed for the requested
/// method/key combination.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum RetryRejection {
    /// POST requires an idempotency key so the server-side cache can
    /// replay the first response on retry.
    #[error("POST without an Idempotency-Key is not retryable")]
    PostWithoutKey,
    /// PATCH/PUT are not retryable in this iteration — server-side
    /// idempotency middleware caches POST only.
    #[error("{0} is not retryable in this iteration (server-side idempotency cache covers POST only)")]
    MethodUnsupported(reqwest::Method),
}

impl RetryRequest {
    pub fn get() -> Self { Self { method: reqwest::Method::GET, has_idempotency_key: false } }
    pub fn head() -> Self { Self { method: reqwest::Method::HEAD, has_idempotency_key: false } }
    pub fn delete() -> Self { Self { method: reqwest::Method::DELETE, has_idempotency_key: false } }
    pub fn post_with_idempotency_key() -> Self {
        Self { method: reqwest::Method::POST, has_idempotency_key: true }
    }
    pub fn try_new(method: reqwest::Method, has_idempotency_key: bool) -> Result<Self, RetryRejection> {
        match method {
            reqwest::Method::GET | reqwest::Method::HEAD | reqwest::Method::DELETE => {
                Ok(Self { method, has_idempotency_key })
            }
            reqwest::Method::POST if has_idempotency_key => Ok(Self { method, has_idempotency_key }),
            reqwest::Method::POST => Err(RetryRejection::PostWithoutKey),
            other => Err(RetryRejection::MethodUnsupported(other)),
        }
    }
    pub fn method(&self) -> &reqwest::Method { &self.method }
    pub fn has_idempotency_key(&self) -> bool { self.has_idempotency_key }
}

/// Decision returned by [`classify`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RetryDecision {
    /// Outcome is retryable (transient failure).
    Retry,
    /// Outcome is terminal — return immediately.
    Stop,
}

/// Classify an attempt outcome from the typed `reqwest` shape.
pub fn classify(
    status: Option<reqwest::StatusCode>,
    err: Option<&reqwest::Error>,
) -> RetryDecision {
    if let Some(e) = err {
        if e.is_connect() || e.is_timeout() { return RetryDecision::Retry; }
        return RetryDecision::Stop;
    }
    let Some(status) = status else { return RetryDecision::Stop; };
    if status.is_server_error()
        || status == reqwest::StatusCode::REQUEST_TIMEOUT
        || status == reqwest::StatusCode::TOO_MANY_REQUESTS
    {
        RetryDecision::Retry
    } else {
        RetryDecision::Stop
    }
}

/// Strongly-typed retry error.
#[derive(Debug, thiserror::Error)]
pub enum RetryError {
    /// Hit `max_attempts` with the last attempt also failing.
    #[error("retry exhausted after {attempts} attempts (last status={last_status:?})")]
    Exhausted {
        attempts: u32,
        last_status: Option<reqwest::StatusCode>,
        last_error: Option<String>,
    },
    /// `overall_timeout` elapsed before the loop completed.
    #[error("retry overall timeout after {elapsed:?}")]
    OverallTimeout { elapsed: Duration },
    /// Per-attempt budget exceeded AND no further retry was possible.
    #[error("retry per-attempt timeout after {elapsed:?}")]
    AttemptTimeout { elapsed: Duration },
    /// Non-retryable transport error wrapping the typed reqwest::Error.
    #[error("network error: {0}")]
    Network(#[source] reqwest::Error),
}

/// Run a request with bounded retry. See module docs.
pub async fn retry_request<F, Fut>(
    policy: &RetryPolicy,
    request: &RetryRequest,
    make_req: F,
) -> Result<reqwest::Response, RetryError>
where
    F: Fn() -> Fut,
    Fut: std::future::Future<Output = Result<reqwest::Response, reqwest::Error>>,
{
    // ... full implementation per the design ...
    unimplemented!() // filled in during Step 3
}

The full implementation is sketched in the scratchpad and lands in Step 3.

Critical files

  • docs/modules/ROOT/pages/plans/canopy-api-retry-middleware.adoc (this file, NEW in Step 1)

  • docs/modules/ROOT/nav.adoc (+ nav entry, Step 1)

  • crates/canopy-api/src/retry.rs (NEW, Step 3)

  • crates/canopy-api/src/lib.rs (+ pub mod retry; + re-exports, Step 4)

  • crates/canopy-api/Cargo.toml (+ rand, + thiserror, Step 4)

  • crates/canopy-test-lib/src/client.rs (Step 5)

  • crates/canopy-test-lib/Cargo.toml (+ canopy-api, Step 6)

  • crates/canopy-test-lib/src/clients/eligibility.rs (Step 7)

  • crates/canopy-test-lib/tests/evil_proxy_test.rs:61 (Step 8)

  • services/canopy-eligibility/src/orchestrator.rs:580-618 (Step 10)

  • CHANGELOG.adoc (Step 11)

  • Shared Crates (Step 12)

  • Testing (Step 12)

Verification

  1. cargo nextest run -p canopy-api retry — 6 unit tests pass.

  2. cargo nextest run -p canopy-mq — 44/44 still pass.

  3. cargo nextest run -p canopy-eligibility --profile integration — 56/56 pass.

  4. cargo nextest run -p canopy-test-lib --run-ignored only inbox_dedup_at_100_percent_failure — passes.

  5. cargo build -p canopy-test-lib — no cycle.

  6. cargo fmt --all — --check + cargo clippy --workspace --tests — -D warnings — clean.

  7. cargo xtask validate — 1734/1734 + e2e + docker, exit 0.

Project-specific gotchas

  • SPDX header on new retry.rs.

  • #![warn(missing_docs)] in canopy-api: every pub symbol needs a /// doc.

  • Clippy -D warnings with zero #[allow] carve-outs.

  • nextest only.

  • No Q1-Q8 in commit messages.

  • Commit title ≤72 chars, prefix ^(feat|fix|chore|refactor|docs|test|ci):.

  • rand 0.9 API: rand::rng().random_range(…​).

Out of scope (epic children)

  • JWKS chaos contract (child B).

  • Outbox chaos contract (child C).

  • Cross-process chaos observability harness (child A).

  • Durable chaos docs (child D).

  • PATCH/PUT retry — needs server-side idempotency cache extension.

  • Server-side retry of inbound requests.

Reuses existing patterns

  • canopy_mq::ConnectionManager::reconnect (crates/canopy-mq/src/connection.rs:93-132).

  • canopy_api::circuit_breaker::CircuitBreaker::record_failure (crates/canopy-api/src/circuit_breaker.rs:60-100) — target: emit shape.

  • canopy_api::idempotency (crates/canopy-api/src/idempotency.rs:343-364, 508-523).

  • uuid::Uuid::now_v7().

  • canopy_test_lib::mock::spawn_router.

  • rand::rng().random_range(…​) per crates/canopy-test-lib/src/evil.rs:207.

  • TestResponse shape + panic-on-transport-error idiom per client.rs:46-50, 250.

CHANGELOG entry template (lands in Step 11)

* *Bounded retry middleware for outbound HTTP + target: "retry"
  observability span (#462 narrowed).* See CHANGELOG.adoc when Step 11
  lands — full multi-paragraph entry.
Edit this page · default