Plan: retry observability contract (#462 narrowed)
On this page
Context — the larger picture
The canopy-test-lib port’s Phase C (MR !318, refs #436) landed four chaos tests in crates/canopy-test-lib/tests/evil_proxy_test.rs. Each chaos test points a typed client at an EvilLayer proxy and asserts a tracing::Event target fires in production code.
Architectural discovery during this planning cycle: SpanCapture::install_scoped (crates/canopy-test-lib/src/observability.rs:120-130) uses tracing::subscriber::set_default, which is thread-local in the test process. Three of the four chaos tests assert on events emitted by code running in different processes (devstack containers for canopy-auth’s JWKS refresh, canopy-mq’s outbox drainer inside services). SpanCapture cannot see those events. Only the retry chaos test (inbox_dedup_at_100_percent_failure at evil_proxy_test.rs:39-82) drives in-process code paths — the typed client / TestClient executes IN the test process, so any retry middleware wired there IS visible to SpanCapture.
Revised structure (split into a GitLab epic + shippable children, per GitLab Workflow requirements):
-
EPIC (new GitLab group-level epic): titled "Chaos observability contracts: cross-process capture + retry/JWKS/outbox". Filed via
glab api groups/gadhs%2Fapplication%2Feligibility/epics -X POST -f title=…. Children linked viaepic_idAPI field per gitlab-workflow.md "Issue-epic linking" — NOT description-only cross-references. -
#462 (renamed via
glab issue update 462 --title "…"): "Retry observability contract". Linked to the new epic viaepic_id. This plan implements #462 in its narrowed form. -
Child A: "Cross-process chaos observability harness" — decide and build the test strategy (in-process production fixtures vs OTEL export vs log-capture).
-
Child B: "JWKS chaos contract" — rewrite
jwks_rotationatevil_proxy_test.rs:140-176to driveJwksProviderbehavior via the chosen harness + addtarget: "jwks"to the 4 emit sites incrates/canopy-auth/src/jwks.rs(81, 98, 212, 216). -
Child C: "Outbox chaos contract" — rewrite
outbox_catches_upatevil_proxy_test.rs:184-220to create an actual outbox failure-then-recovery path + addtarget: "outbox"to the 4 drain-retry emit sites incrates/canopy-mq/src/outbox_drainer.rs(213, 228, 367, 405). -
Child D: "Durable chaos docs + runbook" — update Testing (SpanCapture-is-thread-local lesson) + Shared Crates.
This plan is scoped exclusively to the narrowed #462 (retry only). The epic + other children get filed in Step 2 (after Step 1 lands this canonical plan file so the epic description can include a working clickable plan URL).
Why a retry layer is justified beyond closing the chaos test:
-
The orchestrator’s program-service dispatch (
services/canopy-eligibility/src/orchestrator.rs:600-618) is one-shot — a transient 503 from canopy-snap fails the entire SNAP determination. -
canopy-test-lib::TestClientis one-shot — devstack-flake-induced test failures bubble through to E2E assertions. A bounded retry kills a class of test flakes. -
The server-side idempotency middleware (
crates/canopy-api/src/idempotency.rs:343-364) was built specifically to make client-side retries safe — caching first responses keyed onIdempotency-Keyheader — but no client today emits the key or retries.
Status
| Step | Description | Status |
|---|---|---|
1 |
First implementation action — create the canonical plan file. Copy this file into |
In progress |
2 |
GitLab admin (after Step 1 lands the canonical plan file). Step 1 must complete first so the epic description below can include a working clickable plan URL. (a) Create the epic at the group level via |
Not started |
3 |
|
Not started |
4 |
|
Not started |
5 |
|
Not started |
6 |
|
Not started |
7 |
|
Not started |
8 |
|
Not started |
9 |
6 unit tests in |
Not started |
10 |
|
Not started |
11 |
|
Not started |
12 |
Docs: Shared Crates (new |
Not started |
13 |
Precommit Q1-Q8 + validate + push. |
Not started |
Issue: https://gitlab.com/gadhs/application/eligibility/canopy/-/issues/462 (renamed in Step 2)
Epic: TBD — created in Step 2
Branch: feature/canopy-api-retry-middleware (per Git Workflow — type::feature uses feature/ not fix/)
Labels: priority::medium, service::shared-crates, program::infrastructure, type::feature, workflow::ready
Design
retry.rs API surface
// SPDX-License-Identifier: AGPL-3.0-or-later
//! Bounded exponential-backoff retry for outbound reqwest calls.
//!
//! Coordinates with the server-side idempotency middleware
//! (`canopy_api::idempotency_middleware`) so retried POSTs replay
//! safely. Emits `tracing::info!(target: "retry", ...)` before each
//! retry attempt — operators grep this target in canopy_logs to
//! correlate transient downstream blips with the runbook.
use std::time::Duration;
const BACKOFF_INITIAL_MS: u64 = 100;
const BACKOFF_MAX_MS: u64 = 30_000;
/// Tunables for retry behavior. Fields are private; construct via
/// [`RetryPolicy::default_http`] and refine via the `with_*` setters,
/// which assert invariants (no zero `max_attempts`, no NaN/negative
/// `jitter`).
#[derive(Debug, Clone)]
pub struct RetryPolicy {
max_attempts: u32,
initial_backoff: Duration,
max_backoff: Duration,
overall_timeout: Option<Duration>,
jitter: f64,
}
impl RetryPolicy {
/// HTTP-typical defaults: 3 attempts, 100ms → 30s backoff, ±25%
/// jitter, no overall_timeout.
pub const fn default_http() -> Self {
Self {
max_attempts: 3,
initial_backoff: Duration::from_millis(BACKOFF_INITIAL_MS),
max_backoff: Duration::from_millis(BACKOFF_MAX_MS),
overall_timeout: None,
jitter: 0.25,
}
}
/// Override the attempt cap. **Panics** if `n == 0`.
pub fn with_max_attempts(mut self, n: u32) -> Self {
assert!(n > 0, "RetryPolicy::with_max_attempts: n must be > 0; got 0");
self.max_attempts = n;
self
}
/// Bound total wall-clock across all attempts.
pub fn with_overall_timeout(mut self, d: Duration) -> Self {
self.overall_timeout = Some(d);
self
}
/// Override backoff jitter as a fraction (e.g. 0.25 = ±25%).
/// **Panics** if `j` is NaN, negative, or >= 1.0.
pub fn with_jitter(mut self, j: f64) -> Self {
assert!(
j.is_finite() && j >= 0.0 && j < 1.0,
"RetryPolicy::with_jitter: j must be finite, in [0.0, 1.0); got {j}"
);
self.jitter = j;
self
}
/// Read-only access to `max_attempts` for tests / metrics.
pub fn max_attempts(&self) -> u32 { self.max_attempts }
}
/// Description of the request being retried. Fields are private so
/// callers cannot construct an unsafe descriptor (e.g., POST without an
/// idempotency key). Use the typed constructors.
#[derive(Debug, Clone)]
pub struct RetryRequest {
method: reqwest::Method,
has_idempotency_key: bool,
}
/// Reason a `RetryRequest` cannot be constructed for the requested
/// method/key combination.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
pub enum RetryRejection {
/// POST requires an idempotency key so the server-side cache can
/// replay the first response on retry.
#[error("POST without an Idempotency-Key is not retryable")]
PostWithoutKey,
/// PATCH/PUT are not retryable in this iteration — server-side
/// idempotency middleware caches POST only.
#[error("{0} is not retryable in this iteration (server-side idempotency cache covers POST only)")]
MethodUnsupported(reqwest::Method),
}
impl RetryRequest {
pub fn get() -> Self { Self { method: reqwest::Method::GET, has_idempotency_key: false } }
pub fn head() -> Self { Self { method: reqwest::Method::HEAD, has_idempotency_key: false } }
pub fn delete() -> Self { Self { method: reqwest::Method::DELETE, has_idempotency_key: false } }
pub fn post_with_idempotency_key() -> Self {
Self { method: reqwest::Method::POST, has_idempotency_key: true }
}
pub fn try_new(method: reqwest::Method, has_idempotency_key: bool) -> Result<Self, RetryRejection> {
match method {
reqwest::Method::GET | reqwest::Method::HEAD | reqwest::Method::DELETE => {
Ok(Self { method, has_idempotency_key })
}
reqwest::Method::POST if has_idempotency_key => Ok(Self { method, has_idempotency_key }),
reqwest::Method::POST => Err(RetryRejection::PostWithoutKey),
other => Err(RetryRejection::MethodUnsupported(other)),
}
}
pub fn method(&self) -> &reqwest::Method { &self.method }
pub fn has_idempotency_key(&self) -> bool { self.has_idempotency_key }
}
/// Decision returned by [`classify`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RetryDecision {
/// Outcome is retryable (transient failure).
Retry,
/// Outcome is terminal — return immediately.
Stop,
}
/// Classify an attempt outcome from the typed `reqwest` shape.
pub fn classify(
status: Option<reqwest::StatusCode>,
err: Option<&reqwest::Error>,
) -> RetryDecision {
if let Some(e) = err {
if e.is_connect() || e.is_timeout() { return RetryDecision::Retry; }
return RetryDecision::Stop;
}
let Some(status) = status else { return RetryDecision::Stop; };
if status.is_server_error()
|| status == reqwest::StatusCode::REQUEST_TIMEOUT
|| status == reqwest::StatusCode::TOO_MANY_REQUESTS
{
RetryDecision::Retry
} else {
RetryDecision::Stop
}
}
/// Strongly-typed retry error.
#[derive(Debug, thiserror::Error)]
pub enum RetryError {
/// Hit `max_attempts` with the last attempt also failing.
#[error("retry exhausted after {attempts} attempts (last status={last_status:?})")]
Exhausted {
attempts: u32,
last_status: Option<reqwest::StatusCode>,
last_error: Option<String>,
},
/// `overall_timeout` elapsed before the loop completed.
#[error("retry overall timeout after {elapsed:?}")]
OverallTimeout { elapsed: Duration },
/// Per-attempt budget exceeded AND no further retry was possible.
#[error("retry per-attempt timeout after {elapsed:?}")]
AttemptTimeout { elapsed: Duration },
/// Non-retryable transport error wrapping the typed reqwest::Error.
#[error("network error: {0}")]
Network(#[source] reqwest::Error),
}
/// Run a request with bounded retry. See module docs.
pub async fn retry_request<F, Fut>(
policy: &RetryPolicy,
request: &RetryRequest,
make_req: F,
) -> Result<reqwest::Response, RetryError>
where
F: Fn() -> Fut,
Fut: std::future::Future<Output = Result<reqwest::Response, reqwest::Error>>,
{
// ... full implementation per the design ...
unimplemented!() // filled in during Step 3
}
The full implementation is sketched in the scratchpad and lands in Step 3.
Critical files
-
docs/modules/ROOT/pages/plans/canopy-api-retry-middleware.adoc(this file, NEW in Step 1) -
docs/modules/ROOT/nav.adoc(+ nav entry, Step 1) -
crates/canopy-api/src/retry.rs(NEW, Step 3) -
crates/canopy-api/src/lib.rs(+pub mod retry;+ re-exports, Step 4) -
crates/canopy-api/Cargo.toml(+rand, +thiserror, Step 4) -
crates/canopy-test-lib/src/client.rs(Step 5) -
crates/canopy-test-lib/Cargo.toml(+canopy-api, Step 6) -
crates/canopy-test-lib/src/clients/eligibility.rs(Step 7) -
crates/canopy-test-lib/tests/evil_proxy_test.rs:61(Step 8) -
services/canopy-eligibility/src/orchestrator.rs:580-618(Step 10) -
CHANGELOG.adoc(Step 11) -
Shared Crates (Step 12)
-
Testing (Step 12)
Verification
-
cargo nextest run -p canopy-api retry— 6 unit tests pass. -
cargo nextest run -p canopy-mq— 44/44 still pass. -
cargo nextest run -p canopy-eligibility --profile integration— 56/56 pass. -
cargo nextest run -p canopy-test-lib --run-ignored only inbox_dedup_at_100_percent_failure— passes. -
cargo build -p canopy-test-lib— no cycle. -
cargo fmt --all — --check+cargo clippy --workspace --tests — -D warnings— clean. -
cargo xtask validate— 1734/1734 + e2e + docker, exit 0.
Project-specific gotchas
-
SPDX header on new
retry.rs. -
#![warn(missing_docs)]in canopy-api: everypubsymbol needs a///doc. -
Clippy
-D warningswith zero#[allow]carve-outs. -
nextest only.
-
No Q1-Q8 in commit messages.
-
Commit title ≤72 chars, prefix
^(feat|fix|chore|refactor|docs|test|ci):. -
rand 0.9 API:
rand::rng().random_range(…).
Out of scope (epic children)
-
JWKS chaos contract (child B).
-
Outbox chaos contract (child C).
-
Cross-process chaos observability harness (child A).
-
Durable chaos docs (child D).
-
PATCH/PUT retry — needs server-side idempotency cache extension.
-
Server-side retry of inbound requests.
Reuses existing patterns
-
canopy_mq::ConnectionManager::reconnect(crates/canopy-mq/src/connection.rs:93-132). -
canopy_api::circuit_breaker::CircuitBreaker::record_failure(crates/canopy-api/src/circuit_breaker.rs:60-100) —target:emit shape. -
canopy_api::idempotency(crates/canopy-api/src/idempotency.rs:343-364, 508-523). -
uuid::Uuid::now_v7(). -
canopy_test_lib::mock::spawn_router. -
rand::rng().random_range(…)percrates/canopy-test-lib/src/evil.rs:207. -
TestResponseshape + panic-on-transport-error idiom perclient.rs:46-50, 250.
CHANGELOG entry template (lands in Step 11)
* *Bounded retry middleware for outbound HTTP + target: "retry"
observability span (#462 narrowed).* See CHANGELOG.adoc when Step 11
lands — full multi-paragraph entry.