ADR-037: Signing-Key-Aware Service-Token Acquisition

A sender considers its cached token valid iff it would pass the receiver’s own validate_token (signature, kid, alg, iss, exp/nbf, aud, typ) against a JWKS force-refreshed within a max-age M (single-flighted). On a key/signature failure against a fresh JWKS, the sender re-mints and revalidates the fresh candidate before serving it. This makes sender-validity ≡ receiver-validity: if the sender serves a token, an in-sync receiver accepts it. Validity is the full validator, not kid-membership, so no durable-kid contract is introduced. M = 60s, config-overridable.

Revalidation runs at token-acquisition time (current()), not as a 401-retry after a failed call — so there is no ambiguous-401 problem (an application 401 for a bad passcode is not a token problem), no request replay, no doubling of outbound sends, and no interaction with per-call deadlines.

Verdict caching (#1212). A passing verdict is itself stamped on the cached token (validated_at) and honoured for M: inside the window current() serves with no RSA verify and no JWKS I/O — the original per-call full validation was the H5 scale defect (a duplicate RSA verify on every outbound call, fleet-wide). The staleness trade is explicit: worst-case detection of a deleted signing key is 2·M + one fetch, because the first post-window call both revalidates and triggers the JWKS refresh — at most one stale re-stamp (against the pre-deletion key set) can occur before fresh keys land and the next lapsed window fails closed. Still config-overridable via M, still orders of magnitude below the pre-ADR ~25-minute re-mint window.

Revalidation is a revalidation: Option<Revalidation { provider, max_age }> set only by a new with_self_validation(provider, max_age) builder on ServiceTokenSource. None ⇒ current() behaves exactly as before. Only the two production constructors (canopy-api bootstrap and canopy-portal) opt in; every new_for_tests fixture leaves it None. There is no bypass flag and no test churn — the legacy path is the absence of a provider, not a feature toggle.

JwksProvider refresh is hardened against two pre-existing races: a single-flight refresh_lock — held across the upstream fetch — collapses concurrent refreshes to one fetch and serializes installs, so a slow pre-deletion fetch cannot clobber a newer post-deletion keyset (no write-time compare-and-swap is needed). A generation counter inside the keyset lock is the change-detector for the coalescing re-check: a waiter whose pre-lock snapshot is stale but whose freshness bound is now satisfied adopts the just-installed keyset instead of re-fetching. Token minting is likewise single-flighted under a mint_lock with an explicit cooldown (5s → 60s backoff
jitter) so a known-dead caller neither serves the dead token nor hammers the token endpoint. The lock order L_MINT ≺ L_REFRESH ≺ L_KEYS (and L_MINT ≺ L_CACHED) is acyclic; leaf data-locks are never held across a network .await.

Serve-path freshness never blocks a caller (#1212). The original design had the serve path’s ensure_fresh wait on refresh_lock — so with a stale cache and a degraded IdP, every outbound call queued behind serial up-to-10s fetch attempts: a fleet-synchronized outbound convoy (scale audit H5). Since #1212 the serve path uses ensure_fresh_background: the freshness check try-locks; the winner spawns the fetch as a detached single-flighted task (the owned lock guard rides into it) and every caller — winner included — returns immediately (Pending), proceeding on cached material. A failure debounce (30s) makes a fetch failure short-circuit subsequent MaxAge attempts (blocking and background alike, checked both before and under the lock) to Failed without a probe, so a down IdP is probed at most once per window per process while everyone else fails open instantly. The acquisition path deliberately keeps the blocking ensure_fresh: the mint loop must observe the fetch result synchronously to distinguish revoked-against-fresh-keys from couldn’t-verify.

The design self-heals any caller that re-resolves current() within M of its sends. Callers that reuse a single resolved token beyond M must resolve per send or per bounded batch (< M). With the #1212 verdict cache, per-send resolution is a cache read (no crypto, no I/O inside the window), so the canopy-reporting scoped clients resolve on every request — their per-service RefreshingToken reuse-window workaround (#1040) is retired. The canopy-medicaid ELE scheduler resolves per bounded batch (#1219) and preserves systemic abort — a token-acquisition failure aborts the tick rather than degrading into thousands of per-row errors under a false Ok.