Plan: Signing-key-aware service-token acquisition (self-heal on rotated/deleted key)

On this page

Status

Step Description Status

1

Plan .adoc + ADR-037 + nav + architecture index (#1036)

Done (2026-07-12) — !821

2

C1 — JWKS refresh hardening: single-flight (serialized fetch) + ensure_fresh + for_self_validation + typed validate_current (#1037)

Done (2026-07-12) — #1037

3

C2 — ServiceTokenSource bounded-freshness revalidation + canopy-api bootstrap wiring; closes the live determine-500 (#1038)

Done (2026-07-12) — #1038

4

C3 — canopy-portal service-token self-validation wiring (#1039)

Done (2026-07-12) — #1039

5

C4 — canopy-reporting: resolve token per bounded window during assembly (#1040)

Done (2026-07-12) — #1040

6

C5 — canopy-medicaid ELE scheduler: per-bounded-batch resolve, preserve systemic abort (#1041)

Done (2026-07-12) — #1041

7

Reporting error-swallow fix (independent root cause; prerequisite for C4) (#1042)

Done (2026-07-12) — #1042

Epic: &70
Issues: #1036, #1037, #1038, #1039, #1040, #1041, #1042
Branches: feature/1036-service-token-key-aware-plan, then one feature/<n>-… per child
Relates to: #610 (the seed-profile-brittleness facet historically bundled with this symptom; this epic does not close it)

Context

ServiceTokenSource::current() (crates/canopy-auth/src/service_token.rs) serves a cached client_credentials token while expires_at > now() — a TTL-only check that never revalidates the signature. A token’s true validity is TTL-valid AND signed by a key the issuer still publishes.

When Keycloak rotates then deletes an OIDC token-signing key, the cached token stays TTL-valid but is signed by a kid the issuer no longer publishes. Receivers reject it (JwksProvider::validate_token → 401). The live symptom: WIC / caps / medicaid / tanf POST /v1/determine → canopy-rules-client → 500, lasting from the key deletion until the sender’s next proactive re-mint (sources re-mint ~5 min before expiry; dev Keycloak TTL is 1800s, so the window is up to ~25 min).

This is the IdP token-signing key system (the JWKS the services fetch from canopy-identity / Keycloak), which is distinct from canopy’s own determination-JWS verification keys retained forever in signing_key_history (ADR-036 §6). ADR-036 fixed the receiver keeping old canopy determination-signing keys verifiable; this plan fixes the sender holding a service token whose IdP signing key is gone.

A naive kid-membership check does not work: a sender’s JWKS cache can hold a stale {old, new} superset; after old is deleted, inbound tokens use new (a cache hit), so inbound validation never force-refreshes, and a membership check would read old as still-present and never detect the deletion. Detection requires an authoritatively fresh JWKS.

Manual recovery today is cargo xtask dev reload (restarts the sender, clearing its cache), documented in runbooks/jwks-stale-recovery.adoc — it does recover this bug, but it is a whole-stack bounce and that runbook is written for the receiver-stale-JWKS symptom.

Scope

In scope:

  • Bounded-freshness full revalidation of a cached service token against the receiver’s own validate_token, force-refreshing the JWKS within a config-overridable max-age M (default 60s).

  • Hardening JwksProvider refresh against the two existing races (clobber + split-debounce).

  • Opt-in wiring at the two production ServiceTokenSource construction sites (bootstrap, portal).

  • Adoption fixes for the two consumers that reuse a token beyond M (reporting, ELE scheduler).

  • ADR-037 + a sender-vs-receiver recovery runbook decision tree.

  • An independent reporting error-swallow bug that corrupts federal reports (distinct root cause).

Out of scope:

  • Any change to `validate_token’s validation semantics (blast radius must stay zero).

  • Extending cargo xtask identity verify (self-heal belongs in canopy-auth tests + the #480 chaos harness).

  • Operator ownership of credential rotation (ADR-019 unchanged on that axis).

  • .claude/CLAUDE.md status tables (status lives in Antora + GitLab).

Design

Decision: bounded-freshness full revalidation (opt-in)

A sender considers its cached token valid iff it would pass the receiver’s own validate_token (signature, kid, alg, iss, exp/nbf, aud, typ) against a JWKS force-refreshed within a max-age M (single-flighted). On a key/signature failure against a fresh JWKS it re-mints and revalidates the fresh candidate. This makes sender-validity ≡ receiver-validity, self-heals {old,new}→{new}, and needs no durable-kid contract.

Rejected alternatives (recorded in ADR-037): reactive 401-retry (ambiguous 401, replay, 2N sends, breaks deadlines, 57-site adoption); kid-membership (a weaker signal defeated by the stale {old,new} cache); operational-grace-period-only (a fig leaf that does not self-heal). M = 60s, config-overridable.

Back-compat / opt-in. Revalidation is a revalidation: Option<Revalidation> set only by a new with_self_validation(provider, max_age) builder. None ⇒ current() behaves exactly as today. Only the two production constructors call it; every new_for_tests fixture leaves it None ⇒ zero test churn, no bypass flag.

Pre-implementation gate (verified before C1)

validate_token rejects a present-but-non-"Bearer" typ and enforces exp with jsonwebtoken’s implicit 60s leeway (no explicit skew is set on the Validation). A real Keycloak client_credentials service token was dumped and confirmed to carry claim typ == "Bearer" and aud == "canopy-internal-service" — so a service accepts its own token (no fail-closed loop). C1 sets Validation.leeway explicitly rather than relying on the implicit default.

Design 1 — JWKS refresh hardening (crates/canopy-auth/src/jwks.rs)

Two real races exist today: all three refreshers (refresh, the periodic task, try_forced_refresh) write through one unguarded *self.keys.write() = Some(jwks) (a slow pre-deletion fetch can clobber a newer post-deletion set); and try_forced_refresh reads and writes its debounce timestamp under separate lock acquisitions (concurrent misses all fetch). Fix:

  • Replace keys: Arc<RwLock<Option<JwkSet>>> + last_forced_refresh with keys: Arc<RwLock<KeyCache { jwks, generation: u64, last_success: Option<Instant>, last_attempt: Option<Instant> }>> + refresh_lock: Arc<tokio::sync::Mutex<()>>. Generation lives inside the keyset lock (no tear vs the keyset).

  • One private refresh_single_flight(mode) (mirrors discovery.rs): a pre-lock fast path returns AlreadyFresh for a MaxAge caller already within max-age (no lock taken); otherwise snapshot generation → take refresh_lock → re-check under lock (fast-out / coalesce if generation advanced or the debounce window forbids) → set last_attempt → fetch (keyset lock not held, refresh_lock held) → install + bump generation + set last_success. The refresh_lock is held across the fetch, so writes are serialized and a slow fetch cannot clobber a newer set (this alone fixes the clobber race — no write-time compare-and-swap is needed; the generation counter’s job is change-detection for the coalescing re-check, not a write guard). Typed enum RefreshOutcome { Fresh, AlreadyFresh, Debounced, InFlightCoalesced, Failed(RefreshError) } (RefreshError carries String, so the outcome is Clone).

  • pub async fn ensure_fresh(max_age) → RefreshOutcome = MaxAge mode (force iff last_success older than max-age; never returns Debounced → the fail-open/closed decision keys cleanly off Failed).

  • pub fn for_self_validation(aud) → Self = a sibling sharing the keyset Arc + refresh_lock with expected_audiences = [aud] (audience is per-instance data independent of the shared keyset — one refresh serves both).

  • pub(crate) async fn validate_current(token) → Result<Claims, TokenValidationError> (typed is_key_or_signature(), is_expired(); no internal refresh). (generation() stays #[cfg(test)] — the service-token path keys off RefreshOutcome::Failed + validate_current, not the counter.)

  • Preserve signatures: refresh() → Result<(), anyhow> and try_forced_refresh() → bool wrap the new primitive; validate_token() unchanged (its kid-miss forced-refresh path stays); inject_keys writes a KeyCache. Blast radius of validate_token callers is unchanged.

Design 2 — ServiceTokenSource (crates/canopy-auth/src/service_token.rs)

  • New: revalidation: Option<Revalidation { provider: JwksProvider, max_age: Duration, acquire_deadline: Duration }> (opt-in builder with_self_validation; acquire_deadline defaults to ACQUIRE_DEADLINE, overridable in tests); mint_lock: tokio::sync::Mutex<MintState { cooldown_until, backoff, last_class }> (single-flight mint + cooldown, with AcquireFailClass { Revoked, Unavailable }); CachedToken is unchanged (access_token + expires_at — no generation field); ServiceTokenError::{ KeyRevoked { reason, retry_after }, MalformedToken { detail }, AcquisitionTimeout }. M is not a const — it is the config default settings.oidc_service_token_revalidate_max_age_secs (60s). Consts: ACQUIRE_DEADLINE = 15s, cooldown COOLDOWN_BASE 5s → COOLDOWN_MAX 60s backoff + jitter, and the mint→revalidate retry bounds REVALIDATE_RETRY_BASE = 200ms → REVALIDATE_RETRY_MAX = 2s (capped by ACQUIRE_DEADLINE).

  • current() (signature unchanged): fast-path read-clone-drop (preserve the non-blocking shape); if revalidation == None ⇒ return cached (legacy); else provider.ensure_fresh(max_age) → recheck token-identity/expiry after the await (a concurrent acquisition may have installed a newer token — serve it if access_token changed and it is unexpired) → validate_current(cached):

    • Ok ⇒ serve.

    • Err key/signature and ensure_fresh returned Failed (JWKS unreachable) ⇒ fail-open (serve cached, warn! — re-mint would also fail; breaking all outbound calls on a JWKS blip is worse).

    • Err key/signature and the JWKS was fresh ⇒ fail-closed → acquire_and_revalidate.

    • Err expired ⇒ acquire_and_revalidate (the same path as an absent/cold cache). Err other (aud/iss/typ) ⇒ MalformedToken (a config bug, surfaced not masked).

  • acquire_and_revalidate: single-flight under mint_lock (re-check the cache under the lock and coalesce onto a mint another caller just completed) → mint → ensure_fresh → revalidate the fresh candidate; on a key/signature failure (a lagging token-endpoint node minting under a not-yet-published kid) retry with bounded exponential backoff (no jitter — mint_lock already single-flights the retries) until acquire_deadline, then KeyRevoked { reason, retry_after }. Cooldown is explicit: a known-dead caller in Revoked cooldown gets immediate KeyRevoked { retry_after } (never serves the dead token, never hammers the endpoint); a cold caller in Unavailable cooldown gets NoToken (preserves today’s contract).

  • new_for_tests unchanged (leaves revalidation = None ⇒ legacy ⇒ returns the injected literal) — all 29 downstream fixtures pass untouched.

Lock order (acyclic): L_MINT ≺ L_REFRESH ≺ L_KEYS, L_MINT ≺ L_CACHED; L_KEYS / L_CACHED are leaves, never held across a network .await; L_REFRESH / L_MINT are held across their fetch/mint (the single-flight guarantee, sanctioned tokio-Mutex-across-await), and the validate/read paths never take them.

Wiring (only two production new() sites)

  • Bootstrap (crates/canopy-api/src/bootstrap.rs): svc_jwks = jwks.for_self_validation("canopy-internal-service") — a sibling sharing the inbound provider’s already-warmed keyset + refresh task, scoped to the service audience — then ServiceTokenSource::new(…​).with_self_validation(svc_jwks, max_age), where max_age is settings.oidc_service_token_revalidate_max_age_secs. Sharing the warmed keyset avoids a second warm
    refresh task and sidesteps the AuthLayer::new(jwks) move. canopy-web needs no change — it takes boot.service_token_source.

  • Portal (services/canopy-portal/src/main.rs, bypasses bootstrap): it already builds discovery; build a dedicated svc_jwks = JwksProvider::from_discovery_with_client(&discovery, http.clone())?.with_audience("canopy-internal-service"), warm it (refresh().await) + start_refresh_task, and pass it to .with_self_validation(svc_jwks, max_age). Preserve fail-closed startup — if the provider cannot build/warm, do not mount applicant routes (readyz stays 503).

Adoption inventory (rule: resolve current() per send or per bounded batch < M)

The redesign auto-heals every caller that re-resolves within M of its sends — verified: canopy-web (per-request), the orchestrator (per-determination), portal (per-request), the SNAP/caps/tanf/wic/medicaid/ enrollment /determine handlers (per-request), the source-holding clients (per-call), signing-registration (per-attempt), ELE event handlers (per-event). Exactly two reuse a token beyond M and need change:

Consumer Problem Fix

canopy-medicaid ELE scheduler (scheduler.rs)

one token per statewide tick, reused across thousands of rows over minutes (≫ M)

re-resolve via a per-row time gate (ensure_ele_token_fresh, called before each row in both loops; re-resolves current() only when elapsed ≥ ELE_TOKEN_REFRESH_INTERVAL = 30s ≈ M/2). Preserve systemic abort: a token-acquisition failure ?-propagates out of the advisory-lock closure and aborts the tick; only genuine per-row business failures increment out.errors.

canopy-reporting (clients/mod.rs scoped_source(source))

one scoped token reused across a whole report assembly; a large-roll CMS-416/ACF-199 can exceed M

hold a shared RefreshingToken over the ServiceTokenSource in the scoped clients; each send resolves a bearer that re-resolves current() once per TOKEN_REUSE_WINDOW = 30s (≈ M/2).

Independent bug (own issue #1042 — not the token fix)

canopy-reporting swallows upstream errors into empty data: get_person_income .or_else(|| Ok(Vec::new())) turns a 401 into zero income → a wrong 0% FPL in a federal report; get_tanf* .or(Ok(None)) at three sites. get_optional already shows the correct 404-only contract. Distinct from the token fix (which only reduces 401 frequency).

Fixing the swallow surfaced a second, latent data-integrity bug on the same path (folded into #1042 rather than deferred): the client’s IncomeRecord deserialized a non-existent monthly_amount: Decimal, so the T-MSIS FPL computation silently failed for every person carrying income (the failure was previously masked by the swallow). The wire shape is corrected to IncomeRecord { amount: Option<Decimal>, frequency: String } (crypto-shred aware — None when the money leaf was redacted, per ADR-036, skipped not zeroed) and each record is frequency-normalized to a monthly figure via canopy_reference::money::to_monthly (#861) using the federal snap-budgeting-factors.json before summing; the orphan-household case reads via get_household_optional (404 → Ok(None), not a hard error).

Steps

Step 1: Plan + ADR-037 (this MR, #1036)

Files: docs/modules/ROOT/pages/plans/service-token-key-aware-acquisition.adoc, docs/modules/ROOT/pages/adrs/adr-037-service-token-key-aware-acquisition.adoc, docs/modules/ROOT/nav.adoc, docs/modules/ROOT/pages/architecture.adoc.

Land this plan, ADR-037 (:status: Accepted, narrowly amending ADR-019), the nav entries (ADR-037 after adr-036; a * plan entry under Infrastructure), and the architecture ADR-index bullet — before any code MR. No code change.

Step 2: C1 — JWKS refresh hardening (#1037)

Files: crates/canopy-auth/src/jwks.rs.

Implement Design 1. Preserve refresh / try_forced_refresh / validate_token / inject_keys signatures + behavior. Set Validation.leeway explicitly. Tests: single-flight (N callers → one fetch); single-flight clobber-prevention; ensure_fresh max-age; validate_current typed outcomes; for_self_validation shares the keyset but validates its own audience.

Step 3: C2 — ServiceTokenSource revalidation + bootstrap (#1038)

Files: crates/canopy-auth/src/service_token.rs, crates/canopy-api/src/bootstrap.rs.

Implement Design 2 + bootstrap wiring. Closes the live determine-500. Tests per the Testing section, including the revalidation = None legacy guard.

Step 4: C3 — portal wiring (#1039)

Files: services/canopy-portal/src/main.rs.

Build a self-validating svc provider from the portal’s own discovery; preserve fail-closed startup.

Step 5: C4 — reporting adoption (#1040)

Files: services/canopy-reporting/src/clients/mod.rs, services/canopy-reporting/src/api/mod.rs.

Hold a shared RefreshingToken over the ServiceTokenSource in the scoped clients (scoped_source); each send resolves a bearer that re-resolves current() once per TOKEN_REUSE_WINDOW. Tests: bearer reuse within the window, re-resolve after the window, and acquisition-failure propagation (new_for_tests / an unreachable token source).

Step 6: C5 — ELE scheduler (#1041)

Files: services/canopy-medicaid/src/scheduler.rs.

Per-row time-gated resolve (ensure_ele_token_fresh); preserve systemic ?-abort. Tests: reuse within the window (source untouched), re-resolve after the window (token replaced), and a token-outage acquisition failure aborts the tick (Err propagates) without inflating out.errors.

Step 7: Reporting error-swallow + income wire-shape (#1042)

Files: services/canopy-reporting/src/clients/mod.rs, services/canopy-reporting/src/reporting/medicaid.rs, services/canopy-reporting/Cargo.toml.

Replace the blanket .or_else/.or(Ok(None)) swallows with a 404-only-is-absence contract (mirror get_optional), and correct the IncomeRecord wire shape + frequency normalization the swallow was masking (see the Independent-bug design note). Tests: mock 401/5xx surfaces as an error (not empty data); a real 404 still maps to empty/None; the wire shape deserializes; redacted (None) amounts are skipped.

Testing

Fault-injection / recovery tests, named for the property, run under cargo nextest.

  • canopy-auth — hand-roll a mock /token and mock /certs via tokio::net::TcpListener
    axum::serve (do not add canopy-test-lib — it depends on canopy-auth → dev-dep cycle). Build cached JWTs with a chosen kid via jsonwebtoken::encode. Add an injectable clock (or a tokio test-util dev-feature) for cooldown/max-age timing — no real sleeps.

    • Acceptance (the real bug): sender + receiver caches start {old, new}; the mock issuer changes to {new} only; assert the sender detects within M and re-mints (separate caches — not "inject a cache already missing old").

    • JWKS: refresh_single_flight_coalesces_concurrent_callers (barrier N callers → one fetch, folding in the anti-clobber property); ensure_fresh max-age + Failed-when-unreachable; validate_current typed outcomes; for_self_validation shares the keyset but validates its own audience.

    • Source: kid-deleted → re-mint; serves-valid-cached without re-mint; JWKS-unreachable → fail-open; fresh-JWKS-invalid → fail-closed KeyRevoked; JWKS-unreachable → AcquisitionTimeout; cooldown when the endpoint is unavailable; bounded revalidate backoff; revalidation = None legacy path unchanged (guards the fixtures).

  • C4/C5 — no mock-JWKS rotation; the adoption fixes are exercised at the reuse-window boundary with new_for_tests / an unreachable token source. Reporting: RefreshingToken reuse within the window, re-resolve after it, and acquisition-failure propagation. Scheduler: ensure_ele_token_fresh reuse / re-resolve across the window, plus the token-outage-aborts-the-tick test.

  • Observability — assert state-transition events; rate-limit/aggregate fail_open / cooldown warns (no per-send flood).

Docs, budgets, verification

  • Docs (bundled with the behavior MRs): ADR-037 (amends ADR-019’s JWKS-staleness mitigation to the sender side; preserves operator ownership of credential rotation; states the M-bounded detection latency + fail-open/closed); a new/updated runbook with a sender-stale-token vs receiver-stale-JWKS decision tree, event fields, cooldown/fail-open, and dev reload as valid manual recovery; shared-crates.adoc (add the ServiceTokenSource bullet); idp-integration.adoc (a NOTE only — no contract change); correct roadmap.adoc to separate this sender-auth work from #610’s seed-profile debt; per-child CHANGELOG.adoc under == Unreleased.

  • Budgets: no serde_json::Value added (the scanner counts the literal type, not json!). New public items (RefreshOutcome, ensure_fresh, for_self_validation, with_self_validation, the new ServiceTokenError variants) need doc comments.

  • Verification per MR: cargo fmt --all → cargo clippy -p <crates touched> --all-targets --profile test — -D warnings → cargo xtask quality-budgets --fail-on-regression → targeted cargo nextest run -p <crate>. The full pre-push hook (8 stages) is the merge gate; the bug is unit-reproducible against a mock issuer (no dev reload needed for acceptance).

  • Completion: on the final MR, flip this Status table to Done (YYYY-MM-DD), move the nav entry to plans/archive/, and run the Plan Completion Audit.

Edit this page · default