Stale JWKS Recovery: blanket 401s from long-running program services

On this page

Overview

Program services cache Keycloak’s JWKS (JSON Web Key Set) at startup. If the canopy-keycloak container rotates its signing keys after those services started — which happens after several cargo xtask dev reload cycles, or on clock-drift / TTL mismatch — tokens signed by the current Keycloak are rejected by the stale-cache services. The result is a blanket 401 Unauthorized: invalid token on every upstream call the worker BFF makes.

This is devstack hygiene that decays after the containers have been up for hours, not a code regression. Recognising it quickly saves a long false hunt through session / middleware code.

Which failure is this? (sender-stale-token vs receiver-stale-JWKS)

Two distinct staleness failures look similar (401s) but have different owners since ADR-037:

Receiver-stale JWKS (this runbook) Sender-stale token (ADR-037)

What is stale

A receiving service’s cached JWKS lacks the kid a valid current token was signed with.

A sending service’s cached client_credentials token was signed by a kid the IdP has since deleted.

Who 401s

The receiver rejects a token that is actually fine.

Every receiver correctly rejects the sender’s dead token.

Self-heals?

The receiver force-refreshes on an unknown kid (30s debounce), so this is usually transient; a persistent case is the devstack-hygiene one below.

Yes, automatically for services wired via canopy-api bootstrap (with_self_validation) — they revalidate the cached token against a bounded-fresh JWKS (default M = 60s, oidc_service_token_revalidate_max_age_secs) and re-mint; worst-case detection latency is 2·M + one JWKS fetch (the #1212 revalidation-verdict window plus the JWKS age behind it), no operator action needed. canopy-portal is wired the same way (#1039).

Manual recovery

cargo xtask dev reload (restarts the stale-cache service). Still valid.

cargo xtask dev reload still works (it restarts the sender, clearing its cache), but is rarely needed now — the sender self-heals within 2·M.

If a service builds a bespoke ServiceTokenSource::new without the bootstrap with_self_validation wiring, the sender does not self-heal; wire it per ADR-037. The rest of this runbook covers the receiver-side devstack-hygiene case.

Symptom

E2E specs that exercise the BFF’s program-service calls show empty data instead of seeded rows, or time out. Typical failures:

  • specs/wic.spec.ts — nutritional-risk tab shows the empty state instead of the seeded row.

  • specs/caps.spec.ts — authorization tab shows no provider row.

  • specs/case-search.spec.ts — clicking a result times out.

  • specs/dashboard.spec.ts — per-program cards don’t deep-link.

  • specs/panel-states.spec.ts — no-results state not visible.

Composition / spec-only routes that make no upstream call still pass — a useful disambiguator.

Root signal (read the logs first)

docker logs canopy-canopy-web-1 --tail 80

The tell is blanket 401s on every upstream service simultaneously:

failed to fetch household ... canopy-persons service error: HTTP 401 Unauthorized: invalid token
failed to fetch determinations from canopy-snap ... HTTP 401 Unauthorized: invalid token
failed to fetch recent activity ... canopy-security service error: HTTP 401 Unauthorized: invalid token
... (canopy-applications, canopy-notices, canopy-appeals, canopy-renewals all the same)

Multiple unrelated services rejecting the same forwarded token at once means it is a shared validation problem (stale keys), not a route-specific bug.

Fix

Always drive restarts through xtask — never raw docker compose restart <service> (see Why not docker compose restart <service>).

cargo xtask dev reload     # forces a coordinated bounce; every service refetches JWKS in dependency order
cargo xtask seed           # if you need fresh fixtures afterward
cargo xtask e2e            # full E2E run (let xtask refresh; don't pass --no-refresh)
NOTE

Prefer cargo xtask dev reload over cargo xtask dev refresh for this symptom. refresh auto-detects changes by content hash and may report "up to date" without bouncing services (the #609 SHA-gap), leaving the stale JWKS in place. reload forces the bounce.

Why not docker compose restart <service>

Partial restarts cascade. Restarting only the user-facing program services leaves canopy-rules / canopy-reporting / canopy-eligibility with stale JWKS, so the next run fails with a different 401 cluster (e.g. "rules engine returned error: 401" from a program service calling canopy-rules). Each bespoke restart produces a fresh 401 signature elsewhere in the dependency graph. cargo xtask dev reload restarts in the right order in one shot. (Witnessed 2026-05-22: three rounds of docker compose restart produced three different 401 clusters before the coordinated reload fixed it — 142/142 E2E green after.)

What NOT to do

  • Don’t blame the session refactor / lib+bin restructure / route_layer merge. The 401s are program-service-side, not BFF-side; the failure shape (auth-token-shaped) misleads when you haven’t read the logs.

  • Don’t restart canopy-web alone — the BFF is forwarding a valid token; the program services are rejecting it.

  • Don’t dismiss as "pre-existing flake, retry." The reload IS the fix; the diagnostic step is checking the canopy-web log for the multi-service 401 cluster.

Related

  • Dashboard E2E flake — a different structural flake with overlapping symptoms (empty dashboard); the disambiguator is whether the 401 cluster is in the logs.

  • Troubleshooting — general devstack recovery.

Edit this page · default