Signing Key Rotation

On this page

NOTE

T2-6 / ADR-036 removed the old dual-key (PREV) rotation window. Rotation is now a single-step deploy: there is no CANOPY_VERIFY_KEY{PROGRAM}_PREV env var and no 24–48h grace window. Each program service derives its signing kid from its public key and self-registers that public key into canopy-security’s signing_key_history on boot; the orchestrator’s verifier lazy-loads any since-retired key from there on a cache-miss. A determination signed with a rotated-out key therefore stays verifiable indefinitely, with no operator-managed previous-key window.

When to Rotate

Scheduled rotation

Rotate all program signing keys annually, aligned with the fiscal year boundary (October 1). Schedule rotation during a planned maintenance window.

Emergency rotation

Rotate immediately if any of the following occur:

Private key material is exposed (committed to VCS, leaked in logs, copied to unauthorized system).
Personnel with key access depart the organization without completing off-boarding.
A signing verification failure indicates potential key compromise.
IRS Pub 1075 or HIPAA audit findings require re-keying.

Prerequisites

Deployment access to the target environment (staging or production).
cargo xtask gen-signing-keys available.
Access to the environment-variable / secrets configuration for the target program service and for canopy-eligibility (the verifying orchestrator).
canopy-security reachable from the program service (so the new key registers on boot). If it is briefly unreachable, the program retries registration in the background and on the next boot — verification of the new key still works immediately via the in-memory current key.

Procedure (routine rotation)

The retention store makes rotation a single deploy — no dual-key window.

Step 1: Generate a new key pair

# Replace {program} with: snap, tanf, medicaid, caps, or wic
cargo xtask gen-signing-keys --program {program}

Produces .keys/{program}-private.pem (PKCS#8) and .keys/{program}-public.pem (SPKI).

CAUTION

Never commit private key files to version control. Store the private key in the target environment’s secrets manager immediately.

Step 2: Set the new keys

Variable Value

Variable	Value
`CANOPY_{PROGRAM}__SIGNING_KEY`	Contents of `.keys/{program}-private.pem` (new private key — the signer)
`CANOPY_VERIFY_KEY_{PROGRAM}`	Contents of `.keys/{program}-public.pem` (new public key — the current verifier key)

CANOPY_{PROGRAM}__SIGNING_KEY

Contents of .keys/{program}-private.pem (new private key — the signer)

CANOPY_VERIFY_KEY_{PROGRAM}

Contents of .keys/{program}-public.pem (new public key — the current verifier key)

Replace {PROGRAM} with the uppercase program name (e.g. SNAP). There is no _PREV variable to set.

Step 3: Rolling restart

Roll the program service and canopy-eligibility (to pick up the new current verifier key). On boot the program derives its new key-derived kid (canopy-{program}-{sha256(public)[..16]}) and registers the new public key into signing_key_history. New determinations are signed with the new key.

Step 4: Confirm registration (no PREV cleanup)

Confirm the new key is in the retention store, and that the old key is still present (so its determinations keep verifying):

# Service-authenticated; the JWKS lists every retained public key for the program.
curl -sf -H "Authorization: Bearer $SVC_TOKEN" \
  https://{security-host}/v1/security/signing-keys/{program}/jwks | jq '.keys[].kid'

You should see both the new kid and the prior kid(s). There is no PREV-removal step: determinations signed with the old key verify via the orchestrator’s lazy-load of the old key from signing_key_history (it was registered while the old key was active), so the old key is retained permanently.

Emergency rotation (compromised key)

Immediately set CANOPY_{PROGRAM}__SIGNING_KEY + CANOPY_VERIFY_KEY_{PROGRAM} to a fresh key pair and restart the program service + canopy-eligibility.
Audit all determinations signed during the compromise window.
To distrust the compromised key (so its existing signatures stop verifying), its signing_key_history row must be tombstoned in the blessed maintenance window AND it must not be the current CANOPY_VERIFY_KEY_{PROGRAM}. The store currently retains, never revokes — a dedicated key-revocation surface is a tracked follow-up.

Rollback

If the new key causes problems before it is widely used:

Restore the original CANOPY_{PROGRAM}__SIGNING_KEY + CANOPY_VERIFY_KEY_{PROGRAM} values.
Redeploy the program service and canopy-eligibility.

The old key was never removed from the retention store, so determinations signed with the new key during the brief window still verify (lazy-loaded from signing_key_history, where the new key registered on its boot). No re-keying cleanup is required.

Verification

# 1. Health of the program service + orchestrator
curl -sf https://{program-host}/readyz
curl -sf https://{eligibility-host}/readyz

# 2. The program's current + retained keys are in the JWKS (Step 4 above).

# 3. End-to-end signing: request a test determination and confirm its JWS
#    verifies (the orchestrator verifies every program determination it persists).
cargo xtask test

Confirm that:

Both readiness endpoints return 200.
The program’s JWKS lists the new kid (and retains the prior kid(s)).
No JWS verification errors appear in the service logs.
The full test suite passes.

Edit this page · default

Copy

View