Worker Program Scope — Cutover & Operations (#1515 / ADR-044)

On this page

ADR-044 makes the primary_programs claim a required worker authorization attribute. Before it, a token without the claim was admitted and treated as all programs; after it, such a token is refused at admission — at the OAuth callback and at the slow-path refresh alike. There is no canopy-side override and no role-tier bypass: the deployment’s override is the IdP claim mapper.

Two consequences drive this runbook:

  • a worker whose IdP entry lacks the claim cannot sign in after the deploy;

  • a legacy session row (missing or empty primary_programs) no longer deserializes on a new replica, so its holder is bounced to /login.

Both are fail-closed and both are avoidable by ordering the cutover correctly.

1. IdP inventory and backfill (BEFORE deploying)

For every configured IdP in rulesets/{juris}/idp.toml, enumerate the workers that lack a usable primary_programs value and backfill them. Two distinct defects to look for:

  • the mapper is missing — no worker on that provider will be admitted;

  • the mapper exists but the user attribute is unset — only some workers are refused, which is the harder case to spot.

Keycloak example (adapt per provider):

# Users on this realm with no primary_programs attribute.
kcadm.sh get users -r "$REALM" --fields id,username,attributes -q max=10000 \
  | jq -r '.[] | select((.attributes.primary_programs // []) | length == 0)
           | .username'

Backfill before the deploy, not during. A worker legitimately authorized for everything is granted all five slugs (snap, tanf, medicaid, caps, wic) — the claim mapper is the only place that decision is expressed. chip is accepted and canonicalizes to medicaid.

Privileged workers are not exempt: supervisors, jurisdiction admins, Studio admins, analysts and auditors need the claim exactly like caseworkers.

2. Token preflight (per provider)

The claim must ride the access token, and it must survive a refresh — a mapper configured only on the ID token, or only on the initial grant, produces a deployment where everyone signs in and then gets kicked out an access-token lifetime later.

For each provider, obtain a worker token, then exercise the rotation:

# 1) initial grant carries the claim on the ACCESS token
curl -s -d grant_type=password -d client_id="$CLIENT" \
     -d username="$USER" -d password="$PASS" \
     "$ISSUER/protocol/openid-connect/token" > /tmp/tok.json
jq -r .access_token /tmp/tok.json | cut -d. -f2 | base64 -d 2>/dev/null \
  | jq '.primary_programs'

# 2) the ROTATED token still carries it
curl -s -d grant_type=refresh_token -d client_id="$CLIENT" \
     -d refresh_token="$(jq -r .refresh_token /tmp/tok.json)" \
     "$ISSUER/protocol/openid-connect/token" \
  | jq -r .access_token | cut -d. -f2 | base64 -d 2>/dev/null \
  | jq '.primary_programs'

Both must print a non-empty array of recognized slugs. A null on step 2 with a value on step 1 is the classic "works until it doesn’t" misconfiguration.

3. Canary

Bring up one replica on the new build and watch the admission counters before rolling further:

  • canopy_web.auth.admission_rejected{idp,stage,reason} — the primary signal. stage=login + reason=missing_primary_programs concentrated on one idp means that provider’s mapper was never provisioned. stage=refresh means it stopped emitting for already-signed-in workers.

  • canopy_web.session.decode_failed — legacy session rows meeting a new replica. Expect a burst proportional to the live session count until step 4 is done; a burst that does not decay means the purge missed rows.

Roll forward only when the login rejection rate is at the expected floor (it is not necessarily zero — genuinely unprovisioned workers should be refused).

4. Legacy-session purge

Legacy session rows (missing or empty primary_programs) must not survive into the rollout window, where a still-old replica would re-admit one as an all-programs worker. Sessions live in PostgreSQL (ADR-009) in tower_sessions.session, canopy-web’s own database.

The blunt option is the recommended one. The payload column is bytea holding a MessagePack blob, so any surgical predicate is encoding-coupled; truncating costs every worker one sign-in, during a cutover in which the affected workers are signing in again anyway:

SELECT count(*) FROM tower_sessions.session;   -- how many people you interrupt
TRUNCATE tower_sessions.session;

If that disruption is unacceptable, the surgical form matches on the raw bytes. The worker payload is stored as a JSON-value map (tower-sessions converts via serde_json::to_value before the store’s MessagePack encode), and MessagePack writes map keys verbatim — so the literal primary_programs appears in the blob when the key is present, and an empty array is the single byte 0x90 immediately after it:

-- Inspect before deleting.
SELECT count(*) FROM tower_sessions.session
 WHERE position('primary_programs'::bytea in data) = 0                       -- key absent
    OR position('primary_programs'::bytea || '\x90'::bytea in data) > 0;     -- key present, value []

DELETE FROM tower_sessions.session
 WHERE position('primary_programs'::bytea in data) = 0
    OR position('primary_programs'::bytea || '\x90'::bytea in data) > 0;
CAUTION
That predicate depends on the store’s MessagePack encoding, not on a documented contract. Verify the counts look sane against the total before deleting, and prefer the truncate if they do not.
NOTE
The stored key is deliberately still primary_programs even though the Rust field is program_scope — the #[serde(rename)] is what makes an old replica read a new session correctly. Do not "fix" the name; renaming it would make an old replica see its field missing, default it to [], and re-grant see-all.

5. Drain old replicas

Complete the rollout only after the purge. Until every old replica is drained, the residual exposure is a legacy session reaching one of them — which is exactly the pre-#1515 behavior, no worse, but it is the one window the new build cannot close on its own.

6. Rollback

Rolling back to the pre-#1515 build is safe only with the purge already applied. Without it, the old build re-admits any surviving legacy session as an all-programs worker. Ordering:

  1. apply (or re-apply) the step-4 purge;

  2. deploy the old build;

  3. expect a wave of ordinary re-logins, not errors.

There is no partial rollback: the admission rule, the session schema and the scope type ship together (a half-applied version is the unsafe state).

7. Steady-state operations

Onboarding. Assigning primary_programs becomes part of worker provisioning. A new worker without it is refused with a banner naming the missing claim — the sign-in page does not loop back to the IdP, so the failure is legible to the worker and reportable to the service desk.

Break-glass. Granting a worker emergency cross-program access means granting the claim in the IdP. There is no canopy flag, and adding one would create a second source of truth for authorization scope (ADR-044).

Revocation delay. A scope change takes effect no later than one access-token lifetime, because the stored session scope is authoritative until the next refresh. When that is too slow — a revoked or compromised worker — purge that worker’s session rows, which forces re-admission on the next request:

-- :worker_sub is the OIDC `sub`, stored verbatim in the session payload.
DELETE FROM tower_sessions.session
 WHERE position(:worker_sub::bytea in data) > 0;

Diagnosing "a worker can’t sign in". The banner text names the class; the counter names the provider and the stage. no_role is #1024’s rule (no recognized realm role), missing_primary_programs is an absent or empty claim, malformed_primary_programs is a claim naming a program canopy does not recognize (check for a typo or a sixth program that needs a canopy change, not a mapper change).

Devstack

cargo xtask dev reimport-realm after editing devstack/keycloak/canopy-realm.json — Keycloak imports a realm only when it is absent, so a plain dev refresh leaves fixture claim edits unapplied. See Local Development.

Edit this page · default