Plan: canopy-identity contract + service-identity migration (Issue #424, ADR-019)

On this page

Status

Step Description Status

1

ADR-019 + this plan land as a standalone docs MR (docs/adr-019-service-identity). Two-file change so reviewers audit the architectural decision (canopy-identity as contract not service-container, Keycloak as one reference backend, hard cutover, xtask as dev/CI tooling not production ops plane) independently of the implementation. ADR-019 fixes the contract: required env vars, required OIDC discovery endpoints, required token shape, on-behalf-of via X-Canopy-Actor signed by canopy-signing, conformance via cargo xtask identity verify. Deployer-facing idp-integration.adoc updates land in Step 17 alongside the cutover, when the xtask commands actually exist for operators to use.

Not started

2

Claims API extensions in crates/canopy-auth/src/claims.rs. Add Claims::is_service(&self) → bool (true when the configured roles claim path contains an entry starting with CANOPY_IDENTITY_SERVICE_ROLE_PREFIX, default service:), Claims::service_id(&self) → Option<&str> (the matched role suffix, fallback to azp), Claims::require_service_caller(&self) → Result<(), ApiError> (Forbidden if not a service caller). Add pub actor: Option<Box<Claims>> field with #[serde(skip)] so it never round-trips through JWT serialisation — middleware lifts it in after validating X-Canopy-Actor. Add Claims::roles() lookup that reads from CANOPY_IDENTITY_ROLES_CLAIM_PATH (default realm_access.roles to match Keycloak; configurable for Okta groups, Azure AD roles, custom paths).

Not started

3

canopy-auth::ServiceTokenSource. New module crates/canopy-auth/src/service_token.rs. Constructor takes service_name, client_id, client_secret (from secrets) and the canopy-identity issuer URL (from CANOPY_IDENTITY_INTERNAL_URL, falling back to CANOPY_IDENTITY_ISSUER). On construction, fetches the OIDC discovery document, performs grant_type=client_credentials against the discovered token_endpoint, caches the resulting JWT. Spawns a refresh task that re-acquires 5 minutes before expiry. Public API: async fn current(&self) → Result<String, ServiceTokenError> — returns the cached token, blocks if a refresh is in flight, returns the most-recent-known token if refresh fails (fail-closed at expiry). Workspace dep: oauth2 = "5.0" (well-vetted, pure Rust, rustls).

Not started

4

canopy-signing::ActorTokenIssuer. New module crates/canopy-signing/src/actor_token.rs. Reuses each canopy service’s existing canopy-signing keypair. Mints actor JWTs with aud: canopy-internal-actor, 10-minute TTL, carrying the worker’s normalized claims (sub, preferred_username, the configured roles claim). Distinct aud namespace separates this from determination signing — a leaked determination JWS doesn’t grant actor authority and vice versa.

Not started

5

Auth middleware: actor extraction. Update canopy-auth::middleware::auth_middleware. (a) Validate the bearer against canopy-identity’s JWKS as today (the existing OIDC-discovery + JWKS-fetching path stays). (b) If Claims::is_service() is true AND the request carries X-Canopy-Actor: <jwt>, validate the actor JWT against canopy-signing’s JWKS, check aud == "canopy-internal-actor", set claims.actor = Some(Box::new(actor_claims)). (c) Inject the (possibly actor-enriched) Claims extension. Reject if actor JWT validation fails — never silently drop. canopy-signing’s JWKS is fetched once at startup and cached; refresh on kid cache miss.

Not started

6

Outbound helpers in crates/canopy-auth/src/client_ext.rs:

* RequestBuilder::with_service_identity(self, token_source: &ServiceTokenSource) → Self — replaces existing .bearer_auth(worker_token) calls; fetches the current canopy-identity-issued service token from the source. * RequestBuilder::with_actor(self, actor: Option<&Claims>) → Self — when Some, mints an actor JWT via ActorTokenIssuer and attaches as X-Canopy-Actor. When None (drainer publishes, scheduled jobs with no worker context), no header attached.

Internal services that don’t need user identity call .with_service_identity(…​) only.

Not started

7

Bootstrap wiring. crates/canopy-api/src/bootstrap.rs constructs a ServiceTokenSource per service (using the service’s CANOPY_<SVC>_CLIENT_ID and CANOPY_<SVC>_CLIENT_SECRET) and an ActorTokenIssuer (using the service’s existing canopy-signing keypair). Both stored on BootstrapResult so handlers can inject them as Axum Extension`s. The reqwest `Client extension ClientExt::canopy_internal() returns a builder pre-configured with the token source + actor issuer.

Not started

8

cargo xtask identity verify. New xtask command in xtask/src/cmd/identity.rs. Behavior: (a) hits <issuer>/.well-known/openid-configuration, verifies the four required endpoints (authorization_endpoint, token_endpoint, jwks_uri, optionally end_session_endpoint); (b) fetches jwks_uri, parses JWKs; (c) attempts a client_credentials grant using a configured test service principal, validates the response token shape against the contract; (d) reports per-check pass/fail with diagnostics. Read-only: no mutation of the backend. Safe in dev, CI, and production deploy gating. Args: --issuer <url> overrides CANOPY_IDENTITY_ISSUER; --client-id and --client-secret for the test principal.

Not started

9

cargo xtask identity render --backend <name>. New subcommand emitting reference IaC fragments for the supported backends. Pure code generation, no network, no mutation. Initial backends shipped:

* --backend keycloak — emits a realm.json with the canopy realm definition, 13 service-account clients (with audience mappers + service:canopy-* realm roles), and the worker realm clients (canopy-ui, canopy-api). * --backend authentik — emits an Authentik blueprint YAML covering the same shape. * --backend dex — emits a Dex static-clients + connector-config template.

Backends without a render adapter (Okta, Entra, ForgeRock, custom) require operator-side configuration in their existing tooling. The contract definition + xtask identity verify give them the spec they need.

Not started

10

cargo xtask dev identity provision. Devstack-only command (note the dev namespace prefix). Mutates the devstack Keycloak via admin API to install/update the canopy realm based on the rendered realm.json (Step 9), generates per-stack client secrets, writes them encrypted into secrets/dev.yaml per ADR-017. Idempotent: re-running with existing state is a no-op unless --rotate <service> is passed. Production deployers do NOT run this command — they provision via their own IaC and use xtask identity verify for conformance gating.

Not started

11

devstack/keycloak/canopy-realm.json extension + secrets/dev.yaml entries. The devstack realm is updated to include 13 service-account clients with placeholder secrets (replaced by dev identity provision on first cargo xtask dev start). 13 new encrypted entries in secrets/dev.yaml: CANOPY_<SVC>_CLIENT_ID, CANOPY_<SVC>_CLIENT_SECRET per service. Service principals: canopy-snap, canopy-tanf, canopy-medicaid, canopy-caps, canopy-wic, canopy-applications, canopy-eligibility, canopy-enrollment, canopy-renewals, canopy-appeals, canopy-notices, canopy-security, canopy-persons. Env-var name convention shifts from per-service CANOPY_*_OIDC* to canopy-identity-prefixed CANOPY_IDENTITY_* for the contract-level config; per-service client_id/client_secret keep the per-service prefix.

Not started

12

canopy-eligibility orchestrator switch. services/canopy-eligibility/src/orchestrator.rs:84,139,186,393 — every .bearer_auth(auth_token) becomes .with_service_identity(&svc_token).with_actor(Some(&worker_claims)). The auth_token: &str parameter on DetermineConfig becomes service_token: &ServiceTokenSource + actor: Option<&Claims>. The extract_bearer_token helper in api/handlers.rs is deleted; the worker’s Extension(claims): Extension<Claims> becomes the actor source directly.

Not started

13

canopy-web service-identity wiring. services/canopy-web/src/api/* — every internal HTTP call (8 service clients per the Service Catalog) flips from forwarding the worker bearer to using the service token + actor header. canopy-web is a worker-facing BFF; its inbound handlers continue to expect worker JWTs validated against canopy-identity’s JWKS (same JWKS, different aud). Same shape applies to canopy-portal once it has domain routes.

Not started

14

Cutover: enforce service-identity on internal-only endpoints. Per ADR-019 hard-cutover migration:

* canopy-rules::api::* — all four domain endpoints. claims.require_any_role(…​) (added in #429) is replaced with claims.require_service_caller(). Worker JWTs hitting these endpoints get 401 with service-token-required. * canopy-snap::POST /v1/determine, canopy-tanf, canopy-medicaid, canopy-caps, canopy-wic — internal-only entry; only the orchestrator calls. Same require_service_caller flip. * canopy-verification::* (3 internal endpoints) — already service-internal by name; this just makes it formal.

Each removed role-gate has a corresponding endpoint_requires_service_caller_rejects_worker_jwt test pinning the new shape.

Done (2026-05-10) — slice 1 (MR !233, canopy-rules cutover + program-service rules-client outbound refactor) merged. Slice 2 (program-service /v1/determine inbound cutover for canopy-snap/tanf/medicaid/caps/wic) + slice 3 (canopy-persons + canopy-applications + canopy-enrollment + canopy-renewals + canopy-appeals + canopy-notices inbound cutover plus canopy-reporting outbound ServiceTokenSource) bundled into a single follow-up MR — both slices flip together so receivers and callers stay in lock-step. worker_tokens_rejected_post_cutover (canopy-rules) + worker_token_rejected_on_determine_post_cutover (canopy-snap) + authenticated_caseworker_rejected_post_cutover / service_class_caller_can_read (canopy-persons) regression tests pin the new shape across the three architectural endpoints. canopy-verification (3 internal endpoints) deferred — uses a different auth model (X-Service-Api-Key shared secret, not JWT); already strict and out of scope for ADR-019’s JWT cutover. canopy-security admin gates and canopy-eligibility orchestrator inbound stay worker-accessible by design (auditors + worker entry point). canopy-cli runtime gap noted in CHANGELOG: workers running canopy CLI directly against canopy-persons will 403 post-cutover; follow-up work to either give canopy-cli a service-token mode or route through canopy-web.

15

Audit log enrichment. ADR-014 audit-log writers (canopy-tanf::store::fti_audit_log::insert, canopy-medicaid::store::fti_audit_log::insert, canopy-security::audit::insert) read both claims.service_id() (caller service) and claims.actor.as_ref().map(|a| &a.sub) (acting worker). New columns: actor_service TEXT NOT NULL DEFAULT 'unknown', actor_user_sub TEXT. Forward-only migrations per ADR-016 across the 3 affected services. Existing rows backfill actor_service = 'unknown' /* pre-ADR-019 */.

Not started

16

Tests:

* Per-service service_token_acquisition_smoke (devstack-gated) — assert each canopy-* service successfully exchanges client_credentials for a token at startup. * actor_propagation_regression (devstack-gated, in canopy-eligibility tests) — post /v1/eligibility/determine as worker jane.doe, assert canopy-snap’s audit row has actor_service = "canopy-eligibility" AND actor_user_sub = jane.doe.sub. * xtask_identity_verify_smoke — runs cargo xtask identity verify --issuer <devstack> against the running devstack Keycloak; asserts pass. * endpoint_requires_service_caller_rejects_worker_jwt per Step 14 — every removed role gate gets a regression test.

Not started

17

Docs.

* Security — new "canopy-identity contract" section describing the post-ADR-019 model. * Architecture — request-flow narrative update (today says "worker JWT travels end-to-end"; updates to "worker JWT terminates at BFF / determine entry; service tokens travel onward, X-Canopy-Actor carries worker identity for audit + RBAC"). * docs/modules/ROOT/pages/idp-integration.adoc — the canopy-identity contract goes here as the deployer-facing reference. Per-backend setup notes (Keycloak via xtask identity render --backend keycloak; Authentik via render adapter; Dex via render adapter; Okta / Entra / ForgeRock / custom by hand using the contract). xtask identity verify documented as the conformance gate. * CHANGELOG.adoc — === Changed (Foundations + Outbound flip) + === Security (Cutover with role-gate retirement); separate entries.

Not started

Issue: #424
Branches: docs/adr-019-service-identity (Step 1 — MR 1), feat/e2-service-identity-foundations (Steps 2-11 — MR 2), feat/e2-service-identity-cutover (Steps 12-17 — MR 3)
Labels: type::refactor, priority::medium, service::security, service::shared-crates, program::infrastructure, compliance::pub-1075, workflow::ready

Context

E0.5 (#429, MR !223) added per-endpoint role gates to canopy-rules to plug the reviewer’s "any worker JWT can read every ruleset" finding. Per-endpoint role gates are a tactical close — the architectural endpoint is service identity, where internal services don’t accept worker JWTs at all.

ADR-019 establishes canopy-identity as a contract (env vars + OIDC discovery requirements + token-shape requirements + conformance test). The contract is fulfilled by an OIDC issuer the operator chooses — Keycloak by default in the dev stack; Dex, Authentik, Okta, Entra, ForgeRock, or custom in production. Canopy ships contract + conformance test + reference IaC templates + dev provisioning. Production identity-backend lifecycle is the deployer’s responsibility.

This plan implements the contract on the canopy code side (Claims API, ServiceTokenSource, ActorTokenIssuer, middleware, outbound helpers, bootstrap wiring) and on the dev/CI side (xtask verify/render/dev-provision). Production-side provisioning is not in scope for canopy code — operators use their existing IaC.

Code references

  • services/canopy-eligibility/src/orchestrator.rs:84,139,186,393 — every .bearer_auth(auth_token) is a JWT pass-through call site.

  • services/canopy-eligibility/src/api/handlers.rs:24-28 — extract_bearer_token extracts the worker token from the inbound request to forward downstream.

  • services/canopy-rules/src/api/mod.rs — per-endpoint require_any_role calls (post-#429).

  • crates/canopy-auth/src/claims.rs — Claims API gets the new is_service() / service_id() / actor extensions and the configurable roles claim path.

  • crates/canopy-auth/src/middleware.rs — auth middleware gains actor extraction.

  • crates/canopy-auth/src/jwks.rs — already uses OIDC discovery (per #422); reused as-is for canopy-identity JWKS validation.

  • crates/canopy-signing/src/ — existing ES256 signing infra; actor JWT signing is a small extension.

  • devstack/keycloak/canopy-realm.json — current realm with 2 clients; 13 service-account clients added in Step 11.

  • ADR-014 — audit log row shape (this plan adds actor_service and actor_user_sub columns).

  • ADR-017 — where CANOPY_<SVC>_CLIENT_SECRET lives.

  • CRAIG ADR-011 / 021 / 026 — worker-auth patterns canopy adopts as-is.

Scope

In scope:

  • canopy-identity contract definition (env vars, OIDC discovery requirements, token shape, claim path conventions).

  • Claims API extensions (is_service, service_id, require_service_caller, actor, configurable roles() lookup).

  • ServiceTokenSource (canopy-auth) — OAuth2 client_credentials wrapper with refresh.

  • ActorTokenIssuer (canopy-signing) — service-signed actor JWTs.

  • Auth middleware actor-header extraction + canopy-signing JWKS validation path.

  • Outbound helpers (with_service_identity, with_actor).

  • canopy-eligibility orchestrator + canopy-web flip from JWT pass-through to service identity.

  • Hard-cutover enforcement on internal-only endpoints; per-endpoint role-gate removal.

  • Audit-log enrichment with caller-service + on-behalf-of-user.

  • cargo xtask identity verify — read-only conformance test.

  • cargo xtask identity render --backend {keycloak,authentik,dex} — reference IaC fragment emission.

  • cargo xtask dev identity provision — devstack-only Keycloak realm provisioning.

  • devstack canopy-realm.json extension + secrets/dev.yaml entries.

  • Documentation of the contract for deployers (idp-integration.adoc).

Out of scope:

  • Production identity-backend lifecycle tooling. Canopy does not own provisioning, secret rotation, or admin operations against deployer-owned IAM backends. Deployers use their existing IaC (Terraform, Helm, Ansible, Vault, gitops blueprints, Operator CRs, admin consoles — operator’s choice).

  • A canopy-identity service container. canopy-identity is a contract, not a service we ship. Operators deploy any compliant OIDC issuer.

  • mTLS between services (transport-layer; could layer later).

  • Token-binding (RFC 8473).

  • Per-call audience scoping.

  • Removing worker JWTs from worker-facing entry points (BFFs, /determine worker entry).

Dependencies

  • ADR-019 must merge first (Step 1, MR 1).

  • ADR-017 secret plumbing already in place.

  • canopy-signing’s per-service-keypair + JWKS infrastructure (already exists for ADR-002).

  • OIDC-discovery JWKS validation (already in place per #422).

  • Existing Keycloak in devstack/ (already shipped).

Design

The canopy-identity contract (operator-facing)

Variable Meaning

CANOPY_IDENTITY_ISSUER

OIDC issuer URL.

CANOPY_IDENTITY_INTERNAL_URL

Optional in-cluster network locator for the issuer.

CANOPY_IDENTITY_AUDIENCE

Audience for service tokens. Default canopy-internal-service.

CANOPY_IDENTITY_ROLES_CLAIM_PATH

JSON path to roles array. Default realm_access.roles.

CANOPY_IDENTITY_SERVICE_ROLE_PREFIX

Service-role marker. Default service:.

CANOPY_<SERVICE>_CLIENT_ID

Per-service OAuth2 client_id.

CANOPY_<SERVICE>_CLIENT_SECRET

Per-service OAuth2 client_secret.

Required OIDC discovery: authorization_endpoint, token_endpoint, jwks_uri. Optional: end_session_endpoint.

Required token shape: see ADR-019 §"Required token shape".

Wire shape (post-cutover)

Worker → canopy-web (worker JWT issued by canopy-identity):

POST /cases/{id}/actions/file-appeal HTTP/1.1
Authorization: Bearer eyJ... (worker JWT, iss=canopy-identity-issuer, aud=canopy-ui)

canopy-web → canopy-eligibility (service token issued by canopy-identity + actor JWT signed by canopy-web):

POST /v1/eligibility/determine HTTP/1.1
Authorization: Bearer eyJ... (service token, iss=canopy-identity-issuer, azp=canopy-web, aud=canopy-internal-service, roles=[service:canopy-web])
X-Canopy-Actor: eyJ... (canopy-web-signed actor JWT, sub=jane.doe.uuid, aud=canopy-internal-actor, exp=now+10m)

canopy-eligibility → canopy-snap (service token + new actor JWT signed by canopy-eligibility this hop):

POST /v1/determine HTTP/1.1
Authorization: Bearer eyJ... (service token, azp=canopy-eligibility)
X-Canopy-Actor: eyJ... (canopy-eligibility-signed actor JWT, sub=jane.doe.uuid)

canopy-snap audit row (post-Step 15):

INSERT INTO fti_audit_log (
    ..., actor_service, actor_user_sub, ...
) VALUES (
    ..., 'canopy-eligibility', 'jane.doe.uuid', ...
);

Trust topology

  • canopy-identity issuer (whatever the operator deploys) publishes JWKS for worker + service token validation.

  • canopy-signing publishes its own JWKS for actor JWT validation. Distinct from canopy-identity JWKS.

  • Each canopy-* service trusts:

    • canopy-identity JWKS for bearer-token validation (workers AND services, same JWKS, different aud).

    • canopy-signing JWKS for actor-JWT validation (X-Canopy-Actor header).

  • Cross-stack: each jurisdiction’s stack uses its own canopy-identity issuer + canopy-signing keys. Cross-stack tokens fail signature verification.

Files Touched

File Change

docs/modules/ROOT/pages/adrs/adr-019-service-identity-and-on-behalf-of.adoc

New ADR (MR 1)

docs/modules/ROOT/pages/idp-integration.adoc

canopy-identity contract documented for deployers; per-backend setup notes; xtask verify/render/dev-provision usage

crates/canopy-auth/src/claims.rs

Add is_service, service_id, require_service_caller, actor, configurable roles()

crates/canopy-auth/src/middleware.rs

Actor-header extraction + canopy-signing JWKS validation path

crates/canopy-auth/src/service_token.rs

New module — ServiceTokenSource

crates/canopy-auth/src/client_ext.rs

New trait — with_service_identity, with_actor

crates/canopy-auth/Cargo.toml

Add oauth2 = "5.0"

crates/canopy-signing/src/actor_token.rs

New module — ActorTokenIssuer

crates/canopy-api/src/bootstrap.rs

Construct ServiceTokenSource + ActorTokenIssuer per service

xtask/src/cmd/identity.rs

New module — verify + render --backend <name> + dev provision subcommands

xtask/src/cmd/identity/templates/keycloak/realm.json.tera

Reference Keycloak realm template

xtask/src/cmd/identity/templates/authentik/blueprint.yaml.tera

Reference Authentik blueprint template

xtask/src/cmd/identity/templates/dex/config.yaml.tera

Reference Dex config template

services/canopy-eligibility/src/orchestrator.rs

Replace .bearer_auth(auth_token) with .with_service_identity(…​).with_actor(…​)

services/canopy-eligibility/src/api/handlers.rs

Drop extract_bearer_token; Extension(claims) becomes the actor source

services/canopy-web/src/api/* (8 client call sites)

Same flip as orchestrator

services/canopy-{snap,tanf,medicaid,caps,wic}/src/api/…​

Replace require_any_role with require_service_caller on /v1/determine

services/canopy-rules/src/api/mod.rs

Remove per-endpoint role gates added in #429; replace with require_service_caller

services/canopy-{tanf,medicaid,security}/src/store/…​audit_log…​

Audit-log enrichment

services/canopy-{tanf,medicaid,security}/migrations/<date>_add_actor_columns_to_audit_log.sql

3 forward-only migrations adding actor_service + actor_user_sub

secrets/dev.yaml

13 new CANOPY_<SVC>_CLIENT_ID + CANOPY_<SVC>_CLIENT_SECRET entries (encrypted via SOPS)

devstack/keycloak/canopy-realm.json

Add 13 service-account clients (placeholder secrets) + audience mappers + service:canopy-* realm roles

Security

New "canopy-identity contract" section

Architecture

Request-flow narrative update

CHANGELOG.adoc

=== Changed (impl) + === Security (cutover)

Verification

  1. cargo nextest run -p canopy-auth — unit tests on Claims::is_service, service_id, configurable roles path, ServiceTokenSource mock-server smoke.

  2. cargo nextest run -p canopy-signing — unit tests on ActorTokenIssuer::mint round-trip + signature verification against own JWKS entry.

  3. Devstack-gated per-service service_token_acquisition_smoke — all 13 services successfully exchange client_credentials for a token at startup.

  4. Devstack-gated actor_propagation_regression — worker → canopy-web → canopy-eligibility → canopy-snap audit row carries correct actor_service + actor_user_sub.

  5. cargo xtask identity verify --issuer <devstack> runs in CI as a regression check; passes for the devstack Keycloak realm.

  6. cargo xtask identity render --backend keycloak produces a valid realm.json (compared against the devstack/keycloak/canopy-realm.json ground truth in a test).

  7. Every removed role-gate has a corresponding endpoint_requires_service_caller_rejects_worker_jwt test pinning the new shape.

  8. cargo xtask validate — full battery green at each MR boundary.

  9. Manual smoke: revoke a service’s Keycloak client secret in dev (kcadm.sh …​ reset-secret), restart the service — startup proceeds (fails at first refresh attempt), service token caching means inbound calls work for up to 1h after revocation, then fail closed with a clear log line.

Documentation Updates

  • docs/modules/ROOT/pages/adrs/adr-019-service-identity-and-on-behalf-of.adoc — new ADR (MR 1)

  • Security — "canopy-identity contract" section

  • Architecture — request-flow narrative update

  • docs/modules/ROOT/pages/idp-integration.adoc — canopy-identity contract for deployers; per-backend setup notes; xtask verify/render/dev-provision usage

  • CHANGELOG.adoc — === Changed (impl) + === Security (cutover); separate entries

  • Plan archive: move to plans/archive/ post-MR-3 merge

Edit this page · default