ADR-019: canopy-identity — Identity-Service Contract for Workers and Services

On this page
NOTE

Amended by ADR-043 / retired in part by #1443 (OIDC C1, 2026-08-24). The service identity half of this ADR (per-service client_credentials tokens, service:* roles, audit actor_service) remains the fleet contract. The on-behalf-of half — the X-Canopy-Actor header and its canopy-signing actor JWTs — is RETIRED: worker identity now rides RFC 8693 exchanged bearers (ADR-043 amendment A2 + the receiver-contract slices), and the auth middleware rejects any request still carrying the header (401). The sections below describing the actor mechanism are preserved as the historical design record.

Context

Service-to-service calls in canopy currently pass the worker’s JWT end-to-end. canopy-web extracts the worker’s bearer token from the inbound HTTP request and attaches it to the outbound call to canopy-eligibility, which attaches it to canopy-snap, which attaches it to canopy-rules. Three observed problems:

  1. Per-endpoint role gates pile up at every internal boundary. E0.5 (#429) had to add Claims::require_any_role(&["caseworker","supervisor","admin"]) to POST /v1/evaluate on canopy-rules — an internal endpoint that should never be reachable by anything other than program services. The role check protects against the bad case where a worker JWT does reach canopy-rules, but the bad case shouldn’t be possible in the first place. Every internal service grows a cargo-culted set of these checks; missing one is a vulnerability (canopy-rules went unprotected for months).

  2. Worker JWT lifetime bounds service-to-service calls. The token expires when the worker session does (8h sliding for canopy-web). A long-running orchestrator dispatch or a deferred outbox-drainer publish can outlive that token. Today this is masked because most calls are synchronous and complete in seconds, but canopy-eligibility::orchestrator already times out individual program calls at 5–30 s, and any drift toward longer-running flows walks straight into expired-token-mid-flight failures.

  3. Service-to-service calls have no caller-service identity in audit logs. The audit_events chain (ADR-014) records actor_user = sub from the JWT. When canopy-eligibility calls canopy-snap, the audit row says "actor=worker:jane.doe" — true at the top of the call chain, but useless for "which service initiated this read of FTI?" because the answer is always the same: the worker. Reviewer noted on the 2026-05-07 audit that we cannot distinguish "worker invoked SNAP determination directly" from "worker invoked eligibility orchestration which fanned out to SNAP" without parsing logs.

The architectural endpoint is service identity — each canopy-* service authenticates as itself to other canopy-* services. Worker identity, when relevant for audit or eligibility decisions, travels alongside as an explicit "on-behalf-of" assertion, not as the credential itself.

Constraints

  • Per-jurisdiction, not multi-tenant SaaS. Each jurisdiction runs its own canopy stack. Architectural simplicity per stack is what matters.

  • Operators choose any IAM backend. Per idp-integration.adoc, deployers run whatever they have — Keycloak, Okta, Auth0, Azure AD / Entra ID, Authentik, ForgeRock, PingFederate, on-prem AD with ADFS or LDAP. Canopy must not lock anyone into a specific backend.

  • Canopy is not in the ops business. xtask is dev/CI tooling. Production identity-backend lifecycle (provisioning, secret rotation, access-review) is the deployer’s responsibility via their existing IaC and operations practices. Canopy supplies contract, conformance test, and reference templates — nothing that mutates production infrastructure.

Relationship to CRAIG

CRAIG (the sibling CCWIS project) has converged on three principles for worker authentication:

  • CRAIG ADR-011 — Published JWT contract; any RFC 6749 + OIDC-discovery IdP works; configurable role claim path; no IdP admin API dependency.

  • CRAIG ADR-021 — Per-service aud enforcement, azp/scope/typ validation.

  • CRAIG ADR-026 — worker_identities table populated lazily from observed JWT claims; no calls to IdP admin APIs to map preferred_username → sub.

Canopy adopts these worker-auth patterns as-is. Same contract, same enforcement, same identity-normalization approach. Where canopy is currently ahead of CRAIG (canopy-auth already uses OIDC discovery and OIDC_ISSUER env var convention; CRAIG’s ADR-011 says it should but the code still has Keycloak-specific paths), CRAIG should catch up.

Where canopy genuinely diverges from CRAIG: internal service identity. CRAIG’s call graph is dominated by RabbitMQ async messaging (CRAIG ADR-003); canopy’s orchestrator pattern produces ~30 synchronous service-to-service calls per worker action, each currently passing the worker JWT. CRAIG can defer service-identity work until they hit the same scale; canopy cannot.

Options considered

  1. OAuth2 client_credentials directly against the operator’s IdP, no canopy-side abstraction. Each canopy service registered as a client in the deployer’s IdP. Major IdPs all support this; bare AD/SAML-only deployers must run their own federator (Keycloak/Dex/etc.) outside canopy. Considered. Loses the "single integration point per service" property — every canopy service ends up with operator-IdP-specific config (claim paths, audience values, token endpoints) duplicated.

  2. Self-signed service JWTs with canopy-signing as trust root. Every service self-signs its own service token; canopy-signing publishes an aggregated JWKS. Builds a parallel auth system inside canopy. Confuses canopy-signing’s existing role (signed determinations per ADR-002) with auth identity. Forces every service to validate two kinds of tokens. Rejected.

  3. Build a canopy-identity proxy service that wraps backends. New canopy-* service container that sits between canopy services and the actual IAM backend. Adds latency, a SPOF, and operational footprint. Most of what such a proxy would do is already done by OIDC discovery. Rejected.

  4. Define canopy-identity as an interface contract; backends fulfill it directly. No new service container. canopy services depend on CANOPY_IDENTITY_ISSUER (an OIDC issuer URL); operators point that at any compliant backend. Selected.

Decision

canopy-identity is a contract, not a service container. Every canopy-* service depends on the canopy-identity contract URL — an OIDC issuer that satisfies the requirements below. No new service is introduced. Operators choose any compliant backend (Keycloak by default in the dev stack; Dex, Authentik, Okta, Entra, ForgeRock, or custom in production). Canopy ships the contract definition, a conformance test, and reference IaC templates — but does not own production identity-backend lifecycle.

The canopy-identity contract

Required environment variables

Variable Meaning

CANOPY_IDENTITY_ISSUER

OIDC issuer URL. Used for iss claim validation and to discover the well-known endpoints. Example: https://idp.example.gov/realms/canopy.

CANOPY_IDENTITY_INTERNAL_URL

Optional override of the issuer URL for in-cluster traffic (Docker network, Kubernetes service DNS). Same content, different network locator. Replaces the existing per-service CANOPY_*__OIDC_INTERNAL_URL.

CANOPY_IDENTITY_AUDIENCE

Audience claim canopy services validate against. canopy-internal-service for service-to-service tokens; per-service for worker-facing endpoints (canopy-ui, canopy-api, etc., per CRAIG ADR-021).

CANOPY_IDENTITY_ROLES_CLAIM_PATH

JSON-pointer-style path to the roles array in the JWT. Default realm_access.roles (Keycloak). Operators with backends that emit roles elsewhere (Okta groups, Azure AD roles, custom) override.

CANOPY_IDENTITY_SERVICE_ROLE_PREFIX

Prefix marking a role as a service identity. Default service:. A claim like service:canopy-eligibility in the configured roles path identifies the caller as canopy-eligibility’s service principal.

CANOPY_<SERVICE>_CLIENT_ID

The OAuth2 client_id this service uses for its own client_credentials grant. Example: CANOPY_ELIGIBILITY_CLIENT_ID=canopy-eligibility.

CANOPY_<SERVICE>_CLIENT_SECRET

Secret for the OAuth2 client. Encrypted via SOPS per ADR-017 in secrets/dev.yaml; per-jurisdiction encrypted-config repos in production.

Required OIDC discovery endpoints

The issuer’s /.well-known/openid-configuration document MUST advertise:

  • authorization_endpoint — for worker authorization_code + PKCE flows from the BFFs.

  • token_endpoint — supporting grant_type=client_credentials (services) and grant_type=authorization_code (workers).

  • jwks_uri — for JWT signature verification.

  • end_session_endpoint — RP-initiated logout from the BFFs (recommended; not required).

Required token shape

Service tokens (issued via client_credentials):

  • iss matches CANOPY_IDENTITY_ISSUER.

  • sub is stable across the service principal’s lifetime.

  • azp (or client_id claim) identifies the calling service.

  • aud includes canopy-internal-service.

  • exp, iat standard.

  • The roles claim (at CANOPY_IDENTITY_ROLES_CLAIM_PATH) contains an entry starting with CANOPY_IDENTITY_SERVICE_ROLE_PREFIX. Default: service:canopy-<name>.

Worker tokens (issued via authorization_code to a BFF client):

  • iss matches CANOPY_IDENTITY_ISSUER.

  • sub is the worker’s stable identifier.

  • aud is the requesting BFF client (canopy-ui, canopy-api, etc.).

  • preferred_username, email standard OIDC.

  • Roles claim contains worker roles (no service:* entries). Per-jurisdiction role taxonomy (per idp-integration.adoc).

Canopy Claims deserialization tolerates either string or array aud (per CRAIG ADR-021’s aud_or_vec deserializer). Roles are looked up via the configured path with a default-Keycloak fallback.

Conformance: cargo xtask identity verify

A read-only conformance check that points at any candidate backend and confirms it satisfies the contract. Behavior:

  1. Hits <issuer>/.well-known/openid-configuration. Verifies the four required endpoints are present.

  2. Fetches jwks_uri. Confirms it parses, contains usable signing keys.

  3. Performs a client_credentials grant using a test service principal (configured in the same env vars as a real canopy service). Validates the response token against the contract: iss, aud, exp, role claim shape, audience.

  4. Performs an authorization_code flow with PKCE against a test worker principal (when test credentials are available). Validates the resulting token similarly.

  5. Reports per-check pass/fail with diagnostic detail.

cargo xtask identity verify --issuer <url> is safe to run anywhere — dev, CI, production deployment-gating. It does not mutate the backend. Production deployers run it as part of their canopy rollout pipeline; CI runs it against the devstack Keycloak as a regression check.

Reference templates: cargo xtask identity render

Pure code generation. Emits IaC fragments for backends canopy provides adapters for:

  • cargo xtask identity render --backend keycloak [--out realm.json] — emits a Keycloak realm definition with the worker realm + 13 service-account clients + audience mappers + service:canopy-* realm roles. Operators merge it into their Keycloak Operator CR / Helm chart / Terraform Keycloak provider config.

  • cargo xtask identity render --backend authentik [--out blueprint.yaml] — emits an Authentik blueprint covering the same shape.

  • cargo xtask identity render --backend dex [--out dex.yaml] — emits a Dex static-clients + connector-config template.

Backends without a render adapter (Okta, Entra, ForgeRock, PingFederate, custom) require operator-side configuration in whatever tooling the operator already uses. The contract definition above + the conformance test give them the spec they need.

Lifecycle ownership

Canopy ships:

  • The contract (this ADR + env-var schema in canopy-common config).

  • The conformance test (cargo xtask identity verify).

  • Reference IaC templates (cargo xtask identity render --backend …).

  • Devstack provisioning (cargo xtask dev identity provision — namespaced under dev so the dev-only intent is unmistakable; mutates the devstack Keycloak admin API to install the canopy realm).

  • Client-side code in canopy-auth that consumes the contract (token sources, JWT validators, claims extensions).

Canopy does not ship:

  • Production provisioning that mutates a deployer’s IAM backend.

  • Production secret rotation tooling.

  • A canopy-identity service container.

The application requires service client credentials to exist; it does not own their lifecycle. Production deployments provision them through the deployer’s existing operations stack (Terraform with the relevant IAM provider, Helm/Kustomize values, Ansible, Vault scripts, Authentik blueprints in gitops, Keycloak Operator CRs, or human admin-console clicks) and validate them with cargo xtask identity verify before rolling out canopy services.

On-behalf-of: X-Canopy-Actor

When a service makes an outbound call on behalf of a worker (e.g., canopy-eligibility orchestrating a SNAP determination requested by worker:jane.doe), the bearer token is the calling service’s client_credentials-issued token. Worker identity propagates via an X-Canopy-Actor header carrying a service-signed JWT:

{
  "iss": "canopy-eligibility",
  "sub": "<worker-sub>",
  "preferred_username": "jane.doe",
  "<roles-claim-path>": ["caseworker"],
  "aud": "canopy-internal-actor",
  "exp": <now + 600>,
  "iat": <now>,
  "act_for": "canopy-eligibility"
}

The actor JWT is signed by the calling service’s existing canopy-signing keypair (the same key used for ADR-002 signed determinations). Different aud namespace (canopy-internal-actor) separates the two uses — a leaked determination JWS doesn’t grant actor authority and vice versa. The actor JWT validates against canopy-signing’s JWKS (separate from canopy-identity’s JWKS).

Two JWKS to validate per request when X-Canopy-Actor is present is by design. The bearer is canopy-identity-issued (operator backend); the actor is canopy-signing-issued (canopy-internal). Conflating them would put canopy in the IdP business; keeping them separate keeps the operator’s IAM backend the single source of truth for who and canopy-signing the source of truth for which canopy service signed this assertion.

This pattern works regardless of whether the operator’s backend supports RFC 8693 token-exchange (most don’t, today). Operators with token-exchange-capable backends could in principle use backend-issued actor tokens; canopy doesn’t require it.

Receiving-side validation

canopy-auth::middleware:

  1. Extract Authorization: Bearer <token>. Validate against canopy-identity’s JWKS (cached at startup, refreshed on kid cache miss). Validate aud matches per-endpoint expectation: canopy-internal-service for internal endpoints, the BFF client for worker-facing entry points. Validate exp, iss.

  2. Look up the configured roles claim path. If any entry starts with CANOPY_IDENTITY_SERVICE_ROLE_PREFIX, mark Claims::is_service() == true.

  3. If the request also carries X-Canopy-Actor, validate the actor JWT against canopy-signing’s JWKS, check aud == "canopy-internal-actor", set claims.actor = Some(Box::new(actor_claims)).

  4. Reject if either validation fails. Never silently drop.

Claims API

impl Claims {
    pub fn is_service(&self) -> bool {
        self.roles().iter().any(|r| r.starts_with(SERVICE_ROLE_PREFIX))
    }

    pub fn service_id(&self) -> Option<&str> {
        self.roles().iter()
            .find_map(|r| r.strip_prefix(SERVICE_ROLE_PREFIX))
            .or(self.azp.as_deref())
    }

    pub fn require_service_caller(&self) -> Result<(), ApiError> {
        if self.is_service() { Ok(()) } else { Err(ApiError::Forbidden) }
    }

    pub fn actor(&self) -> Option<&Claims> { self.actor.as_deref() }
}

Claims::roles() reads from the configured path (default realm_access.roles).

Consequences

Positive

  • Single integration point per service. Every canopy-* service depends on one URL (CANOPY_IDENTITY_ISSUER). No per-service operator-IdP integration. No per-service per-IdP claim-mapper hand-coding.

  • Genuine backend pluggability. Services don’t know the backend type because they only consume OIDC discovery + JWKS + standard JWT claims. Keycloak, Dex, Authentik, Okta, Entra, ForgeRock, or a custom Rust binary all work as long as xtask identity verify passes.

  • No new service container. Zero net deployment-surface increase. canopy-auth (existing crate) gains client-side helpers; everything else is reference templates and a verifier.

  • Production identity lifecycle stays with the deployer. Canopy doesn’t claim ownership of secrets, client registrations, or rotation schedules. Operators use their existing IaC.

  • Per-endpoint role gates collapse. Internal endpoints check claims.is_service(). The role-gates added in #429 become claims.require_service_caller() — one line per endpoint.

  • Audit log gains caller-service. audit_events.actor_service = claims.service_id() (always present on service tokens). actor_user = claims.actor.as_ref().map(|a| &a.sub) (when X-Canopy-Actor is present). The reviewer’s "did the worker invoke SNAP directly or via orchestrator" question now has a definitive per-row answer.

  • Service token lifetime decoupled from worker session. Service tokens refresh on the service’s schedule. Long-running flows don’t fail mid-flight on session expiry.

  • Aligns canopy with CRAIG on worker auth. CRAIG ADR-011/021/026 patterns adopted verbatim. Canopy is ahead on implementation; this ADR codifies the shared direction.

Negative

  • Two JWKS to validate per request when X-Canopy-Actor is present. Bearer validates against canopy-identity JWKS; actor validates against canopy-signing JWKS. Mitigated by caching: canopy-auth caches verified JWTs for 30 s, with separate cache hits for bearer and actor tokens.

  • Operators must configure 13 service-account clients in their IAM backend. This is real work, but it’s a one-time per-stack cost (canopy is per-jurisdiction, not multi-tenant SaaS) and xtask identity render provides a starting template for the supported backends.

  • Reference templates can drift from canopy’s expectations. Mitigated by xtask identity verify — operators run it post-provisioning to confirm their backend matches canopy’s contract.

  • Secrets rotation is the deployer’s job. Canopy-side caching survives a rotation event up to the cached service-token TTL (1h default); after that, services need a fresh client_credentials grant. Operators document their rotation procedure; canopy doesn’t ship rotation tooling.

Mitigations

  • JWKS staleness: OIDC discovery + JWKS endpoints publish with sensible cache headers; canopy-auth refreshes on kid cache miss.

  • Per-stack trust isolation: Each canopy stack uses its own canopy-identity issuer URL. Tokens from stack A don’t validate against stack B because the JWKS keys differ.

  • Per-endpoint role gates from #429: Removed during the cutover MR (Phase 3). require_service_caller is the structural replacement.

Migration: hard cutover (pre-1.0)

Pre-1.0 with no production users. Three implementation MRs after the ADR lands:

Phase Behavior

MR 1 — ADR + contract docs

This ADR lands standalone. Plus updates to idp-integration.adoc documenting the contract. No code change.

MR 2 — Foundations

Claims API extensions. ServiceTokenSource (canopy-auth client_credentials wrapper). ActorTokenIssuer (canopy-signing). Auth middleware actor extraction. Outbound helpers (with_service_identity, with_actor). Bootstrap wiring. xtask identity verify + xtask identity render --backend keycloak + xtask dev identity provision. devstack canopy-realm.json extended with 13 service-account clients (placeholder secrets; dev identity provision populates real values). secrets/dev.yaml gains 13 entries. Worker JWTs still accepted at every endpoint.

MR 3 — Outbound flip + cutover

Every canopy-eligibility orchestrator + canopy-web outbound call switches from forwarding worker JWT to with_service_identity + with_actor. Internal-only endpoints flip to require_service_caller. #429 role gates removed. Audit log writers gain actor_service + actor_user_sub columns + writes. End-to-end actor-propagation regression test pins the contract.

Worker-facing entry points (canopy-web, canopy-portal, canopy-applications intake, canopy-eligibility’s /v1/eligibility/determine worker entry, canopy-enrollment caseworker actions, canopy-renewals, canopy-appeals) keep accepting worker JWTs validated against canopy-identity’s JWKS — same JWKS, different aud. That’s the steady state.

Out of scope

  • mTLS between services. Could layer on top of this design. Not a substitute.

  • Token-binding (RFC 8473). Not necessary at canopy’s threat model.

  • Per-service token audience. Single aud: canopy-internal-service is sufficient.

  • Replacing canopy-signing. canopy-signing keeps its existing role (signed determinations, ADR-002) plus a small extension (signed actor JWTs, distinct aud namespace).

  • Worker on-behalf-of without going through a BFF / determine entry. Worker tokens are validated once at the front door; thereafter worker identity travels as actor JWTs. There is no "worker calls canopy-rules directly" path — that was never supposed to be possible and is now structurally prevented.

  • Service catalog, multi-tenancy, projects/domains, quotas, endpoint registry. OpenStack Keystone provides these; canopy doesn’t need them.

  • Production provisioning tooling. Canopy ships dev provisioning + reference templates + a conformance test. Production lifecycle is the deployer’s responsibility.

References

Edit this page · default