Implementation Guide

On this page

This guide provides the technical specification for implementing each service in the Canopy platform. It is the companion to the Roadmap, which tracks progress at a strategic level. Each section below contains enough detail — service patterns, shared crates, event contracts, and coding patterns — for any developer to implement a plan independently.

Shared Architecture

Every Canopy service follows the same structural pattern. Understanding this pattern once makes each subsequent plan straightforward.

Service Pattern

Each service is a standalone Axum binary (services/canopy-<name>/) that:

  1. Loads configuration from environment variables (CANOPY_<SERVICE>__*)

  2. Connects to its own PostgreSQL database (canopy_<service>) — per ADR-001, no service shares a database

  3. Connects to the shared RabbitMQ message bus (canopy.events topic exchange)

  4. Validates Keycloak JWTs via the canopy-auth middleware (RS256, JWKS rotation)

  5. Exposes a versioned REST API under /v1/<service>/…​

  6. Exposes an unauthenticated health check at GET /healthz

  7. Exposes Prometheus metrics at GET /metrics

Directory Layout

services/canopy-<name>/
├── Cargo.toml
├── migrations/
│   ├── YYYYMMDD_create_<name>_tables.sql
│   └── COMPLIANCE.md              # Present in FTI/IEVS services (snap, tanf, medicaid)
├── src/
│   ├── main.rs                    # bootstrap, routes, event subscriptions
│   ├── api/
│   │   └── mod.rs                 # route handlers
│   ├── events.rs                  # event publishing and subscription handlers
│   └── <domain modules>.rs        # service-specific logic
└── tests/
    └── <name>_tests.rs            # integration tests

Shared Crates

Crate Purpose

canopy-common

Configuration loading (Settings), ApiError type (RFC 9457 Problem Details), UUID v7 IDs (CanopyId), pagination, structured logging via tracing

canopy-auth

Keycloak JWKS discovery and caching, Bearer token validation middleware, Claims extraction, role-based access helpers

canopy-db

DbPool wrapper around sqlx::PgPool, health check, migration runner

canopy-mq

RabbitMQ connection via lapin 4, ConnectionManager reconnect supervisor (exponential backoff, single-flight), Publisher (in-memory bounded buffer for events during a broker outage; CANOPY_MQ_BUFFER_MAX env override, default 10 000) and Subscriber (auto re-attaching consume loop), EventEnvelope message format, topic exchange binding

canopy-api

Axum server builder with standard middleware stack (CORS, compression, tracing, auth), health and metrics endpoints, OpenTelemetry integration

canopy-store

Object storage abstraction wrapping object_store crate — uniform put/get/delete/list API across local filesystem (dev) and S3-compatible backends (Garage in devstack, any S3 in production)

canopy-reference

Shared domain enums (strum-derived): BenefitProgram, DeterminationStatus, IncomeType, AssetType, NoticeType, VerificationSource, etc. FIPS codes for all US states and counties. Used by canopy-seed for deterministic test data and by all services for enum-based validation.

canopy-test-lib

Integration test harness: devstack availability guard, typed service clients, Keycloak token provider, test cleanup helpers

Standard Event Envelope

All messages on the RabbitMQ canopy.events topic exchange use a common JSON envelope:

{
  "id": "01942a3b-...",
  "timestamp": "2026-04-15T14:30:00Z",
  "source_service": "canopy-snap",
  "event_type": "snap.determination_completed",
  "payload": { ... }
}

The event_type field doubles as the AMQP routing key, enabling selective subscription.

Data restrictions in events (ADR-004): Event payloads must never contain FTI, IEVS, SSA SOLQ/BINDEX, or HIPAA-scoped data. Events carry IDs, statuses, and timestamps only. The consuming service retrieves full data via the producing service’s API if authorized.

Standard Roles

Services enforce role-based access using Keycloak realm roles:

Role Access Level

admin

Full system administration — user management, service configuration

supervisor

Case oversight, approval workflows, reassignment

eligibility_worker

Process applications, run determinations, manage caseload

intake_worker

Application intake and initial screening only

fiscal_officer

Reporting access, benefit issuance oversight

readonly

View-only access across all modules

Sort & Search Pattern

All paginated list endpoints support server-side sorting and text search via optional query parameters: search, sort_by, sort_dir.

Sort safety: Dynamic ORDER BY columns are validated through a whitelist function that returns hardcoded string literals, preventing SQL injection:

fn validated_sort_column(sort_by: Option<&str>) -> &str {
    match sort_by {
        Some("application_number") => "application_number",
        Some("status") => "determination_status",
        _ => "created_at", // safe default
    }
}

Search: Parameterized ILIKE clauses search across relevant text columns:

AND ($3::TEXT IS NULL
     OR application_number ILIKE '%' || $3 || '%'
     OR applicant_name ILIKE '%' || $3 || '%')

The BFF (canopy-web) passes search, sort_by, and sort_dir as query parameters to backend API calls and forwards them to Askama templates for rendering sortable headers and preserving state across pagination links.

Taking a Stub to Production

Every service starts as a stub with healthz and metrics endpoints. This section describes the standard sequence for implementing a service from its plan.

Step 1: Schema (migrations)

Create sqlx migrations in services/canopy-<name>/migrations/. Naming convention: YYYYMMDDHHMMSS_<description>.sql.

  • UUID primary keys (v7 for time-ordering)

  • TIMESTAMPTZ for all date/time columns (never TIMESTAMP)

  • created_at and updated_at with DEFAULT now() on every table

  • TEXT with CHECK constraints for enum columns (not PostgreSQL enums — they’re hard to alter)

  • Index columns used in WHERE clauses and foreign key lookups

For FTI/IEVS services (canopy-snap, canopy-tanf, canopy-medicaid), add a COMPLIANCE.md in the migrations directory documenting which tables contain restricted data and the regulatory basis.

Step 2: Domain types

Define Rust structs and enums in src/ modules. All domain types should derive serde::Serialize and serde::Deserialize. Use sqlx::FromRow for database row mapping. Reference canopy-reference enums where applicable — do not duplicate enum definitions.

Step 3: API routes

Add route handlers in src/api/mod.rs. Follow the Axum 0.8 pattern with extractors:

// SPDX-License-Identifier: AGPL-3.0-or-later

use axum::{extract::State, Json};
use canopy_auth::Claims;
use canopy_common::{ApiError, Pagination};

pub async fn list_items(
    State(state): State<AppState>,
    claims: Claims,
    pagination: Pagination,
) -> Result<Json<Vec<Item>>, ApiError> {
    claims.require_role("eligibility_worker")?;
    // ...
}

  • All write endpoints return 201 Created with the created resource

  • All list endpoints accept Pagination and return paginated results

  • All error responses use RFC 9457 Problem Details via ApiError

  • SPDX header on every new .rs file

Step 4: Event publishing

Wire event publishing in src/events.rs. Use canopy-mq Publisher to publish to the canopy.events topic exchange. The routing key is the event type (e.g., snap.determination_completed).

Critical: Review ADR-004 data restrictions before defining event payloads. Events from program services must not contain income data, FTI, IEVS results, or PHI. Carry IDs and statuses only.

Step 5: Event subscription

If the service reacts to events from other services, wire subscription handlers in src/events.rs or src/main.rs. Use Subscriber::subscribe() for competing-consumer (database-mutating) handlers. Use Subscriber::subscribe_exclusive() for fan-out (cache invalidation) handlers.

Step 6: Tests

Write integration tests in tests/ using canopy-test-lib. Tests interact through public REST APIs only — never touch PostgreSQL directly. Use testcontainers-rs for isolated PostgreSQL instances in unit/integration tests.

Step 7: Wire into devstack

Add the service to docker-compose.yml with appropriate profiles: tags per ADR-005. Add the database to devstack/postgres/init.sql.

Step 8: Documentation

Update the canonical Antora docs: endpoint detail in docs/modules/ROOT/pages/api/canopy-{service}.adoc, schema detail in data-models/canopy-{service}.adoc, and the service’s capability block + topology in the Service Catalog. Service/endpoint/table knowledge has a single home (the Antora pages above) — .claude/CLAUDE.md carries no feature-status table; it only points at the Service Catalog. Add CHANGELOG.adoc entry under == Unreleased.

Determination Flow

The determination flow is the core business process in Canopy, governed by ADR-002.

Sequence

Applicant/Worker
  │
  ▼
canopy-applications  ──POST /v1/applications──►  creates application record
  │
  ▼
canopy-eligibility   ──POST /v1/eligibility/determine──►  orchestration begins
  │
  ├── GET /v1/persons/{household_id}  ◄── canopy-persons (demographics, income, assets)
  │
  ├── POST /v1/{program}/evaluate     ◄── canopy-snap, canopy-tanf, etc.
  │       │
  │       ├── POST /v1/rules/evaluate  ◄── canopy-rules (JDM ruleset evaluation)
  │       │
  │       └── returns signed JWS determination (ADR-002)
  │
  └── assembles multi-program determination response
      │
      ▼
  publishes eligibility.determination_completed event

Determination Signing (ADR-002)

Program services return signed JWS determinations. The signing infrastructure:

  1. Key pair managed by canopy-eligibility (Ed25519 or RS256)

  2. Program service calls canopy-eligibility signing endpoint with determination payload

  3. canopy-eligibility signs and returns JWS compact serialization

  4. JWS token stored alongside the determination record

  5. Any service can verify the determination by fetching the public key from canopy-eligibility’s JWKS endpoint

This ensures non-repudiation — a determination cannot be tampered with after signing.

Rules Evaluation (ADR-003)

All eligibility logic lives in versioned JDM rulesets evaluated by canopy-rules:

  1. Program service sends household context (income, assets, household composition) to canopy-rules

  2. canopy-rules loads the appropriate ruleset for the jurisdiction (CANOPY_JURISDICTION env var) and program

  3. zen-engine evaluates the ruleset and returns the result

  4. Program service interprets the result and builds the determination

Rulesets are organized per ADR-006:

  • rulesets/federal/ — FPL tables, SNAP allotments, deductions (versioned by fiscal year)

  • rulesets/{jurisdiction}/ — jurisdiction-specific rules and jurisdiction.toml configuration

  • Hot-reloadable via PUT /v1/rulesets/{name} without service restart

Data Isolation (ADR-004)

Services handling restricted federal data have additional requirements:

FTI (Federal Tax Information)

  • Authorized services: canopy-tanf, canopy-medicaid only

  • Storage: FTI columns encrypted at rest; separate audit log table in the same database

  • Audit logging: every FTI access logged with worker ID, timestamp, purpose, and data elements accessed

  • Events: FTI data never appears in event payloads — events carry determination IDs only

  • IRS Pub 1075 compliance: annual safeguard review readiness

IEVS (Income and Eligibility Verification System)

  • Authorized services: canopy-snap only (7 USC 2025(e))

  • Data sources: State wage records (SWR), UI benefits, SSA SDX/BENDEX

  • Events: IEVS match results never appear in event payloads

  • Cross-program: other program services cannot query canopy-snap’s IEVS data

SSA SOLQ/BINDEX

  • Authorized services: services with an active Computer Matching Agreement (CMA)

  • Isolation: separate query infrastructure, separate audit logging

Deployment Profiles (ADR-005)

Canopy supports deploying any program subset via Docker Compose profiles:

Profile Services included

snap-only

Infrastructure + canopy-snap + canopy-enrollment + canopy-renewals + canopy-reporting

tanf-only

Infrastructure + canopy-tanf + canopy-enrollment + canopy-renewals + canopy-reporting

snap-tanf

Infrastructure + canopy-snap + canopy-tanf + canopy-enrollment + canopy-renewals + canopy-reporting

medicaid-chip

Infrastructure + canopy-medicaid + canopy-exchange + canopy-enrollment + canopy-reporting

full

All services

Infrastructure services (always required): canopy-persons, canopy-applications, canopy-rules, canopy-eligibility, canopy-verification, canopy-notices, canopy-appeals, canopy-security, canopy-web

Capability flags: optional service URLs (e.g., CANOPY_EXCHANGE_URL) — if unset, the calling service logs a debug message and skips the call. Required-to-required calls fail fast with 503 on missing peer.

Horizontal Scalability

All Canopy services are designed to scale horizontally behind a load balancer with no single-instance assumptions.

Stateless Request Handling

Services validate Keycloak JWTs locally (JWKS cached and auto-refreshed) and hold no server-side session state for API consumers. The worker BFF (canopy-web) uses PostgreSQL-backed sessions via tower-sessions-sqlx-store — any instance can serve any session. The applicant BFF (canopy-portal) does not depend on tower-sessions; per ADR-026 it uses Redis-primary opaque-token sessions (no Postgres session store), and any instance can serve any session via the shared Redis store.

Event Subscription Patterns

The canopy-mq crate provides two subscription methods for different scaling needs:

Method Queue Type Use Case

Subscriber::subscribe()

Durable, shared

Competing consumers — only one instance processes each message. Use for event handlers that mutate the database.

Subscriber::subscribe_exclusive()

Exclusive, auto-delete

Fan-out to all instances — every instance receives every matching message. Use for cache invalidation and local state updates.

Cache Invalidation Pattern

Services with in-memory caches (e.g., compiled rulesets in canopy-rules) must publish an invalidation event after any mutation so all instances reload:

  1. After a successful write + local cache reload, publish an invalidation event (e.g., rules.cache_invalidated)

  2. Each instance subscribes on a unique exclusive queue (format!("{service}.cache.{uuid}")) so all instances receive the event

  3. On receipt, each instance reloads from the database

  4. The originating instance receives its own event and reloads a second time — this is idempotent and avoids tracking instance IDs

Testing Infrastructure

Design Principles

Principle Description

Public API only

Integration tests interact through REST endpoints — never touch PostgreSQL directly. Validates the same contract real consumers use.

Credential isolation

Pre-configured Keycloak test users with different role combinations cover all access patterns. Tests never modify identity data.

Typed service clients

Per-service HTTP clients wrapping reqwest with bearer token injection.

Self-skipping tests

Integration tests check devstack availability at runtime. If unreachable, tests skip gracefully. cargo test is always safe to run without devstack.

Unique resource names

UUID v7 suffixes on all test-created resources enable safe parallel execution.

Test Profiles (nextest)

Profile Purpose

default

Unit tests only — no devstack required

integration

Full integration tests — requires devstack running

ci

CI-optimized — retries, timeouts, JUnit output

Compliance Testing

Program services with restricted data (FTI, IEVS) require additional test scenarios:

  • Verify restricted data does not appear in event payloads

  • Verify restricted data does not appear in API responses to unauthorized roles

  • Verify audit log entries are created for every restricted data access

  • Verify cross-service API calls do not expose restricted data outside authorized services

DevStack

All infrastructure runs via Docker Compose for local development:

Service Image Port

PostgreSQL 18

postgres:18-alpine (×6 — the shared instance via devstack/postgres, plus 5 per-program-service instances)

5432 (shared) / 5433–5437 (per-program)

RabbitMQ 4.2

rabbitmq:4.2-management-alpine (via devstack/rabbitmq)

5672 / 15672

Keycloak 26.5

keycloak/keycloak:26.5 (via devstack/keycloak)

8180

Garage

dxflrs/garage:v2.2.0 (via devstack/garage)

3900 / 3903

Redis 7

redis:7-alpine (×2 — cache + sessions)

6379 / 6380

Prometheus

prom/prometheus:v3.6.0

9090

Grafana

grafana/grafana:11.6.0

3000

docker-compose.yml (+ the devstack/*/Dockerfile bases) is the source of truth for these pins; update this table in the same MR as any pin bump.

Each Canopy service gets its own database (canopy_rules, canopy_persons, etc.) created by devstack/postgres/init.sql.

Start with cargo xtask dev start. Stop with cargo xtask dev stop. Rebuild after schema changes with cargo xtask dev restart.

CI/CD

The GitLab CI pipeline (.gitlab-ci.yml) has two stages; artifact promotion is build-once / gate-complete per ADR-040:

Stage When Jobs

test

MR, main, tag

All blocking gates in parallel: cargo-fmt, cargo-clippy, cargo-test (nextest ci profile), cargo-doctest, integration-tests (full devstack via DinD; main + tags only — it pulls the build jobs' staging refs instead of compiling in-daemon, so the suite tests the exact digests promotion retags, #1073), the compliance/policy audit family (adr-011-, adr-031-, compliance-, typed-id-path-audit, route-authz-audit, ci-config-lint, quality-budgets, adr-013-plan-lint, secrets-yaml-lint), GitLab SAST / secret-detection / dependency-scanning, cargo-audit (cargo-deny)
cargo-machete, coverage (MR only) — plus the two image builds: build-service-image (root Dockerfile, all service binaries) and build-portal-image (services/canopy-portal/Dockerfile, the Dioxus dx bundle), each pushed once to an immutable commit-SHA *staging ref (…/build:$CI_COMMIT_SHA, …/build/portal:$CI_COMMIT_SHA) with a pinned-syft CycloneDX SBOM retained per digest.

promote

main + tags

docker-promote — declares no needs:, so it waits for the entire test stage (every blocking gate gates every production-registry mutation), then retags the tested staging digests into the production repositories ($CI_REGISTRY_IMAGE:<short-sha> / :<tag> + the /portal twins) via registry-side docker buildx imagetools create — never a rebuild. latest moves only when the promoted commit is the current main head, serialized by a resource_group. Also: sbom (source-level cargo SBOM, tags only) and pages (Antora docs, main only).

The image-build and promote rules share one YAML-anchored artifact-input map (every COPY source of both Dockerfiles, the Dockerfiles, .dockerignore, .gitlab-ci.yml); cargo xtask ci-config-lint — run in CI and in the pre-push battery — parses the Dockerfiles and fails if the map misses an input, and statically enforces the no-needs: / no-rebuild / guarded-latest invariants.

Rust jobs cache .cargo + target/ in per-shape cache families with a single writer each (cargo-xtask- — the xtask gate jobs, written only by ci-config-lint; cargo-clippy-; cargo-test-*, also read by cargo-doctest; cargo-cov-shared for the MR-only coverage job), and every rust job wipes a restored target/ that exceeds its CARGO_CACHE_BOUND_KB before building. Cache size therefore follows a bounded sawtooth (issue #1067: the previous single shared pull-push cache grew monotonically until the runner disk filled mid-link, failing cargo-test on every main pipeline).

NOTE
The merge gate for functional correctness remains the local pre-push battery (see Contributor Workflow Conventions); the pipeline’s promotion barrier decides what reaches the container registry, not what merges.
Edit this page · default