Developer Guide

On this page

Prerequisites

Host-side tools:

  • Rust stable toolchain (Edition 2024, MSRV 1.94 — pinned by rust-toolchain.toml) — install via rustup

  • Docker Engine with the Docker Compose v2 plugin — daemon running and your user added to the docker group (sudo usermod -aG docker $USER, then re-login)

  • Git with commit signing configured (GPG or SSH/EdDSA) — see Commit signing

  • glab CLI — used by cargo xtask validate to enforce the public-visibility check (brew install glab / winget install GLab.GLab / pacman -S glab)

Cargo subcommands (install via cargo install):

  • cargo-nextest (required for cargo xtask test and cargo xtask validate): cargo install cargo-nextest --locked

  • cargo-deny (required for the cargo deny check step of validate; warn-only if missing): cargo install cargo-deny --locked

  • sqlx-cli (optional, for running migrations manually): cargo install sqlx-cli --no-default-features --features postgres

  • sccache (optional, caches rustc output to speed up clean and cross-checkout builds): pacman -S sccache / cargo install sccache --locked — see Build caching with sccache (optional)

Not required on the host:

  • Node.js / Playwright — E2E tests run inside the canopy-e2e container. Install Node.js only if you want to build the Antora docs site locally with npx antora.

  • psql — the seed tool loads SQL via docker exec into the postgres container.

Commit signing

Commit signing is required; the pre-push hook and CI both enforce it.

# GPG (default)
git config --global user.name  "Your Name"
git config --global user.email "your.email@dhs.ga.gov"
git config --global user.signingkey <KEY_ID>
git config --global commit.gpgsign true

# or SSH / EdDSA
git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub
git config --global commit.gpgsign true

Upload the public half to GitLab under Profile → SSH Keys with usage Signing (or GPG Keys for GPG).

Quick Start

# Clone the repository
git clone https://gitlab.com/gadhs/application/eligibility/canopy.git
cd canopy

# Activate pre-push hooks
git config core.hooksPath .githooks

# Configure commit signing
git config user.email "your.email@dhs.ga.gov"
git config user.signingkey YOUR_KEY_FINGERPRINT
git config commit.gpgsign true

# Copy and configure environment
cp .env.example .env

# Optional — explicitly start the devstack. `cargo xtask test`, `e2e`, and
# `validate` will auto-start it on first run if no .devstack/ markers exist.
cargo xtask dev start

# Run the test suite (cold-starts the devstack if needed)
cargo xtask test

# Run pre-push validation (preflight + fmt + clippy + nextest + docker build)
cargo xtask validate

The devstack script builds all Docker images, starts infrastructure, creates per-service databases, loads Keycloak realm configuration, and waits for all services to become healthy.

Repository Structure

canopy/
├── .claude/                   # Claude Code directives and convention docs
│   ├── CLAUDE.md              # Project context — read first
│   └── docs/                  # Tier 1-3 convention documents
├── .githooks/                 # Pre-push hook (runs cargo xtask validate)
├── .gitlab-ci.yml             # CI/CD pipeline
├── crates/                    # Shared crates (used by all services)
│   ├── canopy-api/            # Axum server builder, health/metrics
│   ├── canopy-auth/           # Keycloak JWT validation
│   ├── canopy-common/         # Settings, errors, IDs, pagination
│   ├── canopy-db/             # PostgreSQL pool and migrations
│   ├── canopy-mq/             # RabbitMQ publisher/subscriber
│   ├── canopy-reference/      # Domain enums, FIPS codes
│   ├── canopy-store/          # S3-compatible object storage
│   └── canopy-test-lib/       # Integration test harness
├── devstack/                  # Docker infrastructure configs
│   ├── garage/                # S3-compatible object storage
│   ├── grafana/               # Monitoring dashboards
│   ├── keycloak/              # Identity provider (realm config)
│   ├── postgres/              # Database init scripts
│   ├── prometheus/            # Metrics collection
│   └── rabbitmq/              # Message broker
├── docs/                      # Antora documentation site
│   └── modules/ROOT/pages/
│       ├── adrs/              # Architecture Decision Records
│       ├── plans/             # Implementation plans
│       ├── roadmap.adoc       # Phased implementation roadmap
│       └── ...
├── rulesets/                  # JDM eligibility rulesets (per jurisdiction)
│   ├── federal/               # Federal parameters (FPL, allotments)
│   └── georgia/               # Georgia-specific rules + jurisdiction.toml
├── services/                  # 19 independent service binaries
│   ├── canopy-appeals/
│   ├── canopy-applications/
│   ├── canopy-caps/
│   ├── canopy-eligibility/
│   ├── canopy-enrollment/
│   ├── canopy-exchange/
│   ├── canopy-medicaid/
│   ├── canopy-notices/
│   ├── canopy-persons/
│   ├── canopy-portal/         # Applicant-facing BFF (Dioxus 0.7 fullstack + Fluent i18n)
│   ├── canopy-renewals/
│   ├── canopy-reporting/
│   ├── canopy-rules/
│   ├── canopy-security/
│   ├── canopy-snap/
│   ├── canopy-tanf/
│   ├── canopy-verification/
│   ├── canopy-web/            # Worker-facing BFF (htmx + Alpine.js)
│   └── canopy-wic/
├── tools/
│   └── canopy-seed/           # Deterministic test data generator
├── xtask/                     # Build automation (cargo xtask)
├── Cargo.toml                 # Workspace manifest
├── docker-compose.yml         # Devstack orchestration
└── Dockerfile                 # Multi-stage production build

For a per-service breakdown — ports, databases, restricted-data scope, events, and links to each service’s API and data-model pages — see the Service Catalog.

Configuration

Canopy uses layered YAML configuration (ADR-012) with SOPS-encrypted secrets at rest (ADR-017). For each service the loader resolves values in this order, lowest to highest precedence:

  1. config/{service}/default.yaml — checked-in baseline

  2. config/{service}/site.yaml — environment overlay (CANOPY_ENV, default dev)

  3. CANOPY_{SERVICE}__* environment variables (top precedence)

Secrets (signing keys, internal API key, FTI DB URLs, AES-256-GCM SSN encryption key) live in secrets/dev.yaml — SOPS-encrypted with age (X25519 + ChaCha20-Poly1305). Decryption happens at deploy time / cargo xtask dev start and the values inject as CANOPY_{SERVICE}__* env vars consumed by the existing EnvSecretProvider (audit-logged per access).

First run on a fresh checkout

# 1. Generate your developer age keypair (one-time per dev machine).
#    No host install of age/sops needed — they ship in the
#    canopy-devtools compose service.
cargo xtask secrets init

# 2. The output prints your public age key (age1...). Open an MR adding
#    that line to `.sops.yaml` recipients. An existing recipient runs:
#       cargo xtask secrets add-recipient age1...
#    to re-encrypt the data key for you.

# 3. Once your key is in `.sops.yaml`, you can decrypt and view secrets:
cargo xtask secrets edit       # opens $EDITOR on a decrypted view
cargo xtask secrets decrypt    # prints flat dotenv stream

# 4. Bring up devstack — sops decryption + injection happens automatically.
cargo xtask dev start

For CI: paste the output of cargo xtask secrets init --for-ci (the private age key) into the GitLab masked variable CANOPY_CI_AGE_KEY, and add the public key to .sops.yaml.

Common Tasks

Managing the devstack

All container lifecycle goes through cargo xtask dev — never run docker compose directly (raw compose calls bypass the coordinated restart ordering and produce partial-JWKS-state cascades). Containers are named canopy-{service} on a single canopy-net network.

cargo xtask dev start                          # Start all services (full profile)
cargo xtask dev start --infra-only             # Infra only (no Canopy services)
cargo xtask dev start --profile snap-only      # SNAP UAT deployment (19 services)
cargo xtask dev start --shared-db              # Single postgres (saves 5 containers)
cargo xtask dev start --profile snap-only --shared-db  # SNAP + shared DB
cargo xtask dev stop                           # Stop all containers, preserve data
cargo xtask dev reload --shared-db             # Rebuild + restart, preserving data
cargo xtask dev restart --shared-db            # Wipe all data, rebuild from scratch
cargo xtask dev refresh                        # Auto-detect changes, minimum rebuild
cargo xtask dev clean --confirm                # Stop + wipe all volumes (destructive)
cargo xtask dev status                         # Running services + staleness report
cargo xtask dev logs [service]                 # Follow container logs

The devstack provides PostgreSQL (shared + 5 program-specific), RabbitMQ, Keycloak, Garage (S3), Redis, and — under the observability profile — Prometheus (9090) and Grafana (3000). For the canonical port map and the note that host-published ports are OS-ephemeral (discover them with cargo xtask dev status or .ports.env), see the Service Catalog.

Deployment profiles (ADR-005) select which program services start: snap-only (19 services, SNAP UAT), tanf-only (16), medicaid-chip (17, includes exchange), caps-only / wic-only (14 each), and full (all 29, the default). Infrastructure services — postgres, rabbitmq, keycloak, garage, redis, canopy-rules, canopy-persons, canopy-applications, canopy-eligibility, canopy-security, canopy-web, canopy-verification — run in every profile.

Staleness guard

After dev start, dev reload, or dev restart, xtask writes SHA-256 marker files to .devstack/ tracking source code, dependencies, Dockerfile, compose config, rulesets, migrations, and seed source. Before integration or E2E tests it checks these markers and performs the minimum Docker action needed:

  • Nothing changed → skip Docker entirely

  • Source / rulesets / config changed → cached rebuild (docker compose up -d --build)

  • Cargo.toml / Cargo.lock / Dockerfile changed → no-cache rebuild

  • Existing migration modified or deleted → volume wipe + rebuild

  • Seed source or rulesets changed → reseed after rebuild

cargo xtask dev refresh runs the auto-detection without running tests; cargo xtask dev status shows the report. Pass --no-refresh to test or e2e to skip the auto-check.

Running a single service

# Start only the devstack infrastructure (no Canopy services)
cargo xtask dev start --infra-only

# Run one service in development mode
cargo run -p canopy-persons

Running tests

# Full test battery (fmt + clippy + nextest)
cargo xtask test

# Unit tests only (no devstack needed)
cargo nextest run --workspace --lib

# Integration tests (requires devstack)
cargo nextest run --workspace --profile integration

# Single service tests
cargo nextest run -p canopy-snap

# E2E tests (requires devstack + services running)
cargo xtask e2e

The E2E suite (~103 Playwright tests) covers auth, dashboard, case search/detail, applications, renewals, navigation, RBAC, light + dark accessibility, workflow guidance, and the CAPS + WIC tabs. Node.js and the Playwright browsers are not needed on the host — the suite runs inside the canopy-e2e container.

Seeding test data

After starting the devstack, populate the databases with realistic, deterministic test data:

cargo xtask seed                           # Random seed, 50 households
cargo xtask seed --seed 42                 # Deterministic (reproducible)
cargo xtask seed --households 50           # More data
cargo xtask seed --jurisdiction georgia    # Explicit jurisdiction (default)
# (#1142: every `cargo xtask seed` run dynamically resets the service DBs
# first — seeded AND runtime tables — so a reseed always yields a fully
# consistent universe; the old `--reset` flag is gone.)

The seed tool (tools/canopy-seed) generates deterministic SQL plus a TypeScript manifest (tests/e2e/lib/seed.ts) for the Playwright suite — the same --seed value yields identical output. SQL files are written to test-results/seed/ and loaded into the databases via docker exec into the postgres container (no host psql required).

Working with migrations

# Create a new migration
sqlx migrate add -r create_snap_tables --source services/canopy-snap/migrations

# Run pending migrations (done automatically by devstack)
sqlx migrate run --source services/canopy-snap/migrations --database-url postgres://canopy:canopy@localhost/canopy_snap

Working with rulesets

Rulesets are versioned JDM files in rulesets/{jurisdiction}/. See ADR-003 and ADR-006 for the organization model.

# Validate all rulesets parse correctly
cargo nextest run -p canopy-rules --lib

# Hot-reload a ruleset in a running canopy-rules instance
curl -X PUT http://localhost:8001/v1/rulesets/snap-eligibility \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @rulesets/georgia/snap-eligibility.json

Building Docker images

# Build all service images
cargo xtask validate

# Build the shared service image (the root Dockerfile builds ALL service
# binaries into one image; compose selects each service's binary via
# `command:` — there is no per-service SERVICE build-arg)
docker build -t canopy:dev .

# Real build args: CARGO_FEATURES (default empty) and CARGO_PROFILE
# (default release; the devstack passes `devstack` for fast opt-0 builds)
docker build --build-arg CARGO_PROFILE=devstack -t canopy:dev .

Build caching with sccache (optional)

Host-side Rust builds — the compile step of cargo xtask test / validate, cargo nextest, and cargo build — can be sped up with sccache, which caches rustc compilation outputs and reuses them across clean builds, branch switches, CI, and other checkouts of this repo or sibling Rust projects sharing the same dependencies (axum, tokio, sqlx, …). It is purely optional and changes no build output.

pacman -S sccache            # or: cargo install sccache --locked

Enable it for all cargo builds. This lives in cargo’s config (read by cargo, so it is shell-independent):

# ~/.cargo/config.toml
[build]
rustc-wrapper = "sccache"

The cache defaults to 10 GiB under ~/.cache/sccache; that is usually enough, since sccache stores compressed, content-addressed, de-duplicated compile units (far smaller than a target/ tree). To resize it without a shell-specific environment variable, use sccache’s own config file — the sccache server reads it at startup, so it applies identically from any shell, SSH, or CI:

# ~/.config/sccache/config
[cache.disk]
size = 32212254720   # bytes; 30 GiB

Apply a size change with sccache --stop-server (it relaunches on the next build); inspect hit rates with sccache --show-stats.

WARNING

Do not set a single shared CARGO_TARGET_DIR (or build.target-dir) across multiple repos to reuse compiled dependencies. Canopy and its sibling repos each define a package named xtask, so a shared flat target directory makes them all emit target/debug/xtask — whichever repo built last wins, and cargo xtask <cmd> then silently runs another repo’s binary (symptom: error: unrecognized subcommand). A shared target directory also serialises concurrent builds on one build lock (Blocking waiting for file lock on build directory). Let each repo keep its own ./target and use sccache for cross-project dependency reuse instead: sccache shares the compile cache, not the target directory, so there are no binary collisions or lock contention.

NOTE
sccache cannot cache incremental compilation, so it helps dependency builds (always non-incremental) and clean / CI builds the most. With profile.dev incremental left on, your own crates still rebuild incrementally (sccache passes those through uncached); set CARGO_INCREMENTAL=0 (e.g. in CI) to make them cacheable too.

Working with documentation

# Build the Antora site locally (requires Node.js)
npx antora antora-playbook.yml

# Verify Tier 1 doc integrity
cargo xtask check-docs

# Fix Tier 1 docs (sync from template)
cargo xtask check-docs --fix --yes

CI vs local differences

  • CI uses Docker-in-Docker (DinD) — services are reachable at hostname docker, not localhost.

  • CI runs nextest with --profile ci (JUnit XML output, thread limits).

  • The pre-push hook runs cargo xtask validate; CI additionally runs the integration + E2E suites and security scans.

Coding Conventions

SPDX Headers

Every new .rs file must include the license header:

// SPDX-License-Identifier: AGPL-3.0-or-later

Error Handling

Use canopy_common::ApiError for all API error responses. ApiError implements RFC 9457 Problem Details and returns structured JSON:

{
  "type": "about:blank",
  "title": "Not Found",
  "status": 404,
  "detail": "Person with id 01942a3b-... not found"
}

Map internal errors to ApiError variants — never expose internal error messages, stack traces, or database details to API consumers.

Database Patterns

  • UUID v7 primary keys (time-ordered)

  • TEXT columns with CHECK constraints for enums (not PostgreSQL enum types)

  • TIMESTAMPTZ for all date/time columns

  • _cents suffix for monetary amounts stored as integers (never floating point)

  • Index all foreign key columns and columns used in WHERE clauses

  • Per ADR-001: no cross-database foreign keys

Event Publishing

  • Routing key = event type (e.g., snap.determination_completed)

  • Payload = IDs and statuses only — no PII, no income data, no restricted federal data

  • Use canopy_mq::Publisher — do not construct AMQP messages directly

  • Always publish after the database transaction commits (not inside the transaction)

Git Workflow

  • Branch naming: feature/<name>, fix/<name>, chore/<name>

  • Commit messages: imperative mood, focused on "why" not "what"

  • All commits signed (GPG/EdDSA)

  • MRs require at least one approval

  • Pre-push hook runs cargo xtask validate — do not bypass with --no-verify

See docs/modules/standards/pages/git-workflow.adoc for the complete branching and commit conventions.

Architecture Overview

Canopy is a microservice-based integrated eligibility system governed by six ADRs:

ADR Summary

ADR-001

Each benefit program is an independent service with its own PostgreSQL database

ADR-002

Program services return signed JWS determinations, never raw data

ADR-003

All eligibility logic in versioned JDM files evaluated by canopy-rules

ADR-004

FTI, IEVS, SSA SOLQ/BINDEX isolated to authorized program services

ADR-005

Any jurisdiction deploys any program subset via Docker Compose profiles

ADR-006

Rulesets organized by jurisdiction with shared federal parameters

The first four ADRs are driven by federal data-use law rather than engineering preference. Program-service isolation (ADR-001) exists because IRS Pub 1075, HIPAA, Computer Matching Agreements, and FNS IEVS restrictions cannot be satisfied by a shared schema — no cross-program database access is permitted. The black-box determination contract (ADR-002) means the eligibility orchestrator (canopy-eligibility) submits application contexts and receives signed JWS determination objects; it never queries program databases or touches restricted federal data. Ruleset-as-data (ADR-003) keeps all eligibility logic in versioned JDM files evaluated by the shared canopy-rules service, so a threshold change is a data change with no code deployment. Legally-scoped data tenancy (ADR-004) isolates each federal source (FTI, IEVS, SSA SOLQ/BINDEX, FDSH) to the program services statutorily authorized to use it, with FTI audit logs kept separate from application audit logs.

For the canonical per-service topology — listen ports, databases, restricted-data scope, and events — see the Service Catalog. For the shared crates (canopy-common, canopy-auth, canopy-db, canopy-mq, canopy-store, canopy-api, canopy-reference, canopy-signing, canopy-rules-client, canopy-typst, canopy-test-lib) see the Shared Crates Reference. Document generation (Typst PDF rendering of notices, forms, and reports) is described under canopy-typst there and in ADR-010. The event bus (RabbitMQ topic exchange canopy.events, with the per-service event_outbox table and OutboxDrainer per ADR-018) is documented in the Service Catalog.

Database topology in local development

Per ADR-001, program services are legally isolated — each program’s data lives in its own PostgreSQL instance to enforce federal data tenancy. Infrastructure services share a single instance. cargo xtask dev start reproduces this production topology with six PostgreSQL containers:

Container Port Databases

postgres (shared)

5432

14 databases: canopy_rules, canopy_persons, canopy_applications, canopy_eligibility, canopy_verification, canopy_enrollment, canopy_renewals, canopy_notices, canopy_exchange, canopy_appeals, canopy_reporting, canopy_security, canopy_web, canopy_portal

postgres-snap

5433

canopy_snap

postgres-tanf

5434

canopy_tanf

postgres-medicaid

5435

canopy_medicaid

postgres-caps

5436

canopy_caps

postgres-wic

5437

canopy_wic

The --shared-db flag (cargo xtask dev start --shared-db) consolidates every database onto the single shared instance on port 5432, saving five containers' worth of memory. Use it for local work when you do not need to exercise per-program DB isolation — not for production or for integration testing of ADR-001 compliance. No cross-database queries or foreign data wrappers are permitted between program databases in either mode.

Session storage differs between the two BFFs. For canopy-web (worker portal), PostgreSQL remains the authoritative session store and Redis 7 (Alpine, port 6379) is a read-through LRU cache (128 MB maxmemory, allkeys-lru eviction, AOF persistence). For canopy-portal (applicant portal), Redis is the primary session store: per ADR-026 the portal is Postgres-free for sessions and uses opaque-token sessions held in a dedicated redis-sessions instance configured with noeviction (sessions must never be silently dropped). The portal has no tower-sessions Postgres session store, and its canopy_portal database (created on the shared instance above) is not used for sessions.

See the Implementation Guide for detailed technical specifications and the Roadmap for the phased delivery plan.

Edit this page · default