Scaling & Deployment
On this page
Overview
Canopy uses Docker Compose with deployment profiles (ADR-005) to support modular deployments. Any jurisdiction can deploy any subset of benefit programs. This runbook covers starting the devstack, scaling services, deploying updates, and rolling back.
Docker Compose Profiles
Canopy defines the following deployment profiles:
| Profile | Services Included |
|---|---|
|
Shared infrastructure (postgres, rabbitmq, keycloak, garage) + canopy-rules, canopy-persons, canopy-applications, canopy-eligibility, canopy-snap, canopy-enrollment, canopy-renewals, canopy-notices, canopy-appeals, canopy-reporting, canopy-security, canopy-verification, canopy-web |
|
Shared infrastructure + canopy-rules, canopy-persons, canopy-applications, canopy-eligibility, canopy-tanf, canopy-notices, canopy-appeals, canopy-reporting, canopy-security, canopy-web |
|
Shared infrastructure + canopy-rules, canopy-persons, canopy-applications, canopy-eligibility, canopy-medicaid, canopy-notices, canopy-appeals, canopy-reporting, canopy-security, canopy-web |
|
All services across all programs (SNAP, TANF, Medicaid/CHIP, CAPS, WIC stubs) |
|
Each program service gets its own PostgreSQL instance (postgres-snap, postgres-tanf, postgres-medicaid). This is the production-recommended topology per ADR-001/ADR-004. |
Starting the Devstack
Basic Startup
# Start with the default profile (full) and shared database
cargo xtask dev start --shared-db
# Start with a specific profile
cargo xtask dev start --profile snap-only
# Start with isolated databases (production topology)
cargo xtask dev start --profile isolated-db
The --shared-db flag collapses all PostgreSQL instances into a single container with multiple databases.
This saves resources during local development but does not reflect production isolation boundaries.
Refreshing After Code Changes
When you modify source code, templates, or configuration:
cargo xtask dev refresh
dev refresh auto-detects which services have changed and performs the minimum rebuild necessary:
-
Rust source changes: rebuilds affected service containers
-
Migration changes: re-runs migrations
-
Ruleset changes: restarts canopy-rules to reload JDM files
-
Template/asset changes: rebuilds the affected BFF container
-
Docker Compose config changes: recreates affected containers
This is faster than a full dev stop && dev start cycle.
Environment Variable Reference
Each Canopy service is configured via environment variables.
The naming convention is CANOPY_{SERVICE}__{SETTING} (double underscore separates service from setting).
Common Variables (All Services)
| Variable | Example | Description |
|---|---|---|
|
|
HTTP listen port for the service |
|
|
PostgreSQL connection string (service-specific database name) |
|
|
RabbitMQ connection string for event publishing/consuming |
|
Keycloak base URL for JWKS fetching and token validation |
|
|
|
Active jurisdiction (selects |
|
Base URL of the canopy-rules service (used by program services for ruleset evaluation) |
Per-Service Port Assignments
| Service | Default Port |
|---|---|
canopy-rules |
3001 |
canopy-persons |
3002 |
canopy-applications |
3003 |
canopy-eligibility |
3004 |
canopy-snap |
3010 |
canopy-tanf |
3011 |
canopy-medicaid |
3012 |
canopy-enrollment |
3020 |
canopy-renewals |
3021 |
canopy-notices |
3022 |
canopy-appeals |
3023 |
canopy-reporting |
3024 |
canopy-security |
3025 |
canopy-verification |
3026 |
canopy-exchange |
3027 |
canopy-web |
8080 |
canopy-portal |
8090 |
Database URLs by Instance
| Instance | DATABASE_URL Pattern |
|---|---|
Shared postgres |
|
postgres-snap |
|
postgres-tanf |
|
postgres-medicaid |
|
Horizontal Scaling
Individual services can be scaled horizontally using Docker Compose’s --scale flag.
All Canopy services are stateless (session state lives in PostgreSQL) and safe to run as multiple instances behind a load balancer.
Scaling a Service
# Scale canopy-snap to 3 instances
docker compose --profile full up -d --scale canopy-snap=3
# Scale multiple services
docker compose --profile full up -d \
--scale canopy-eligibility=2 \
--scale canopy-snap=3 \
--scale canopy-persons=2
Scaling Considerations
-
canopy-rules: Safe to scale. Each instance loads rulesets independently. Consider scaling if ruleset evaluation latency increases under load.
-
canopy-eligibility: Orchestrates parallel calls to program services. Scale if request queuing is observed.
-
Program services (canopy-snap, canopy-tanf, canopy-medicaid): Scale independently based on program-specific load.
-
canopy-web / canopy-portal: BFF services are lightweight; scale if concurrent user sessions increase.
-
canopy-security: Subscribes to RabbitMQ with wildcard
#binding. Multiple instances will compete for messages (each event processed once). Scale for throughput, not redundancy. -
PostgreSQL: Not horizontally scalable via
--scale. Use read replicas or connection pooling (pgBouncer) for database scaling. -
RabbitMQ: Not horizontally scalable via
--scale. Use RabbitMQ clustering for HA.
Rolling Deployment
Image references (ADR-040)
The repo’s docker-compose.yml is the dev stack — every canopy service uses build: (compiled from source) and carries no image: ref, so a bare docker compose up -d never pulls a deployable artifact. Deployments run from the registry instead, via the in-repo prebuilt override (docker-compose.prebuilt.yml, #1073):
export COMPOSE_FILE=docker-compose.yml:docker-compose.prebuilt.yml
export CANOPY_PREBUILT_IMAGES=true
export CANOPY_PREBUILT_SERVICE_IMAGE="$CI_REGISTRY_IMAGE:${IMAGE_TAG}"
export CANOPY_PREBUILT_PORTAL_IMAGE="$CI_REGISTRY_IMAGE/portal:${IMAGE_TAG}"
IMAGE_TAG is one of the ADR-040 production refs: an immutable :<short-sha> (preferred — what promotion minted), a release :<tag>, or :latest. All 18 service entries point at the ONE shared service image (the root Dockerfile builds every service binary; each compose service selects its own command:); canopy-portal is the second image.
Procedure
Deploy updates with zero downtime by rolling through services one at a time (with the prebuilt override exported as above).
#!/usr/bin/env bash
set -euo pipefail
IMAGE_TAG="${1:?Usage: deploy.sh <image-tag>}"
export COMPOSE_FILE=docker-compose.yml:docker-compose.prebuilt.yml
export CANOPY_PREBUILT_IMAGES=true
export CANOPY_PREBUILT_SERVICE_IMAGE="${CI_REGISTRY_IMAGE:?}:${IMAGE_TAG}"
export CANOPY_PREBUILT_PORTAL_IMAGE="${CI_REGISTRY_IMAGE:?}/portal:${IMAGE_TAG}"
SERVICES=(
canopy-rules
canopy-persons
canopy-applications
canopy-snap
canopy-tanf
canopy-medicaid
canopy-eligibility
canopy-enrollment
canopy-renewals
canopy-notices
canopy-appeals
canopy-reporting
canopy-security
canopy-verification
canopy-web
)
for SERVICE in "${SERVICES[@]}"; do
echo "Deploying ${SERVICE} with image tag ${IMAGE_TAG}..."
# Update the service with the new image (pulled from the prebuilt
# override's registry ref — the base compose file has no image refs)
docker compose up -d --no-deps --pull always "${SERVICE}"
# Wait for health check to pass
echo "Waiting for ${SERVICE} health check..."
RETRIES=30
while [ "${RETRIES}" -gt 0 ]; do
STATUS=$(docker inspect --format='{{.State.Health.Status}}' \
"$(docker compose ps -q "${SERVICE}" | head -1)" 2>/dev/null || echo "starting")
if [ "${STATUS}" = "healthy" ]; then
echo "${SERVICE} is healthy."
break
fi
RETRIES=$((RETRIES - 1))
sleep 2
done
if [ "${RETRIES}" -eq 0 ]; then
echo "ERROR: ${SERVICE} failed health check. Aborting deployment."
echo "Run rollback procedure to restore previous version."
exit 1
fi
done
echo "All services deployed successfully."
Signed-field emission gates (ADR-028 verifier-tolerant-first)
The roll order above rebuilds emitters (canopy-snap) BEFORE the verifier (canopy-eligibility). Any release that adds
fields to the signed SignableDetermination therefore ships them tolerant-first: the struct lands fleet-wide with
emission behind a default-off gate, and the gate flips ONLY after every verifier runs the tolerant build — an old
verifier reconstructing canonical bytes without the new fields would quarantine every valid determination minted with
them.
Current gates:
-
CANOPY_SNAP__EMIT_POLICY_ATTESTATION(#1467, ADR-028 Amendment 6) — bindspolicy_target+evaluated_as_ofinto snap envelopes. Config defaultfalse. Flip AFTER the whole fleet (canopy-eligibility above all) runs a build ≥ #1467, then restart canopy-snap. The devstack compose sets ittrue(single atomic deploy). Rollback of canopy-eligibility to a pre-#1467 build REQUIRES flipping this off first.
Parameter-set cutover check (#1467, every October 1)
canopy-snap fails determinations closed (422) when no snap-cola parameter set is in force at the evaluation date — i.e. October 1 arrives and the new FY files are not staged. Before each federal fiscal year:
-
Stage the new
snap-{allotments,deductions,income-limits}-<yr>.jsonBESIDE the old ones (seerulesets/federal/README.adoc— complete triple, agreeing_fiscal_year, duplicated values aligned). -
cargo xtask policy audit --source federalmust pass (grace 0 for snap-cola). -
Restart canopy-snap; the boot inventory log lines (
snap parameter set loaded) must show the new window. -
Emergency relief valve if the files cannot land in time:
CANOPY_SNAP__ALLOW_EXPIRED_PARAM_SET=trueserves the newest EXPIRED set — wrong benefits by design, error-logged per use, remove it the moment the files land.
Health Check Verification
Every Canopy service exposes a GET /healthz endpoint that returns 200 OK when the service is ready to accept traffic.
# Check all services
for PORT in 3001 3002 3003 3004 3010 3011 3012 3020 3021 3022 3023 3024 3025 3026 8080; do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:${PORT}/healthz")
echo "Port ${PORT}: ${STATUS}"
done
All endpoints must return 200.
A 503 indicates the service is starting or unhealthy.
Deployment Order
Deploy services in dependency order:
-
Infrastructure: PostgreSQL, RabbitMQ, Keycloak (managed separately)
-
Foundation: canopy-rules (rulesets must be available before program services start)
-
Data services: canopy-persons, canopy-applications
-
Program services: canopy-snap, canopy-tanf, canopy-medicaid
-
Orchestrator: canopy-eligibility (depends on program services)
-
Downstream services: canopy-enrollment, canopy-renewals, canopy-notices, canopy-appeals, canopy-reporting
-
Security: canopy-security (audit subscriber)
-
BFF layer: canopy-web, canopy-portal
Rollback
If a deployment fails or introduces a regression, roll back to the previous version.
ADR-040 refs are immutable per commit — rollback is a redeploy of the previous known-good :<short-sha>, never a "hope the old image is still cached" restart.
Quick Rollback
# Stop all services
docker compose down
# Redeploy the previous known-good immutable ref through the same
# prebuilt override (see "Image references" above)
CANOPY_PREBUILT_SERVICE_IMAGE="$CI_REGISTRY_IMAGE:<previous-short-sha>" \
CANOPY_PREBUILT_PORTAL_IMAGE="$CI_REGISTRY_IMAGE/portal:<previous-short-sha>" \
docker compose up -d
Per-Service Rollback
To roll back a single service without affecting others:
# Roll back canopy-snap to the previous image ref. Requires the "Image
# references" export block active (the override interpolates BOTH image
# vars whole-file, so CANOPY_PREBUILT_PORTAL_IMAGE must be set too).
CANOPY_PREBUILT_SERVICE_IMAGE="$CI_REGISTRY_IMAGE:<previous-short-sha>" \
docker compose up -d --no-deps --pull always canopy-snap
# Verify health
curl -s http://localhost:3010/healthz
Database Rollback Considerations
-
If the failed deployment included database migrations, a restore from backup may be necessary. See the Database Backup & Restore runbook.
-
Canopy migrations are forward-only (no
downmigrations). Rolling back a migration requires restoring from a pre-migration backup.
Post-Deployment Verification
After every deployment or scaling change, verify the environment:
# Run the full test battery against the running services
cargo xtask test
Verification Checklist
-
All
/healthzendpoints return200 OK. -
All
/metricsendpoints are responding (Prometheus scrape targets). -
cargo xtask testpasses all unit and integration tests. -
Sample eligibility determination returns expected results via canopy-eligibility API.
-
RabbitMQ management UI shows all queues with active consumers.
-
canopy-web dashboard loads and case search returns results.
-
canopy-security audit events are being persisted (check recent entries).
Smoke Test
# Quick smoke test: create a person, submit an application, run determination
curl -s http://localhost:3002/api/v1/persons \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${TOKEN}" \
-d '{"first_name":"Test","last_name":"User","date_of_birth":"1990-01-01","ssn":"000-00-0000"}'
# Verify the event was published to RabbitMQ
# Check canopy-security for the corresponding audit entry