Scaling & Deployment

On this page

Overview

Canopy uses Docker Compose with deployment profiles (ADR-005) to support modular deployments. Any jurisdiction can deploy any subset of benefit programs. This runbook covers starting the devstack, scaling services, deploying updates, and rolling back.

Docker Compose Profiles

Canopy defines the following deployment profiles:

Profile Services Included

snap-only

Shared infrastructure (postgres, rabbitmq, keycloak, garage) + canopy-rules, canopy-persons, canopy-applications, canopy-eligibility, canopy-snap, canopy-enrollment, canopy-renewals, canopy-notices, canopy-appeals, canopy-reporting, canopy-security, canopy-verification, canopy-web

tanf-only

Shared infrastructure + canopy-rules, canopy-persons, canopy-applications, canopy-eligibility, canopy-tanf, canopy-notices, canopy-appeals, canopy-reporting, canopy-security, canopy-web

medicaid-chip

Shared infrastructure + canopy-rules, canopy-persons, canopy-applications, canopy-eligibility, canopy-medicaid, canopy-notices, canopy-appeals, canopy-reporting, canopy-security, canopy-web

full

All services across all programs (SNAP, TANF, Medicaid/CHIP, CAPS, WIC stubs)

isolated-db

Each program service gets its own PostgreSQL instance (postgres-snap, postgres-tanf, postgres-medicaid). This is the production-recommended topology per ADR-001/ADR-004.

Starting the Devstack

Basic Startup

# Start with the default profile (full) and shared database
cargo xtask dev start --shared-db

# Start with a specific profile
cargo xtask dev start --profile snap-only

# Start with isolated databases (production topology)
cargo xtask dev start --profile isolated-db

The --shared-db flag collapses all PostgreSQL instances into a single container with multiple databases. This saves resources during local development but does not reflect production isolation boundaries.

Refreshing After Code Changes

When you modify source code, templates, or configuration:

cargo xtask dev refresh

dev refresh auto-detects which services have changed and performs the minimum rebuild necessary:

  • Rust source changes: rebuilds affected service containers

  • Migration changes: re-runs migrations

  • Ruleset changes: restarts canopy-rules to reload JDM files

  • Template/asset changes: rebuilds the affected BFF container

  • Docker Compose config changes: recreates affected containers

This is faster than a full dev stop && dev start cycle.

Environment Variable Reference

Each Canopy service is configured via environment variables. The naming convention is CANOPY_{SERVICE}__{SETTING} (double underscore separates service from setting).

Common Variables (All Services)

Variable Example Description

CANOPY_{SERVICE}__PORT

3000

HTTP listen port for the service

DATABASE_URL

postgres://canopy:canopy@postgres:5432/canopy_rules

PostgreSQL connection string (service-specific database name)

RABBITMQ_URL

amqp://canopy:canopy@rabbitmq:5672

RabbitMQ connection string for event publishing/consuming

KEYCLOAK_URL

http://keycloak:8080

Keycloak base URL for JWKS fetching and token validation

JURISDICTION

georgia

Active jurisdiction (selects rulesets/{jurisdiction}/ and jurisdiction.toml)

RULES_URL

http://canopy-rules:3000

Base URL of the canopy-rules service (used by program services for ruleset evaluation)

Per-Service Port Assignments

Service Default Port

canopy-rules

3001

canopy-persons

3002

canopy-applications

3003

canopy-eligibility

3004

canopy-snap

3010

canopy-tanf

3011

canopy-medicaid

3012

canopy-enrollment

3020

canopy-renewals

3021

canopy-notices

3022

canopy-appeals

3023

canopy-reporting

3024

canopy-security

3025

canopy-verification

3026

canopy-exchange

3027

canopy-web

8080

canopy-portal

8090

Database URLs by Instance

Instance DATABASE_URL Pattern

Shared postgres

postgres://canopy:canopy@postgres:5432/{database_name}

postgres-snap

postgres://canopy:canopy@postgres-snap:5432/canopy_snap

postgres-tanf

postgres://canopy:canopy@postgres-tanf:5432/canopy_tanf

postgres-medicaid

postgres://canopy:canopy@postgres-medicaid:5432/canopy_medicaid

Horizontal Scaling

Individual services can be scaled horizontally using Docker Compose’s --scale flag. All Canopy services are stateless (session state lives in PostgreSQL) and safe to run as multiple instances behind a load balancer.

Scaling a Service

# Scale canopy-snap to 3 instances
docker compose --profile full up -d --scale canopy-snap=3

# Scale multiple services
docker compose --profile full up -d \
  --scale canopy-eligibility=2 \
  --scale canopy-snap=3 \
  --scale canopy-persons=2

Scaling Considerations

  • canopy-rules: Safe to scale. Each instance loads rulesets independently. Consider scaling if ruleset evaluation latency increases under load.

  • canopy-eligibility: Orchestrates parallel calls to program services. Scale if request queuing is observed.

  • Program services (canopy-snap, canopy-tanf, canopy-medicaid): Scale independently based on program-specific load.

  • canopy-web / canopy-portal: BFF services are lightweight; scale if concurrent user sessions increase.

  • canopy-security: Subscribes to RabbitMQ with wildcard # binding. Multiple instances will compete for messages (each event processed once). Scale for throughput, not redundancy.

  • PostgreSQL: Not horizontally scalable via --scale. Use read replicas or connection pooling (pgBouncer) for database scaling.

  • RabbitMQ: Not horizontally scalable via --scale. Use RabbitMQ clustering for HA.

Rolling Deployment

Image references (ADR-040)

The repo’s docker-compose.yml is the dev stack — every canopy service uses build: (compiled from source) and carries no image: ref, so a bare docker compose up -d never pulls a deployable artifact. Deployments run from the registry instead, via the in-repo prebuilt override (docker-compose.prebuilt.yml, #1073):

export COMPOSE_FILE=docker-compose.yml:docker-compose.prebuilt.yml
export CANOPY_PREBUILT_IMAGES=true
export CANOPY_PREBUILT_SERVICE_IMAGE="$CI_REGISTRY_IMAGE:${IMAGE_TAG}"
export CANOPY_PREBUILT_PORTAL_IMAGE="$CI_REGISTRY_IMAGE/portal:${IMAGE_TAG}"

IMAGE_TAG is one of the ADR-040 production refs: an immutable :<short-sha> (preferred — what promotion minted), a release :<tag>, or :latest. All 18 service entries point at the ONE shared service image (the root Dockerfile builds every service binary; each compose service selects its own command:); canopy-portal is the second image.

Procedure

Deploy updates with zero downtime by rolling through services one at a time (with the prebuilt override exported as above).

#!/usr/bin/env bash
set -euo pipefail

IMAGE_TAG="${1:?Usage: deploy.sh <image-tag>}"
export COMPOSE_FILE=docker-compose.yml:docker-compose.prebuilt.yml
export CANOPY_PREBUILT_IMAGES=true
export CANOPY_PREBUILT_SERVICE_IMAGE="${CI_REGISTRY_IMAGE:?}:${IMAGE_TAG}"
export CANOPY_PREBUILT_PORTAL_IMAGE="${CI_REGISTRY_IMAGE:?}/portal:${IMAGE_TAG}"
SERVICES=(
  canopy-rules
  canopy-persons
  canopy-applications
  canopy-snap
  canopy-tanf
  canopy-medicaid
  canopy-eligibility
  canopy-enrollment
  canopy-renewals
  canopy-notices
  canopy-appeals
  canopy-reporting
  canopy-security
  canopy-verification
  canopy-web
)

for SERVICE in "${SERVICES[@]}"; do
  echo "Deploying ${SERVICE} with image tag ${IMAGE_TAG}..."

  # Update the service with the new image (pulled from the prebuilt
  # override's registry ref — the base compose file has no image refs)
  docker compose up -d --no-deps --pull always "${SERVICE}"

  # Wait for health check to pass
  echo "Waiting for ${SERVICE} health check..."
  RETRIES=30
  while [ "${RETRIES}" -gt 0 ]; do
    STATUS=$(docker inspect --format='{{.State.Health.Status}}' \
      "$(docker compose ps -q "${SERVICE}" | head -1)" 2>/dev/null || echo "starting")
    if [ "${STATUS}" = "healthy" ]; then
      echo "${SERVICE} is healthy."
      break
    fi
    RETRIES=$((RETRIES - 1))
    sleep 2
  done

  if [ "${RETRIES}" -eq 0 ]; then
    echo "ERROR: ${SERVICE} failed health check. Aborting deployment."
    echo "Run rollback procedure to restore previous version."
    exit 1
  fi
done

echo "All services deployed successfully."

Signed-field emission gates (ADR-028 verifier-tolerant-first)

The roll order above rebuilds emitters (canopy-snap) BEFORE the verifier (canopy-eligibility). Any release that adds fields to the signed SignableDetermination therefore ships them tolerant-first: the struct lands fleet-wide with emission behind a default-off gate, and the gate flips ONLY after every verifier runs the tolerant build — an old verifier reconstructing canonical bytes without the new fields would quarantine every valid determination minted with them.

Current gates:

  • CANOPY_SNAP__EMIT_POLICY_ATTESTATION (#1467, ADR-028 Amendment 6) — binds policy_target + evaluated_as_of into snap envelopes. Config default false. Flip AFTER the whole fleet (canopy-eligibility above all) runs a build ≥ #1467, then restart canopy-snap. The devstack compose sets it true (single atomic deploy). Rollback of canopy-eligibility to a pre-#1467 build REQUIRES flipping this off first.

Parameter-set cutover check (#1467, every October 1)

canopy-snap fails determinations closed (422) when no snap-cola parameter set is in force at the evaluation date — i.e. October 1 arrives and the new FY files are not staged. Before each federal fiscal year:

  1. Stage the new snap-{allotments,deductions,income-limits}-<yr>.json BESIDE the old ones (see rulesets/federal/README.adoc — complete triple, agreeing _fiscal_year, duplicated values aligned).

  2. cargo xtask policy audit --source federal must pass (grace 0 for snap-cola).

  3. Restart canopy-snap; the boot inventory log lines (snap parameter set loaded) must show the new window.

  4. Emergency relief valve if the files cannot land in time: CANOPY_SNAP__ALLOW_EXPIRED_PARAM_SET=true serves the newest EXPIRED set — wrong benefits by design, error-logged per use, remove it the moment the files land.

Health Check Verification

Every Canopy service exposes a GET /healthz endpoint that returns 200 OK when the service is ready to accept traffic.

# Check all services
for PORT in 3001 3002 3003 3004 3010 3011 3012 3020 3021 3022 3023 3024 3025 3026 8080; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:${PORT}/healthz")
  echo "Port ${PORT}: ${STATUS}"
done

All endpoints must return 200. A 503 indicates the service is starting or unhealthy.

Deployment Order

Deploy services in dependency order:

  1. Infrastructure: PostgreSQL, RabbitMQ, Keycloak (managed separately)

  2. Foundation: canopy-rules (rulesets must be available before program services start)

  3. Data services: canopy-persons, canopy-applications

  4. Program services: canopy-snap, canopy-tanf, canopy-medicaid

  5. Orchestrator: canopy-eligibility (depends on program services)

  6. Downstream services: canopy-enrollment, canopy-renewals, canopy-notices, canopy-appeals, canopy-reporting

  7. Security: canopy-security (audit subscriber)

  8. BFF layer: canopy-web, canopy-portal

Rollback

If a deployment fails or introduces a regression, roll back to the previous version.

ADR-040 refs are immutable per commit — rollback is a redeploy of the previous known-good :<short-sha>, never a "hope the old image is still cached" restart.

Quick Rollback

# Stop all services
docker compose down

# Redeploy the previous known-good immutable ref through the same
# prebuilt override (see "Image references" above)
CANOPY_PREBUILT_SERVICE_IMAGE="$CI_REGISTRY_IMAGE:<previous-short-sha>" \
CANOPY_PREBUILT_PORTAL_IMAGE="$CI_REGISTRY_IMAGE/portal:<previous-short-sha>" \
docker compose up -d

Per-Service Rollback

To roll back a single service without affecting others:

# Roll back canopy-snap to the previous image ref. Requires the "Image
# references" export block active (the override interpolates BOTH image
# vars whole-file, so CANOPY_PREBUILT_PORTAL_IMAGE must be set too).
CANOPY_PREBUILT_SERVICE_IMAGE="$CI_REGISTRY_IMAGE:<previous-short-sha>" \
docker compose up -d --no-deps --pull always canopy-snap

# Verify health
curl -s http://localhost:3010/healthz

Database Rollback Considerations

  • If the failed deployment included database migrations, a restore from backup may be necessary. See the Database Backup & Restore runbook.

  • Canopy migrations are forward-only (no down migrations). Rolling back a migration requires restoring from a pre-migration backup.

Post-Deployment Verification

After every deployment or scaling change, verify the environment:

# Run the full test battery against the running services
cargo xtask test

Verification Checklist

  1. All /healthz endpoints return 200 OK.

  2. All /metrics endpoints are responding (Prometheus scrape targets).

  3. cargo xtask test passes all unit and integration tests.

  4. Sample eligibility determination returns expected results via canopy-eligibility API.

  5. RabbitMQ management UI shows all queues with active consumers.

  6. canopy-web dashboard loads and case search returns results.

  7. canopy-security audit events are being persisted (check recent entries).

Smoke Test

# Quick smoke test: create a person, submit an application, run determination
curl -s http://localhost:3002/api/v1/persons \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -d '{"first_name":"Test","last_name":"User","date_of_birth":"1990-01-01","ssn":"000-00-0000"}'

# Verify the event was published to RabbitMQ
# Check canopy-security for the corresponding audit entry
Edit this page · default