Production Deployment Guide

On this page

Overview

Canopy is a containerized microservices system. In development, it runs via Docker Compose (cargo xtask dev start). In production, it deploys to any container orchestration platform (Kubernetes, ECS, Docker Swarm) or can run as standalone Docker containers behind a load balancer.

This guide covers the production deployment architecture. For local development, see Local Development.

Prerequisites

Component Version Notes

Container runtime

Docker 24+ or Kubernetes 1.28+

All services ship as Alpine-based containers

PostgreSQL

16+

6 isolated databases per ADR-001 (or shared instance with separate DBs)

RabbitMQ

3.13+

Durable queues, management plugin recommended

Keycloak

24+

OIDC identity provider with RS256 JWT

Redis

7+

Session LRU cache for BFF services (ADR-009)

S3-compatible storage

Any

MinIO, AWS S3, Garage — for PDF notice storage

TLS certificates

—

For all public-facing endpoints

Infrastructure Requirements

Service Tier (19 application services for SNAP-only profile)

Each service is stateless — no local disk, no in-memory sessions, no sticky routing. Scale horizontally by adding replicas.

Service Group vCPU Memory Instances Notes

Program services (snap, tanf, medicaid, caps, wic)

0.5

256 MB

2+

One active per program; scale for throughput

Infrastructure services (rules, persons, applications, eligibility, enrollment, renewals, appeals, reporting, security)

0.5

256 MB

2+

Scale persons/eligibility first under load

canopy-notices

1

512 MB

2+

Handles Typst PDF rendering (CPU-bound) via canopy-typst (ADR-010); size CPU for notice render throughput

BFF services (canopy-web, canopy-portal)

0.5

512 MB

2+

Server-rendered web layer; no PDF rendering (delegated to canopy-notices)

Data Tier

Component vCPU Memory Disk Notes

PostgreSQL (per program DB)

1

1 GB

50 GB SSD

Enable WAL archiving for PITR. One instance per program per ADR-001, or shared instance with separate databases.

RabbitMQ

1

1 GB

20 GB

Durable queues. 3-node cluster for HA.

Keycloak

1

1 GB

10 GB

External PostgreSQL backend recommended for HA

Redis

0.5

128 MB

—

LRU eviction (maxmemory 128mb), AOF persistence. Session cache only — PostgreSQL is authoritative.

S3 / MinIO

0.5

512 MB

100 GB+

Versioning enabled for notice PDFs. Grows with notice volume.

Deployment Profiles (ADR-005)

Canopy supports selective program deployment via Docker Compose profiles or equivalent Kubernetes label selectors:

Profile Services Use Case

snap-only

19

SNAP UAT (September 2026 target)

tanf-only

16

TANF-only deployment

medicaid-chip

17

Medicaid + CHIP

caps-only

14

CAPS/CCDF only

wic-only

14

WIC only

full

29

All programs (default)

Services not included in a profile gracefully degrade — the eligibility orchestrator returns pending_verification for unavailable programs instead of failing.

Environment Configuration

All configuration is via environment variables following the CANOPY_{SERVICE}__{KEY} convention.

Required Variables (all services)

Variable Description Example

CANOPY_{SVC}__PORT

Service listen port

8013

CANOPY_{SVC}__DATABASE_URL

PostgreSQL connection (use ?sslmode=require in production)

postgres://user:pass@db:5432/canopy_snap?sslmode=require

CANOPY_{SVC}__RABBITMQ_URL

RabbitMQ AMQP URL (use amqps:// in production)

amqps://user:pass@rabbit:5671/%2f

CANOPY_{SVC}__KEYCLOAK_ISSUER

Public JWT issuer URL

https://auth.dhs.ga.gov/realms/canopy

CANOPY_{SVC}__KEYCLOAK_URL

Internal JWKS fetch URL (may differ from issuer in containerized deployments)

http://keycloak:8080/realms/canopy

CANOPY_{SVC}__JURISDICTION

Jurisdiction identifier for rulesets

georgia

CANOPY_ENV

Runtime environment (production or development)

production

Secrets

Variable Description

CANOPY_ENCRYPTION_KEY

AES-256-GCM key for SSN encryption. Generate: openssl rand -base64 32. Must be consistent across all services that encrypt/decrypt PII.

CANOPY_{PROGRAM}__SIGNING_KEY

ECDSA P-256 private key PEM for determination signing. Generate: cargo xtask gen-signing-keys --program snap. One per program service.

CANOPY_VERIFY_KEY_{PROGRAM}

Current public verification key PEM. Used by canopy-eligibility to verify determination signatures; retired keys are lazy-loaded from canopy-security’s signing_key_history (ADR-036 — the _PREV slot was removed).

CANOPY_INTERNAL_API_KEY

Service-to-service API key for internal endpoints (IEVS, SAVE). Same value across all services.

Store secrets in a vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault). Never commit secrets to the repository.

Optional Tuning

Variable Default Description

db_max_connections

10

PostgreSQL connection pool size per service

db_idle_timeout_secs

600

Idle connection timeout

rate_limit_rpm

6000

Rate limit per IP per minute (0 = disabled)

body_limit

2097152

Max request body size in bytes (2 MiB)

session_ttl_seconds

28800 / 1800

Session TTL (canopy-web: 8h, canopy-portal: 30min)

CANOPY_SESSION_SECURE

true

Set Secure flag on session cookies (disable only in development)

Database Setup

  1. Create databases (one per program service per ADR-001):

    CREATE DATABASE canopy_snap;
CREATE DATABASE canopy_tanf;
CREATE DATABASE canopy_medicaid;
CREATE DATABASE canopy_caps;
CREATE DATABASE canopy_wic;
CREATE DATABASE canopy_rules;
CREATE DATABASE canopy_persons;
CREATE DATABASE canopy_applications;
CREATE DATABASE canopy_eligibility;
CREATE DATABASE canopy_enrollment;
CREATE DATABASE canopy_renewals;
CREATE DATABASE canopy_notices;
CREATE DATABASE canopy_appeals;
CREATE DATABASE canopy_reporting;
CREATE DATABASE canopy_security;
CREATE DATABASE canopy_web_sessions;

  2. Migrations run automatically on service startup via sqlx::migrate!(). No manual migration step required — except canopy-reporting (below).

  3. Enable SSL: add ?sslmode=require to all DATABASE_URL values in production.

canopy-reporting role provisioning (#1456, ADR-004 A8b)

canopy-reporting runs under a least-privilege role split and will refuse to boot as a broad role outside development. Before its first start:

  1. Provision the login carrier: ALTER ROLE canopy_reporting_app LOGIN and set its password from the secret manager (the role itself is created NOLOGIN/passwordless by migration 20261111000000; the credential is never in source).

  2. Point CANOPY_REPORTINGDATABASE_URL at canopy_reporting_app and CANOPY_REPORTINGMIGRATION_DATABASE_URL at the elevated migrator identity (see Configuration Reference).

Full cutover sequencing, rollback, and the standing per-migration ownership-transfer convention: Reporting Credential-Cutover Runbook.

Monitoring

Health Checks

Every service exposes:

  • GET /healthz — returns JSON: {"status": "ok", "checks": {"database": "ok", "rabbitmq": "ok"}}

  • GET /metrics — Prometheus text format (requires otel compile feature + OTEL_EXPORTER_OTLP_ENDPOINT)

Configure your load balancer to health-check /healthz on each service.

Recommended Alert Rules

Metric Threshold Action

Service health check failure

> 30 seconds

Restart container, investigate logs

Database connection pool saturation

active = max_connections

Increase db_max_connections or scale service

RabbitMQ queue depth

> 10,000 messages

Scale consumers, check for slow subscribers

HTTP error rate

> 1% of requests

Check service logs, recent deployments

P99 response latency

> 5 seconds

Profile slow queries, check resource limits

Disk usage (PostgreSQL)

> 80%

Expand volume, review retention policies

Log Aggregation

All services emit structured JSON logs to stdout via tracing_subscriber. Configure your container orchestrator to collect stdout and forward to your log aggregation platform (Loki, ELK, CloudWatch, Splunk).

Log level controlled by CANOPY_{SVC}__LOG_LEVEL (default: info). Set to debug for troubleshooting.

Backup & Disaster Recovery

PostgreSQL

  • Enable WAL archiving and continuous archiving (PITR)

  • Daily pg_dump to S3 for each program database

  • Test restore procedures quarterly

  • RTO target: 1 hour (restore from PITR)

  • RPO target: 5 minutes (WAL segment interval)

RabbitMQ

  • Durable queues with disk persistence (default in Canopy)

  • 3-node cluster with quorum queues for HA

  • Messages survive broker restart; unacked messages redelivered

S3 / Notice Storage

  • Enable versioning on the notice PDF bucket

  • Cross-region replication for disaster recovery

  • Notices are write-once (append-only) — no deletion in normal operation

Keycloak

  • Export realm configuration: docker exec keycloak /opt/keycloak/bin/kc.sh export --realm canopy --dir /tmp/export

  • Backup Keycloak’s PostgreSQL database separately

  • Store realm export alongside database backups

TLS Configuration

  • All public endpoints: TLS 1.2+ enforced. rustls handles TLS in-process (no OpenSSL).

  • Database connections: Add ?sslmode=require to DATABASE_URL for production.

  • RabbitMQ: Use amqps:// protocol with TLS-enabled RabbitMQ.

  • Internal service communication: HTTP by default within the container network. For defense-in-depth, enable mTLS via service mesh (Istio, Linkerd) or TLS termination at sidecar.

  • Certificate management: Automate renewal via Let’s Encrypt (ACME), AWS ACM, or your PKI. Set calendar reminders for any manually managed certificates.

High Availability

Canopy services are stateless — scale horizontally by running multiple replicas behind a load balancer.

  • Services: 2+ replicas per service. Health-check-aware load balancing.

  • PostgreSQL: Primary + streaming replica with automatic failover (Patroni, RDS Multi-AZ, Cloud SQL HA).

  • RabbitMQ: 3-node cluster with quorum queues. Mirror all durable queues.

  • Redis: Sentinel or cluster mode for session cache failover.

  • S3: Inherently highly available (cloud-managed) or MinIO with erasure coding.

  • Keycloak: Run 2+ replicas with shared PostgreSQL backend and Infinispan cache clustering.

Environment Promotion

Artifact promotion follows the ADR-040 build-once contract (there is no separate staging environment today):

  1. Build once: the pipeline builds both deployable images (the shared service image + canopy-portal) exactly once, under pipeline-internal immutable staging refs ($CI_REGISTRY_IMAGE/build:$CI_COMMIT_SHA + /build/portal:$CI_COMMIT_SHA).

  2. Gate against the same digests: integration tests run the devstack from those exact staging refs (docker-compose.prebuilt.yml, #1073) — test-what-you-promote.

  3. Promote by retag, never rebuild: docker-promote retags the gated digests into the production repositories — an immutable :<short-sha> per promoted commit, plus mutable :<tag> / :latest conveniences (and the /portal twins). A resource_group serializes concurrent promotions.

Deployment consumes the production refs via the prebuilt compose override — see the Scaling & Deployment runbook. The ci-config-lint xtask gate asserts these invariants against .gitlab-ci.yml on every push.

Edit this page · default