Secret Management & Rotation

On this page

Overview

All sensitive Canopy configuration values — database connection strings, RabbitMQ credentials, Keycloak secrets, the SSN encryption key — are read through the canopy-secrets crate’s SecretProvider trait rather than directly from std::env::var.

The trait gives Canopy three things:

  1. Audit trail. Every secret access produces a structured tracing::info! event with target = "canopy.secrets" carrying the service name and secret key (never the value). Required for IRS Pub 1075 §9.4.1.4 audit of access to sensitive data.

  2. Rotation seam. The provider returns a fresh value on every get(key) call. Phase 1 backs that with std::env::var, which is process-lifetime; rotation requires a service restart (see below). Phase 2 (HashiCorp Vault) returns versioned values that can be rotated without restart.

  3. Vault on-ramp without call-site churn. Every call site that needs a secret already takes &dyn SecretProvider (via ServiceSettings::load_with_secrets in canopy-api’s `bootstrap). Swapping EnvSecretProvider for a future VaultSecretProvider is a one-line change at the construction site.

Concern Phase 1 (today) Phase 2 (when Vault is provisioned)

Backend

EnvSecretProvider (std::env::var)

VaultSecretProvider (HashiCorp Vault HTTP API)

Audit log

tracing::info! target = "canopy.secrets"

Same — trait shared

Rotation

Restart the service (env var is process-lifetime)

Live rotation; provider re-reads on next get()

Storage

.env files (dev), env-var injection from secrets manager (prod)

Vault KV-v2 or similar

Access control

Filesystem permissions on the env source

Vault policies + AppRole / Kubernetes auth

Audit log format

Every successful secret read emits:

{
  "level": "INFO",
  "target": "canopy.secrets",
  "service": "canopy-snap",
  "secret": "CANOPY_SNAP__DATABASE_URL",
  "source": "env",
  "message": "secret accessed"
}

The secret value is never logged. The source field will become "vault" once phase 2 lands.

To observe the audit log in dev:

CANOPY_RULES__LOG_LEVEL=info cargo xtask dev start
docker logs canopy-canopy-snap-1 2>&1 | grep canopy.secrets

In production these events flow through the standard tracing pipeline (OpenTelemetry → Loki / OpenSearch / similar) and live alongside other audit-relevant events.

Phase 1 rotation procedure (current)

Phase 1 secrets are environment variables read once at process startup. To rotate a credential:

  1. Update the value in the secrets store (1Password, AWS Secrets Manager, etc.) and any environment-specific override files (.env, docker-compose.override.yml, Kubernetes Secret).

  2. Restart the affected services. For the devstack:

    docker compose restart canopy-snap canopy-tanf canopy-medicaid

    For production deployments, follow your standard rolling-restart procedure (Step 5 phase 2 will replace this with Vault’s secrets revoke semantics).

  3. Verify rotation: tail the audit log for canopy.secrets events on the restarted services and confirm the next get() returned without error.

NOTE

Database URL rotations require coordinated DB migration. If you rotate CANOPY_*__DATABASE_URL in a way that points to a different database, run cargo xtask migrate snapshot against the old database first (per Devstack Migration Snapshot) so you can roll back if the new database is unhealthy.

Common rotations

Secret Env var Rotation cadence

Database password

CANOPY_*__DATABASE_URL

Annually + on suspected compromise. Coordinate with PostgreSQL ALTER USER …​ PASSWORD (no downtime if connections re-handshake).

RabbitMQ password

CANOPY_*__RABBITMQ_URL

Annually + on suspected compromise. Coordinate with RabbitMQ rabbitmqctl change_password.

Keycloak realm signing key

Rotated in Keycloak admin

Per Keycloak’s automatic rotation policy. Canopy fetches via JWKS; no per-service rotation needed (operational-infrastructure plan Step 7’s auto-reconnect already handles JWKS refresh).

SSN encryption key

CANOPY_ENCRYPTION_KEY

Currently per-deployment lifetime. Rotation requires re-encrypting existing rows — tracked separately because data migration is a schema-level concern, not a secret-management one. Coordinate with the canopy-persons backfill plan when rotation is required.

Service-to-service signing keys (ECDSA P-256)

CANOPY_SIGNING_KEY_*

Per runbooks/signing-key-rotation.adoc (independent of this runbook — signing keys have their own zero-downtime rotation protocol).

Phase 2 outlook (Vault)

When HashiCorp Vault is provisioned, ship a VaultSecretProvider that:

  1. Authenticates via AppRole (production) or Kubernetes auth (in-cluster) to a Vault namespace owned by the Canopy deployment.

  2. Resolves secrets from a KV-v2 mount (e.g., kv/canopy/<env>/).

  3. Caches values per-process for a TTL configurable per secret — short TTL on credentials that should rotate frequently, long TTL on values that change rarely.

  4. Emits the same audit-log shape as EnvSecretProvider with source = "vault".

The bootstrap path constructs VaultSecretProvider instead of EnvSecretProvider; everything downstream is unchanged. Tracked as the phase 2 portion of operational-infrastructure plan Step 5.

Implementation references

  • crates/canopy-secrets/src/lib.rs — trait + env-backed impl

  • crates/canopy-common/src/settings.rs — ServiceSettings::load_with_secrets

  • crates/canopy-api/src/bootstrap.rs — wires the provider into every service

  • Plan: Operational Infrastructure — Step 5

Edit this page · default