ADR-017: Encrypted Secrets at Rest with SOPS + age

On this page

Context

ADR-012 (Accepted 2026-04-23) ratified layered YAML configuration but explicitly excluded secrets from checked-in YAML: "Secrets never in checked-in YAML. Enforced by a CI lint that greps for common secret key names in config/*/.yaml." The reasoning was sound for plaintext YAML but left the at-rest representation of secrets unsolved — environment variables on the deploy host carry the values, and operators manage them outside the repo entirely.

Op-infra plan Step 5 phase 1 (MR !152) shipped canopy-secrets with SecretProvider trait + EnvSecretProvider (env-var-backed) + structured audit logging on every secret access, intending phase 2 to be a HashiCorp Vault backend. The Vault direction was reconsidered after enumerating canopy’s actual secret inventory:

  • ~15-25 distinct secret values per deployment (DB connection strings, RabbitMQ URL, JWS signing keys per program, AES-256-GCM SSN encryption key, internal service-to-service API key, Keycloak client secret, FTI-scoped DB URLs in canopy-security)

  • All long-lived (no rotation cadence today)

  • No dynamic secrets (Postgres credentials are static, not Vault-generated per request)

  • No lease lifecycle (consumers don’t poll for renewal)

Vault’s killer features (dynamic secrets, lease management, centralized rotation) are dead weight against this inventory. Operating a HA Vault cluster (3+ nodes, Raft consensus, audit log retention) for static-secret storage that env vars already provide is a meaningful operational burden with no proportional win.

Three alternatives merit consideration alongside Vault:

  1. Raw age — single-recipient or multi-recipient file-level encryption. Tiny tool surface (~30-page spec), modern primitives (X25519 + ChaCha20-Poly1305). Weakness: file-level encryption produces opaque diffs in PR review — any rotation produces a different blob, reviewer cannot tell which secret changed.

  2. Pure GPG (Saltstack pillar pattern) — armored PGP messages embedded in YAML. Familiar at organizations with existing GPG infrastructure. Weakness: GPG UX is famously brittle (keyring corruption, agent deadlocks, expired subkeys, TTY issues in CI/Docker).

  3. AWS Secrets Manager / equivalent cloud-vendor backends — IAM-rooted access, audit via CloudTrail. Strength for AWS-native deployments. Weakness: ties canopy to a specific cloud, which is a deployer-level decision rather than a framework-level one.

A fifth option emerged from discussion: SOPS + age. SOPS (Mozilla → CNCF) is a value-level encryption layer over a chosen backend (age, GPG, AWS-KMS, GCP-KMS, Azure Key Vault, Vault). Files look like normal YAML with only the values encrypted; keys and structure stay plaintext, so PR diffs show which value changed. SOPS+age combines SOPS’s diff-review property with age’s modern primitives and clean onboarding UX.

Per-jurisdiction operational separation is the user’s stated precedent at github.com/georgiacyber/kinetic: code lives in the public canopy repo; deployment configuration including secrets lives in a per-jurisdiction private repo. Canopy ships only secrets/dev.yaml for devstack and the integration test suite, with fake values only. Per-jurisdiction prod secrets are explicitly out of canopy’s scope.

Decision

SOPS + age for encrypted secrets at rest. Single file secrets/dev.yaml in the canopy repo, encrypted to two recipients (the primary developer’s age public key and the CI runner’s age public key). Decryption happens at deploy time (or cargo xtask dev start time in dev); the decrypted values are injected as environment variables and consumed via the existing CANOPY_{SERVICE}__* runtime contract. The EnvSecretProvider from phase 1 keeps working unchanged — at runtime, every service still reads its secrets from env vars and emits the audit log entry via target = "canopy.secrets".

This decision amends ADR-012. ADR-012’s "Secrets never in checked-in YAML" remains correct for plaintext YAML in config/. Encrypted YAML in secrets/ is the new at-rest mechanism for the env-var-injected secrets that ADR-012 left to deployer practice. The CI lint ADR-012 calls for is implemented as secrets-yaml-lint over config/*/.yaml only; secrets/*.yaml is excluded by path.

Out of scope: per-jurisdiction prod secrets, hot rotation without restart, multi-key SSN-encryption-key support (separate follow-up if/when first rotation is needed), Vault HA cluster operation, sealed-secrets operator support.

Rotation Mechanics

Rolling restart is the standard rotation mode. Drain a replica from the load balancer → it finishes in-flight requests → it shuts down → it starts with the new secret value (the freshly-decrypted dev.yaml values injected via the deploy mechanism) → it passes health check → LB routes again. Repeat replica-by-replica.

The constraint is not "all replicas restart simultaneously." The constraint is expand-contract at the credential level: during the rollout, both the old and the new credential must be simultaneously valid at the dependency. This is the same forward-only discipline as ADR-016's schema rotation pattern.

Per-secret-type rotation patterns:

Secret type Rotation mechanic

Database URL / Postgres password

Add the new password at Postgres (ALTER ROLE … WITH PASSWORD … accepts both via pg_hba.conf rules or successive password changes during the window); update secrets/dev.yaml; roll the fleet; remove the old password. PgPool cannot live-update; restart is required for new connections to use the new password.

RabbitMQ URL

Add a parallel user; roll; drop the old user.

JWS signing keys (per program)

Pair-aware via existing crates/canopy-signing/src/verifier.rs::VerifyingKeyRegistry (CURRENT + PREVIOUS per program). Rotation: generate new keypair K_new; update orchestrator’s verifier to CURRENT=K_new, PREVIOUS=K_old (one orchestrator restart cycle); roll the program service fleet to sign with K_new; after determinations signed under K_old age out, drop PREVIOUS from the verifier. The pair-aware pattern was built for this.

Keycloak client secret

Keycloak supports multiple client secrets per client. Add the new secret in Keycloak; update SOPS file; roll; remove the old.

Internal service-to-service API key

Receiver accepts a list of valid keys during rotation window; roll; remove the old.

AES-256-GCM SSN encryption key (called-out exception)

Cannot be rotated by credential change alone — at-rest data encrypted under K_old is not readable by K_new. Two paths: (a) extend crates/canopy-common/src/crypto.rs to accept a list of decryption keys (CURRENT + PREVIOUS[]); rolling restart works because every replica during the rollout decrypts both K_old and K_new ciphertext; lazy re-encrypt on read eventually migrates rows. (b) Bulk re-encryption migration job that rewrites every SSN from K_old to K_new before the credential rotation. This constraint is independent of secret-store choice — Vault, AWS-SM, and SOPS+age all hit it. Tracked as a follow-up filed at the end of Step 5 of the implementation plan.

Consequences

Positive

  • Compliance integrity preserved. The FTI audit hash chain (ADR-014) and audit_events chain (ADR-004) stay intact under all secret rotations because secrets don’t touch their schemas. JWS-signed determination history (ADR-002) survives signing-key rotation via the CURRENT/PREVIOUS pair-aware pattern.

  • Audit-via-git on rotations. Per-value SOPS encryption means PR diffs show exactly which secret changed. Reviewer can distinguish "rotated SNAP signing key" from "rotated all signing keys" from "swapped SNAP and TANF signing keys." Raw age would produce opaque blobs.

  • No runtime infrastructure to operate. No HA Vault cluster, no AWS dependency, no managed-service contract. The encryption layer is a static binary (sops) running at deploy time.

  • Onboarding friction is bounded. New contributor: install age and sops (one each), run cargo xtask secrets init to generate an age keypair, propose adding the public key in .sops.yaml via PR, an existing recipient runs cargo xtask secrets add-recipient. ~5 minutes once the tooling is in place.

  • Existing EnvSecretProvider audit log continues to function. Phase 1’s target = "canopy.secrets" access log fires on every provider.get(key) call regardless of where the env var’s value originated. Pub 1075 §9.4.1.4 access auditing remains satisfied.

Negative

  • One binary to install on developer + deploy machines (sops). Static Go binary, packaged in major distros, but it is one more thing in the prerequisites list.

  • SOPS file format is not an RFC. The format is documented but defined by the sops binary’s behavior. If sops becomes unmaintained, an in-house decryptor is feasible (the format is small) but is non-zero work.

  • Project governance. SOPS was Mozilla until ~2022, transferred to the getsops GitHub org, became a CNCF Sandbox project in 2024. Maintenance is ongoing (Linkerd, Flux, ArgoCD all rely on it) but trusts the project’s continuity.

  • Expand-contract discipline at the credential level. Operators rotating a secret have to remember to add the new credential at the dependency before updating the SOPS file and rolling. Same discipline as schema migrations. ADR-016’s expand-contract pattern is the model.

  • AES-256-GCM SSN encryption key rotation requires multi-key support. Tracked separately; not solved by this ADR.

Neutral

  • Runtime contract unchanged. Services read CANOPY_{SERVICE}__* env vars exactly as today. The change is at deploy time, not runtime. Existing tests, existing handlers, existing audit emit sites all continue to work.

  • age (Go reference impl) and rage (Rust impl) are interchangeable. Either produces compatible keypairs and SOPS doesn’t care which generated them. Implementer picks whichever their package manager has.

  • The developer’s existing GPG commit-signing key is unaffected. GPG continues to sign git commits; age is a separate identity for SOPS encryption only.

Alternatives Considered

  1. HashiCorp Vault (the original op-infra Step 5 phase 2 direction). Operational complexity (HA cluster, Raft, audit log retention) disproportionate to canopy’s static-secret inventory; killer features (dynamic secrets, leases) unused. Rejected in favor of SOPS+age. Issue #346 closed as superseded.

  2. Raw age (no SOPS layer). Smaller tool surface, smaller spec to depend on. Weakness: file-level encryption produces opaque diffs in PR review. The audit-via-git property — the entire point of secrets-in-source — degrades to "we know the file changed; we don’t know which value." Rejected in favor of SOPS+age. The one-binary cost of sops earns reviewable rotation diffs.

  3. Pure GPG (Saltstack pillar pattern). Familiar to the user from github.com/georgiacyber/kinetic. Weaknesses: GPG UX is brittle in CI/Docker/headless environments (TTY issues, agent state); no value-level encryption without an additional tool. Rejected in favor of SOPS+age, which provides the same in-git encryption pattern with cleaner UX.

  4. AWS Secrets Manager / Cloud-vendor SDKs. Native IAM-rooted access, audit via CloudTrail. Strength for AWS-native deployments. Deferred to deployer choice — canopy itself stays cloud-neutral. A jurisdiction deploying to AWS may layer Secrets Manager on top of (or instead of) SOPS+age in their own deployment-config repo; canopy doesn’t enforce one path.

Related ADRs

Edit this page · default