Plan: Layered Config + Encrypted Secrets Migration (ADR-012 + ADR-017)
On this page
- Status
- Prerequisites for the implementer
- Context
- Scope
- Design
- Steps
- Step 0 — Plan ratification (this MR, docs-only)
- Step 1 —
cargo xtask secretstooling +secrets-yaml-lintCI job - Step 2 — Layered YAML loader in
canopy-common - Step 3 — canopy-snap canary migration
- Steps 4-21 — Per-service migrations (one MR each)
- Step 22 — Cleanup pass + retire
.env.example+ archive plan
- Files Touched
- Verification
- Documentation Updates
- Existing utilities to reuse
- Notes for the implementer
Status
| Step | Description | Status |
|---|---|---|
0 |
Plan ratification: ADR-017 + this plan + amendment NOTE on ADR-012 + nav/CLAUDE.md/CHANGELOG; close #291 and #346 as superseded; open 22 tracking issues |
Done (2026-05-02) — MR !163 |
1 |
|
Done (2026-05-02) — MR !166 |
2 |
Layered YAML loader in |
Done (2026-05-02) — MR !168 |
3 |
canopy-snap canary migration: |
Done (2026-05-02) — MR !170 + !171 |
4 |
canopy-exchange migration (zero env vars; validates template handles trivial case). Empty |
Done (2026-05-02) — MR !172 |
5 |
canopy-persons migration: |
Done (2026-05-02) — MR !173 |
6 |
canopy-verification migration (single |
Done (2026-05-02) — MR !174 |
7 |
canopy-applications migration (jurisdiction + rulesets_dir → YAML, ApplicationsConfig) |
Done (2026-05-02) — MR !175 |
8 |
canopy-enrollment migration (jurisdiction + rulesets_dir → YAML, EnrollmentConfig) |
Done (2026-05-02) — MR !176 |
9 |
canopy-renewals migration (jurisdiction + rulesets_dir → YAML, RenewalsConfig) |
Done (2026-05-02) — MR !176 |
10 |
canopy-notices migration (jurisdiction + rulesets_dir → YAML, NoticesServiceConfig — kept alongside existing |
Done (2026-05-02) — MR !176 |
11 |
canopy-rules migration: |
Done (2026-05-02) — MR !177 |
12 |
canopy-appeals migration: |
Done (2026-05-02) — MR !177 |
13 |
canopy-security migration: SecurityConfig (breach + FTI verify intervals); FTI DB URLs route through |
Done (2026-05-02) — MR !178 |
14 |
canopy-tanf migration: TanfConfig (rules_url, rulesets_dir) + signing_key via SOPS. |
Done (2026-05-02) — MR !179 |
15 |
canopy-caps migration: CapsConfig + signing_key via SOPS. |
Done (2026-05-02) — MR !179 |
16 |
canopy-wic migration: WicConfig + signing_key via SOPS. |
Done (2026-05-02) — MR !179 |
17 |
canopy-medicaid migration: MedicaidConfig (jurisdiction, rulesets_dir, rules_url, persons_url) + signing_key via SOPS. |
Done (2026-05-02) — MR !180 |
18 |
canopy-eligibility migration: EligibilityConfig (jurisdiction, rulesets_dir, persons_url, keys_dir). Verifying keys stay env-or- |
Done (2026-05-02) — MR !181 |
19 |
canopy-portal migration: groundwork only (YAML mount + |
Done (2026-05-02) — MR !182 |
20 |
canopy-web migration: WebConfig (jurisdiction, rulesets_dir, session_secure, keycloak_client_id, keycloak_external_url, keycloak_internal_url, redirect_url). |
Done (2026-05-02) — MR !182 |
21 |
canopy-reporting migration: ReportingConfig (jurisdiction + rulesets_dir + 7 downstream URLs). |
Done (2026-05-02) — MR !182 |
22 |
Cleanup: retire |
Done (2026-05-02) — this MR |
Prerequisites for the implementer
This plan is self-contained: code snippets, exact file paths, struct signatures, and verification commands per step are below. A contextless agent or new contributor should be able to execute Steps 1-22 from this document alone.
Mandatory reading before any code change:
-
.claude/CLAUDE.md— project conventions, GitLab scoped labels, commit signing, glossary. -
.claude/docs/delivery-protocol.md— preflight checks, delivery checklist. -
.claude/docs/git-workflow.md— branch naming, commit message types (feat|fix|chore|refactor|docs|test|ci:), GPG signing, pre-commit Q1-Q8 protocol. -
.claude/docs/security-baseline.md,.claude/docs/coding-conventions.md,.claude/docs/testing.md. -
The plan template at
docs/modules/ROOT/pages/plans/_template.adoc.
Tools required on developer machine before Step 1:
-
Standard:
cargo,cargo-nextest, Rust stable Edition 2024,docker+docker compose,gitwith GPG signing,glabCLI. -
ageandsopsare not host requirements — they ship in thecanopy-devtoolscompose service (added in Step 1) alongsideyq. Mirrors the existing pattern fornode/npm/Playwright (seetests/e2e/Dockerfile) and the postgres CLIs (pg_dump/pg_restore, invoked viadocker execfromxtask::cmd::migrate).
Context
ADR-012 (Accepted 2026-04-23) ratified layered YAML config with env-var overrides but the loader is not yet implemented — crates/canopy-common/src/settings.rs::ServiceSettings::load uses only the config crate’s Environment source. Issue #291 tracked the migration but stayed workflow::needs-spec because the spec was incomplete.
ADR-017 (this plan ratifies it) introduces SOPS+age-encrypted secrets at rest, replacing the rejected Vault direction (issue #346).
The two concerns share the same env-var sprawl (~110 std::env::var() calls across 19 services), the same per-service touch (each service’s main.rs + Cargo.toml + docker-compose.yml entry), and the same runtime contract. Doing them in a single per-service migration MR halves the touch on each service.
Scope
In scope:
-
ADR-017 ratification + ADR-012 amendment NOTE.
-
Layered YAML loader implementation in
canopy-common(the missing half of ADR-012). -
cargo xtask secretssubcommand +secrets-yaml-lintCI job + CI age-key wiring. -
Per-service migration of all 19 canopy services.
-
secrets/dev.yamlwith fake values only for devstack and integration tests. -
Closing #291 and #346 as superseded.
Out of scope:
-
Per-jurisdiction prod secrets (each jurisdiction operates their own private deployment-config repo, kinetic-style).
-
CLI argument layer (excluded by ADR-012).
-
Live SIGHUP/inotify config reload.
-
Multi-key SSN-encryption-key rotation support (filed as follow-up at end of Step 5).
-
PITR runbook (independent deliverable, #353).
Design
Coupling. ADR-017 amends ADR-012 — secrets at rest live in encrypted YAML; tunables in plaintext YAML; both surface to services via CANOPY_{SERVICE}__* env vars at runtime. The runtime read code is unchanged; only the at-rest representation and deploy-time injection differ.
Per-service migration MR template (canopy-snap is the canary in Step 3; Steps 4-21 substitute service names):
-
Create
config/{service}/default.yamlwith the service’s tunables. -
Create
services/{service}/src/config.rswith{Service}Configstruct using#[serde(deny_unknown_fields)]. -
Refactor
services/{service}/src/main.rs: replace directstd::env::varreads of tunables withServiceSettings::load_typed::<{Service}Config>("CANOPY_{SERVICE}", None); route secret reads throughEnvSecretProvider::get_optional. -
Add
canopy-secrets = { workspace = true }toservices/{service}/Cargo.tomlif not already present. -
Modify
docker-compose.yml: remove migrated tunable env vars; mount the YAML config volume; keep secret env vars (they come from SOPS-decrypted dev.yaml at deploy time). -
Add the service’s secrets (fake values) to
secrets/dev.yamlviacargo xtask secrets edit. -
Add unit test
{service}_config_loads_from_default_yaml. -
Verify:
cargo nextest run -p {service}+cargo xtask validate+cargo xtask e2e.
File layout (final state in canopy repo):
.sops.yaml # SOPS recipient rules
secrets/
dev.yaml # encrypted, fake values only
config/
canopy-{snap,tanf,medicaid,...}/
default.yaml
docs/modules/ROOT/pages/
adrs/adr-017-encrypted-secrets-at-rest.adoc
plans/secret-and-config-migration.adoc # archived after Step 22
Recipient model. Two recipients on secrets/dev.yaml: primary developer’s age public key + CI runner’s age public key. CI runner private key in GitLab masked variable CANOPY_CI_AGE_KEY; CI jobs that bring up devstack set SOPS_AGE_KEY: $CANOPY_CI_AGE_KEY.
Special-case routing — shared: section. Secrets consumed by canopy-common (e.g. CANOPY_ENCRYPTION_KEY) have no service prefix. The YAML→dotenv walk treats top-level shared: specially: leaves emit CANOPY_<KEY_UPPER> (single underscore, no service prefix). Documented as a doc-comment on xtask::cmd::secrets::walk_to_dotenv.
Config root path resolution. ServiceSettings::load_typed(prefix, config_root: Option<&Path>). Resolution priority: explicit Some(&Path) (used by tests) → CANOPY_CONFIG_ROOT env var (used by deployments mounting at /app/config) → default . (PWD; used in dev when running from repo root).
Steps
Step 0 — Plan ratification (this MR, docs-only)
Branch: chore/adr-017-and-config-migration-plan (merged 2026-05-02 as MR !163).
Files created:
-
docs/modules/ROOT/pages/adrs/adr-017-encrypted-secrets-at-rest.adoc— full ADR (Context, Decision, Rotation Mechanics, Consequences, Alternatives Considered, Related ADRs). -
docs/modules/ROOT/pages/plans/secret-and-config-migration.adoc(this file).
Files modified:
-
ADR-012 — added a NOTE block at the top of
== Decision:Amended by ADR-017 (2026-05-02). Secrets at rest now ship as SOPS-encrypted YAML; the runtime contract (env vars) is preserved. The "secrets never in checked-in YAML" line below applies to plaintext YAML in \`config/\only.` Existing prose preserved. -
docs/modules/ROOT/nav.adoc— added ADR-017 entry under Architecture & Design and a Foundational Migrations subsection linking this plan. -
.claude/CLAUDE.md— added an architecture-list bullet for ADR-017; lightly amended the ADR-012 bullet to note the amendment. -
CHANGELOG.adoc— entry under=== Added.
GitLab actions (after merge):
-
Close #346 (Vault) and #291 (env-var → YAML migration) with supersession comments.
-
Open 22 step-tracking issues (#354 through #375) labeled
[secret-and-config-migration] Step N: <heading>withtype::*,priority::medium(Steps 1-3) orpriority::low(Steps 4-22),program::*,service::*,workflow::ready.
Verification: cargo xtask check-docs clean, cargo xtask docs plan-lint clean (no Deferred rows), cargo xtask validate --skip-docker green.
Commit: docs: ADR-017 SOPS+age + secret-and-config-migration plan (closes #291, #346).
Step 1 — cargo xtask secrets tooling + secrets-yaml-lint CI job
Branch: feat/xtask-secrets-tooling. Tracking issue: #354.
Containerized tooling. age and sops ship in a new canopy-devtools compose service rather than as host requirements (matches the canopy-e2e Playwright pattern). xtask::devtools wraps docker compose --profile devtools run --rm so call sites stay readable.
Files created:
-
tools/canopy-devtools/Dockerfile— alpine-based image withage,sops,yq. Versions confirmed at build time:age ≥ 1.1,sops ≥ 3.8. Yq is mikefarah’s Go binary (single-file). -
xtask/src/devtools.rs— wrapper module. Three primary fns:pub fn run(tool: &str, args: &[&str], opts: RunOpts) -> Result<Output>; pub fn run_interactive(tool: &str, args: &[&str], opts: RunOpts) -> Result<()>; pub struct RunOpts { pub age_key: AgeKeyMode, pub interactive: bool } pub enum AgeKeyMode { None, ReadOnly, ReadWrite }The wrapper resolves host UID/GID via
id -u/id -gand passes--user UID:GIDto docker so files written by the container land owned by the developer. For sops-decrypt operations,AgeKeyMode::ReadOnlybind-mounts~/.config/sops/ageand setsSOPS_AGE_KEY_FILE=/sops-age/keys.txtinside the container.AgeKeyMode::ReadWriteis used only bysecrets initto write a freshkeys.txt. -
xtask/src/cmd/secrets.rs— new module. Five subcommands matching the action enum:// SPDX-License-Identifier: AGPL-3.0-or-later #[derive(clap::Args)] pub struct Args { #[command(subcommand)] pub action: Action, } #[derive(clap::Subcommand)] pub enum Action { /// Generate dev age keypair (if missing) and print the public key. /// Pass `--for-ci` to instead emit a transient keypair for the /// GitLab `CANOPY_CI_AGE_KEY` masked variable. Init { #[arg(long = "for-ci")] for_ci: bool }, /// Open `secrets/dev.yaml` in `$EDITOR` via sops (decrypts on read, /// re-encrypts on save). Edit, /// Print decrypted secrets/dev.yaml as a flat dotenv stream. Decrypt, /// Append a new age recipient to `.sops.yaml` and re-encrypt the data key. AddRecipient { age_key: String }, /// Print a fresh ECDSA P-256 PEM (paste into `secrets edit`). GenerateSigningKey, }Decrypt-walk algorithm. SOPS is invoked with
--output-type jsonand the result is parsed withserde_json::Value(noserde_yamldep — it’s archived; the lockfile literally says0.9.34+deprecated). Walk rules:-
Service-keyed (top-level matches
canopy-<name>): leaves emitCANOPY_<SVC_UPPER>__<KEY_UPPER>=<value>. -
Shared (top-level is
shared): leaves emitCANOPY_<KEY_UPPER>=<value>(single underscore, no service prefix). Used for secrets consumed bycanopy-common. -
Skip: the top-level
sops:key (SOPS metadata). -
Multi-line value escaping: replace
\nwith literal\\nand wrap in double quotes.GenerateSigningKeycallscanopy_signing::keygen::generate_key_pair()directly (no container needed; pure Rust).
-
-
.sops.yamlat workspace root with both age recipients (developer + CI):creation_rules: - path_regex: ^secrets/dev\.yaml$ age: >- <DEVELOPER-AGE-PUBLIC-KEY>, <CI-RUNNER-AGE-PUBLIC-KEY>The CI runner’s age private key is stored in GitLab as masked variable
CANOPY_CI_AGE_KEY(Settings → CI/CD → Variables → Masked + Protected). Generated viacargo xtask secrets init --for-ci(transient — never persisted to disk on the dev machine). -
secrets/dev.yaml— encrypted, placeholder contents only. Per-service entries land in Steps 3-21.
Files modified:
-
docker-compose.yml— addcanopy-devtoolsservice entry under adevtoolsprofile (so it doesn’t start ondev start):canopy-devtools: profiles: [devtools] build: context: ./tools/canopy-devtools image: canopy-devtools:latest volumes: - .:/work -
xtask/src/cmd/mod.rs— registerpub mod secrets;and addSecrets(secrets::Args)to theCommandenum. -
xtask/src/main.rs— registermod devtools;and addCommand::Secrets(args) ⇒ cmd::secrets::run(args)to the dispatch. -
.gitlab-ci.yml— newsecrets-yaml-lintjob (insert afteradr-013-plan-lint):secrets-yaml-lint: image: alpine:3.21 stage: test script: - | set -euo pipefail VIOLATIONS=0 if [ -d config ]; then for f in $(find config -name '*.yaml' 2>/dev/null); do if grep -inE '(password|secret|signing_key|private_key|_token|api_key)\s*:\s*[A-Za-z0-9/+]{16,}' "$f" | grep -v '# allow-secret:'; then echo "VIOLATION: plaintext secret-shaped value in $f" VIOLATIONS=$((VIOLATIONS + 1)) fi if grep -in 'BEGIN .*PRIVATE KEY' "$f" | grep -v '# allow-secret:'; then echo "VIOLATION: PEM private key embedded in $f" VIOLATIONS=$((VIOLATIONS + 1)) fi done fi if [ "$VIOLATIONS" -gt 0 ]; then echo "ERROR: $VIOLATIONS plaintext-secret violation(s)" exit 1 fi echo "OK: no plaintext secrets in config/" rules: - if: $CI_COMMIT_BRANCH - if: $CI_MERGE_REQUEST_IIDHeuristic is conservative; false positives escape with a
# allow-secret: <reason>annotation. If it proves too noisy, follow up by migrating togitleaksortrufflehog. Uses a stockalpine:3.21image rather than.rust-basesince it’s pure regex (no Rust toolchain needed; ~2s job).NOTESOPS_AGE_KEYenv wiring on the cargo-test / e2e jobs is deferred to Step 3. No CI job invokessopsuntil the canopy-snap canary’sxtask::cmd::devdecrypt-at-start integration ships. Wiring it here would create a dead-code "what ifCANOPY_CI_AGE_KEYisn’t set" risk for zero current benefit.
Operational prerequisite (before Step 3 merges, not this MR): configure GitLab masked variable CANOPY_CI_AGE_KEY from the output of cargo xtask secrets init --for-ci.
Tests:
-
cargo xtask secrets initround-trip on a clean machine: generates age keypair, writes to~/.config/sops/age/keys.txt, prints public key. -
cargo xtask secrets init --for-ciemits a transient keypair with paste instructions. -
cargo xtask secrets decryptround-tripssecrets/dev.yaml(placeholder payload) and emits dotenv-shaped output. -
Unit tests in
xtask/src/cmd/secrets.rs::tests(no docker required) cover the pure-Rust helpers:Test Asserts decrypt_walk_emits_dotenv_for_simple_valueservice-keyed top-level →
CANOPY_<SVC>__<KEY>=<value>decrypt_walk_emits_shared_without_service_prefixshared.<key>→CANOPY_<KEY>(single underscore)decrypt_walk_skips_sops_metadatasops:block excluded from outputdecrypt_walk_handles_multi_line_pemPEM blocks →
\n-escaped + double-quotedformat_dotenv_quotes_values_with_spacesvalues with whitespace / quotes get escaped
append_recipient_to_inline_ageinline
age: a,bform correctly extended to folded scalarappend_recipient_to_folded_agefolded
age: >-\n a,\n bform gets new line at correct indentparse_public_key_from_age_keygen_stdout# public key: age1…parser handles age-keygen output
Verification:
-
cargo nextest run -p xtask cmd::secrets— 9 unit tests pass. -
docker compose --profile devtools build canopy-devtoolssucceeds; tools report expected versions (age 1.2.x,sops 3.9.x,yq 4.x). -
cargo xtask secrets init— generates dev keypair if missing; idempotent on second run. -
cargo xtask secrets init --for-ci— emits ephemeral keypair with masked-variable paste instructions. -
cargo xtask secrets decrypt— round-trips placeholdersecrets/dev.yamland emits dotenv lines. -
cargo xtask secrets edit— opens$EDITORin the container; saving re-encrypts in place. -
cargo xtask validate --skip-docker— green. -
Manual: push a deliberate
config/test/default.yamlwithpassword: AAAAAAAAAAAAAAAAAAAA, observesecrets-yaml-lintjob fails; revert.
Commit: feat: cargo xtask secrets tooling + canopy-devtools container + secrets-yaml-lint CI job (closes #354).
Step 2 — Layered YAML loader in canopy-common
Branch: feat/canopy-common-layered-yaml-loader. Tracking issue: #355.
#[serde(deny_unknown_fields)] on ServiceSettings is deferred to Step 22 (cleanup pass). Reason: every service still passes service-specific env vars (CANOPY_SNAPRULES_URL, CANOPY_TANFSIGNING_KEY, etc.) through the same prefix as the shared CANOPY_<SVC>PORT / DATABASE_URL / etc. fields that ServiceSettings consumes. Adding deny_unknown_fields to ServiceSettings here would crash every service’s canopy_api::bootstrap call until the per-service config structs (Steps 3-21) replace those direct env reads. Per-service structs introduced from Step 3 onwards DO carry deny_unknown_fields so the typo-catching benefit lands service-by-service.
load_typed_internal(prefix, config_root, env_name, env_override) hook that injects env vars via config::Environment::source(Some(map)) instead of mutating std::env. Required by crates/canopy-common/src/lib.rs:9 (#![forbid(unsafe_code)]), which rules out the unsafe { std::env::set_var(…) } pattern that the original plan sketch used. Production callers use the public [load_typed] without the override.
Files modified:
-
crates/canopy-common/src/settings.rs— extendServiceSettings::loadto be the layered loader per ADR-012. Current implementation reads only env vars:// BEFORE: pub fn load(prefix: &str) -> Result<Self, config::ConfigError> { Config::builder() .add_source(Environment::with_prefix(prefix).separator("__").try_parsing(true)) .build()? .try_deserialize() }New implementation: a free-standing module-level
load_typed<T>(the public API) plus a#[doc(hidden)]load_typed_internalthat takes the env-overlay name and an optional env-var override map. The override is the test hook (forced byforbid(unsafe_code); see the NOTE above).pub fn load(prefix: &str) -> Result<Self, config::ConfigError> { load_typed::<Self>(prefix, None) } /// Generic typed loader. See module docs for layering rules. pub fn load_typed<T: serde::de::DeserializeOwned>( prefix: &str, config_root: Option<&Path>, ) -> Result<T, config::ConfigError> { let env_name = std::env::var("CANOPY_ENV").unwrap_or_else(|_| "dev".to_string()); load_typed_internal::<T>(prefix, config_root, &env_name, None) } #[doc(hidden)] pub fn load_typed_internal<T: serde::de::DeserializeOwned>( prefix: &str, config_root: Option<&Path>, env_name: &str, env_override: Option<config::Map<String, String>>, ) -> Result<T, config::ConfigError> { let service = prefix.to_lowercase().replace('_', "-"); let root: PathBuf = config_root .map(Path::to_path_buf) .or_else(|| std::env::var_os("CANOPY_CONFIG_ROOT").map(PathBuf::from)) .unwrap_or_else(|| PathBuf::from(".")); let default_path = root.join(&service).join("default"); let env_path = root.join(&service).join(env_name); let env_source = Environment::with_prefix(prefix) .separator("__") .try_parsing(true) .source(env_override); let builder = Config::builder() .add_source(File::with_name(default_path.to_str().unwrap()).required(false)) .add_source(File::with_name(env_path.to_str().unwrap()).required(false)) .add_source(env_source); let config = builder.build()?; // DEBUG attribution per ADR-012 §Negative. if tracing::enabled!(tracing::Level::DEBUG) && let Ok(map) = config.clone().try_deserialize::<serde_json::Value>() { tracing::debug!(prefix = %prefix, service = %service, env = %env_name, config = %map, "config loaded"); } config.try_deserialize() }Both YAML files are optional — services that have no tunables today rely on env vars alone, and the loader silently succeeds if neither file exists.
load_with_secretskeeps its current signature; secrets continue to come fromSecretProvider, never from YAML. -
crates/canopy-common/Cargo.toml— addtempfile = { workspace = true }to[dev-dependencies].config,tracing, andserde_jsonalready present.
Tests: extend the existing test module in settings.rs. Tests pass Some(tmpdir.path()) as config_root and inject env vars via Some(env_map) on load_typed_internal. No std::env::set_var (forbidden by crate-level #[forbid(unsafe_code)]); env vars never escape the test scope.
fn env_map(pairs: &[(&str, &str)]) -> Map<String, String> {
pairs.iter().map(|(k, v)| ((*k).to_string(), (*v).to_string())).collect()
}
#[derive(Debug, Deserialize, PartialEq)]
#[serde(deny_unknown_fields)]
struct TestConfig { port: u16, jurisdiction: String }
#[test]
fn load_typed_falls_back_to_env_only_when_yaml_absent() {
let tmp = tempdir().unwrap();
let env = env_map(&[("CANOPY_TEST__PORT", "7000"),
("CANOPY_TEST__JURISDICTION", "georgia")]);
let cfg: TestConfig = load_typed_internal("CANOPY_TEST", Some(tmp.path()),
"dev", Some(env)).unwrap();
assert_eq!(cfg, TestConfig { port: 7000, jurisdiction: "georgia".into() });
}
// + load_typed_layers_yaml_under_env_override
// + load_typed_env_overlay_wins_over_default
// + load_typed_dev_overlay_falls_back_to_default_when_missing
// + load_typed_deny_unknown_fields_catches_typo_in_yaml
// + load_typed_service_path_derives_from_prefix
Verification:
-
cargo nextest run -p canopy-common— all tests pass (71 → 71 + 6 new). -
cargo xtask validate— green. -
Manual: temporarily create
config/canopy-test/default.yamlwith a port; observe a service that uses prefixCANOPY_TESTpicks it up; revert.
Commit: feat(canopy-common): layered YAML config loader (closes #355).
Step 3 — canopy-snap canary migration
Branch: feat/canopy-snap-config-migration. Tracking issue: #356.
This is the canary; Steps 4-21 follow this template exactly with service-specific substitutions.
Files created:
-
config/canopy-snap/default.yaml— non-secret tunables. Schema:jurisdiction: georgia rulesets_dir: rulesets rules: url: http://canopy-rules:8001 verification: url: http://canopy-verification:8005Confirmed by reading
services/canopy-snap/src/main.rslines 56, 59, 73, 77 — these are the four current tunable env reads. No secrets in this file. -
services/canopy-snap/src/config.rs— typed config struct:// SPDX-License-Identifier: AGPL-3.0-or-later //! Service-specific config schema for canopy-snap (ADR-012). use std::path::PathBuf; use serde::Deserialize; #[derive(Debug, Deserialize)] #[serde(deny_unknown_fields)] pub struct SnapConfig { pub jurisdiction: String, pub rulesets_dir: PathBuf, pub rules: RulesClientConfig, pub verification: VerificationClientConfig, } #[derive(Debug, Deserialize)] #[serde(deny_unknown_fields)] pub struct RulesClientConfig { pub url: String, } #[derive(Debug, Deserialize)] #[serde(deny_unknown_fields)] pub struct VerificationClientConfig { pub url: String, }
Files modified:
-
services/canopy-snap/Cargo.toml— addcanopy-secrets = { workspace = true }to[dependencies]. -
services/canopy-snap/src/lib.rs(or wherever modules are declared) — addpub mod config;. -
services/canopy-snap/src/main.rs— refactor lines 56-85. Today:// BEFORE: let jurisdiction = std::env::var("CANOPY_SNAP__JURISDICTION").expect("CANOPY_SNAP__JURISDICTION is required"); let rulesets_dir = std::env::var("CANOPY_RULESETS_DIR").unwrap_or_else(|_| "rulesets".into()); let rules_url = std::env::var("CANOPY_SNAP__RULES_URL").unwrap_or_else(|_| "http://localhost:8001".into()); let verification_url = std::env::var("CANOPY_SNAP__VERIFICATION_URL").unwrap_or_else(|_| "http://localhost:8005".into()); let internal_api_key = std::env::var("CANOPY_INTERNAL_API_KEY").unwrap_or_else(|_| "canopy-internal-dev-key".into()); let signing_key_pem = std::env::var("CANOPY_SNAP__SIGNING_KEY").unwrap_or_default();After:
// AFTER: use canopy_snap::config::SnapConfig; use canopy_secrets::{EnvSecretProvider, SecretProvider}; let snap_config: SnapConfig = canopy_common::settings::ServiceSettings::load_typed("CANOPY_SNAP", None) .expect("load canopy-snap config"); let secrets: EnvSecretProvider = EnvSecretProvider::new("canopy-snap"); let internal_api_key = secrets.get_optional("CANOPY_INTERNAL_API_KEY") .expect("read CANOPY_INTERNAL_API_KEY") .unwrap_or_else(|| "canopy-internal-dev-key".into()); let signing_key_pem = secrets.get_optional("CANOPY_SNAP__SIGNING_KEY") .expect("read CANOPY_SNAP__SIGNING_KEY") .unwrap_or_default();Update every downstream reference:
jurisdiction→snap_config.jurisdiction,rulesets_dir→snap_config.rulesets_dir,rules_url→&snap_config.rules.url,verification_url→&snap_config.verification.url. Noterulesets_dirwas aStringand is now aPathBuf— coerce at call sites if needed. -
docker-compose.yml— locate the canopy-snap service block. Remove these lines from theenvironment:block:-
CANOPY_SNAP__JURISDICTION: "georgia" -
CANOPY_SNAP__RULES_URL: "http://canopy-rules:8001" -
CANOPY_RULESETS_DIR(if explicitly set) -
CANOPY_SNAP__VERIFICATION_URL(if explicitly set)Add a
volumes:entry to mount the YAML config:volumes: - ./config/canopy-snap:/app/config/canopy-snap:roKeep these lines (still env-var-injected at runtime; they are secrets or runtime control vars):
-
CANOPY_SNAP__DATABASE_URL(DB password) -
CANOPY_SNAP__RABBITMQ_URL(broker password) -
CANOPY_SNAPKEYCLOAK_ISSUER,CANOPY_SNAPKEYCLOAK_URL(auth endpoints) -
CANOPY_SNAP__SIGNING_KEY(sources from SOPS viaxtask dev start) -
CANOPY_INTERNAL_API_KEY(secret) -
CANOPY_ENV: "development"(controls which YAML overlay loads)
-
-
secrets/dev.yaml(encrypted; edit viacargo xtask secrets edit):canopy-snap: signing_key: | -----BEGIN PRIVATE KEY----- <generated PEM> -----END PRIVATE KEY----- internal_api_key: canopy-internal-dev-key-NOT-REALGenerate the fake signing key with
cargo xtask secrets generate-signing-key(added in Step 1) and paste into the editor session. -
xtask/src/cmd/dev.rs— locate thedev startaction. Before invokingdocker compose up, run the SOPS decrypt step and write the output to a tmpfile that compose consumes via--env-file:// Near the top of dev_start, after preflight checks: let env_file = tempfile::Builder::new() .prefix("canopy-dev-") .suffix(".env") .tempfile() .context("create tempfile for sops dotenv")?; let decrypted = ProcessCommand::new("sops") .args(["--decrypt", "secrets/dev.yaml"]) .output() .context("invoke sops --decrypt secrets/dev.yaml")?; if !decrypted.status.success() { let stderr = String::from_utf8_lossy(&decrypted.stderr); bail!("sops decrypt failed: {stderr}"); } let yaml: serde_yaml::Value = serde_yaml::from_slice(&decrypted.stdout)?; let mut dotenv = crate::cmd::secrets::walk_to_dotenv(&yaml)?; dotenv.push_str("CANOPY_CONFIG_ROOT=/app/config\n"); std::fs::write(env_file.path(), dotenv)?; // Pass --env-file=<env_file.path()> to the docker compose invocation. // `env_file` is dropped at end of dev_start, which removes the tempfile.Extract the YAML→dotenv walk from
cmd/secrets.rsinto a publicwalk_to_dotenv(yaml: &serde_yaml::Value) → Result<String>sodev.rscan call it without code duplication. Usetempfile::NamedTempFilefor cleanup (itsDropimpl removes the file — preferable to actrlchandler).
Tests:
-
All existing canopy-snap tests must still pass with no edits to test code itself. They read the same env vars at runtime; the change is at the loading layer.
-
New unit test in
services/canopy-snap/src/config.rs::tests:#[test] fn snap_config_loads_from_default_yaml() { use canopy_common::settings::ServiceSettings; let _guard = test_mutex(); let _cd = ChangeDir::to(repo_root()); let cfg: SnapConfig = ServiceSettings::load_typed("CANOPY_SNAP", None).expect("load"); assert_eq!(cfg.jurisdiction, "georgia"); assert_eq!(cfg.rules.url, "http://canopy-rules:8001"); }
Verification:
-
cargo nextest run -p canopy-snap— all tests pass. -
cargo xtask dev start— devstack comes up healthy; canopy-snap reads its config from YAML + SOPS-decrypted secrets;docker compose logs canopy-snap | headshows no missing-env-var warnings. -
cargo xtask validate— full battery green. -
cargo xtask e2e— Playwright tests pass; SNAP-determination flow exercises the signing path end-to-end with the SOPS-injected fake key. -
Manual:
docker compose logs canopy-snap | grep 'config loaded'(DEBUG) — confirms the layered loader fired.
Commit: feat(canopy-snap): migrate to layered YAML config + SOPS-encrypted signing key (closes #356).
Steps 4-21 — Per-service migrations (one MR each)
NOTE (lessons from Step 3, post MR !171): the per-service config struct (a) does NOT carry #[serde(deny_unknown_fields)] during the migration window — the layered loader’s env source feeds every CANOPY_<SVC>* variable through, including the shared baseline keys (port, database_url, rabbitmq_url, keycloak_*) consumed by ServiceSettings; deny_unknown_fields rejects those. (b) Fields are flat (e.g. rules_url: String), not nested (e.g. rules.url) — the existing env-var convention is single separator, and nested structs would require RULESURL (double __), a breaking change to the ops contract. Both are restored in Step 22 once the prefix-sharing is resolved.
Migration template (mirrors Step 3 with service-specific substitutions):
-
Create
config/{service}/default.yamlwith the service’s tunables (read itsmain.rsto enumerate; cross-check against the inventory below). -
Create
services/{service}/src/config.rswith{Service}Config(flat fields; nodeny_unknown_fields— see lessons above). Skip the struct entirely when a service has zero tunables — adding an empty struct that’s never instantiated is dead code (canopy-exchange in Step 4 takes this route). -
Modify
services/{service}/src/main.rs(andlib.rsto declare the new module): replace directstd::env::varreads of tunables withcanopy_common::settings::load_typed::<{Service}Config>("CANOPY_{SERVICE}", None); route secret reads throughEnvSecretProvider::get_optional. -
Modify
services/{service}/Cargo.toml: addcanopy-secrets = { workspace = true }andconfig = { workspace = true }if not present. -
Modify
docker-compose.yml: remove migrated tunable env vars from the service’senvironment:block; mount./config/{service}:/app/config/{service}:ro; setCANOPY_CONFIG_ROOT: "/app/config"; add${CANOPY_<SVC>__SIGNING_KEY:-}/${CANOPY_INTERNAL_API_KEY:-canopy-internal-dev-key}substitutions for any secrets the service consumes. -
Add the service’s secrets (fake values) to
secrets/dev.yamlviacargo xtask secrets edit. Usecanopy-{svc}for service-specific secrets,shared:for cross-service ones (e.g.CANOPY_INTERNAL_API_KEY,CANOPY_ENCRYPTION_KEY). -
Add a unit test
{service}_config_loads_from_default_yamlmirroring Step 3’s pattern.
Per-MR verification (template, identical to Step 3):
-
cargo nextest run -p {service}— all existing + new tests pass. -
cargo xtask dev start— devstack stays healthy. -
cargo xtask validate— green. -
cargo xtask e2e— Playwright passes.
Per-MR commit message template: feat({service}): migrate to layered YAML config + SOPS secrets (closes #<step-issue>). Branch naming: feat/{service}-config-migration.
Service inventory (canonical migration order — easiest first, hardest last):
| Step | Service | Secrets | Tunables | Special notes |
|---|---|---|---|---|
4 (#357) |
canopy-exchange |
0 |
0 |
Stub service; trivial migration; validates the template handles the zero-content case. The MR delivers an empty |
5 (#358) |
canopy-persons |
0 (in service main; encryption key consumed by canopy-common) |
1 ( |
Special: |
6 (#359) |
canopy-verification |
1 ( |
0 |
Single-secret migration. |
7 (#360) |
canopy-applications |
0 |
2 ( |
Pure tunable migration. |
8 (#361) |
canopy-enrollment |
0 |
2 |
Same shape as Step 7. |
9 (#362) |
canopy-renewals |
0 |
2 |
Same shape. |
10 (#363) |
canopy-notices |
0 |
2 |
Same shape. |
11 (#364) |
canopy-rules |
0 |
3 ( |
|
12 (#365) |
canopy-appeals |
0 |
3 ( |
|
13 (#366) |
canopy-security |
2 ( |
3 ( |
FTI-scoped DB URLs are secrets (read-only credentials). |
14 (#367) |
canopy-tanf |
1 ( |
2 ( |
|
15 (#368) |
canopy-caps |
1 (signing key) |
2 |
Same shape as Step 14. |
16 (#369) |
canopy-wic |
1 (signing key) |
2 |
Same shape. |
17 (#370) |
canopy-medicaid |
1 (signing key) |
4 ( |
|
18 (#371) |
canopy-eligibility |
N (verifying keys for 5 programs: |
4 ( |
Verifying keys are public but managed alongside signing keys; place under |
19 (#372) |
canopy-portal |
0 |
1 ( |
Plus session-secret if/when wired. |
20 (#373) |
canopy-web |
1 (Keycloak client secret if present) |
7 ( |
Largest tunable surface among UI services. |
21 (#374) |
canopy-reporting |
0 |
9 service URLs + 1 |
Largest service-URL mesh. Pure tunable migration. |
Step 3 already migrated canopy-snap. Total services migrated: 1 canary + 18 in Steps 4-21 = 19. Verify against find services -maxdepth 1 -type d -name 'canopy-*' before starting the run; if a 20th service has been added, insert it into the order after Step 21 by complexity.
Step 22 — Cleanup pass + retire .env.example + archive plan
Branch: chore/secret-and-config-migration-cleanup. Tracking issue: #375.
Files modified:
-
.env.example— delete. Schema source of truth is now per-serviceconfig/{service}/default.yaml. -
docker-compose.yml— final cleanup pass. Comb every service’senvironment:block; if any block has been reduced to only secrets, leave it. If any block is now empty, remove it. Update top-of-file comments to point atconfig/{service}/default.yamlfor the schema. -
developer-guide.adoc — update the "Configuration" section (or add one if absent) to describe the YAML+env layered model. Add a "First run on a fresh checkout" subsection that walks through
cargo xtask secrets init. Cross-link to ADR-012 and ADR-017. -
docs/modules/ROOT/pages/plans/secret-and-config-migration.adoc(this file) — flip every Status row toDone (YYYY-MM-DD). Runcargo xtask docs plan-archiveto move the file intoarchive/. -
docs/modules/ROOT/pages/plans/archive.adoc— add a row in the appropriate section. -
CHANGELOG.adoc— final entry under=== Changed:* Secret and config migration complete. All 19 canopy services migrated to layered YAML config + SOPS-encrypted secrets.
Verification:
-
cargo xtask validate— green. -
cargo xtask docs plan-lint— clean. -
cargo xtask docs plan-archive— successfully moves the migration plan intoarchive/. -
cargo xtask e2e— green. -
Audit grep:
grep -rn 'std::env::var("CANOPY_' services/ crates/— every remaining hit is a secret read or a runtime control variable (CANOPY_ENV,CANOPY_CI). -
Schema audit:
for f in config/*/default.yaml; do yq . "$f" > /dev/null || echo "BROKEN: $f"; done. -
Onboarding rehearsal: clone the repo on a fresh machine, install
age+sops, runcargo xtask secrets init, propose new public key in.sops.yamlvia PR, get added by an existing recipient, runcargo xtask dev start, runcargo xtask test— full happy path on a truly fresh checkout.
Commit: chore: retire .env.example, archive secret-and-config-migration plan.
Files Touched
| File | Change |
|---|---|
|
Step 2: layered loader extension; add |
|
Step 1: |
|
Step 3: decrypt-at-dev-start integration |
|
Step 1: |
|
Step 1: SOPS recipient rules with developer + CI recipients |
|
Step 1 creates empty; Steps 3-21 grow it with each service’s fake secrets |
|
Steps 3-21: per-service tunables |
|
Steps 3-21: refactor env reads to typed config + secret provider |
|
Steps 3-21: per-service config struct (without |
|
Steps 3-21: add |
|
Steps 3-22: remove tunable env vars; mount YAML configs |
|
Step 0 |
|
Step 0 |
|
Step 0: NOTE block referencing ADR-017 |
|
Step 0: ADR-017 + plan entries |
|
Step 0: ADR-017 architecture bullet |
|
Every step: one entry per merged MR |
|
Step 22: deleted |
Verification
End-to-end, after Step 22:
-
cargo xtask validate --timing— green, full battery (~5-6 minutes). -
cargo xtask e2e— Playwright suite passes. -
cargo xtask docs plan-lint— clean. -
cargo xtask docs plan-archive— moves this plan toarchive/. -
Audit grep:
grep -rn 'std::env::var("CANOPY_' services/ crates/— every remaining hit is a secret read or a runtime control variable (CANOPY_ENV,CANOPY_CI). -
Schema audit: every
config/{service}/default.yamlparses (for f in config/*/default.yaml; do yq . "$f" > /dev/null; done). -
Onboarding rehearsal: fresh clone +
age+sopsinstall +cargo xtask secrets init+cargo xtask dev startsucceeds.
Documentation Updates
-
docs/modules/ROOT/pages/developer-guide.adoc— Configuration section + first-run onboarding (cargo xtask secrets init). -
.claude/docs/services.md— note the secrets layer if any service exposes new endpoints (none currently planned). -
CHANGELOG.adoc— entry per merged MR. -
.claude/CLAUDE.md— architecture-list bullet for ADR-017 (Step 0).
Existing utilities to reuse
-
canopy-common::settings::ServiceSettings::load_with_secrets— keeps its current signature; secrets continue to flow through here. -
canopy-secrets::EnvSecretProvider— the audit-logging env-var-backed provider from phase 1 (MR !152). Use asEnvSecretProvider::new(service_name)in every service’smain.rsafter migration. -
canopy-api::bootstrap— every service’s main callscanopy_api::bootstrap("CANOPY_{SERVICE}", "canopy-{service}").await?already; the migration doesn’t change this. -
configcrate (workspace dep) — supportsConfig::builder().add_source(File::with_name(…).required(false))for layered YAML; already in `canopy-common’s deps. -
xtask/src/cmd/migrate.rs— pattern reference for structuring a new xtask subcommand with sub-actions.
Notes for the implementer
-
Don’t bundle steps. Each step is independently revertible. If Step 7 (canopy-applications) breaks something, the MR is reverted, the plan moves on; subsequent steps don’t depend on it directly.
-
YAML mounts in compose vs. baked into image. For dev, YAML mounts are fine (live edits). For production, deployers may prefer baking config into the image at build time. ADR-012 doesn’t mandate either; document both in the runbook.
-
The
EnvSecretProvideraudit log logs every secret access withtarget = "canopy.secrets". After migration, this log should fire for every signing-key / API-key / DB-password read, not for tunable reads. Verify post-Step-22 that the log is only secrets, not config. -
Secrets in
dev.yamlare FAKE. Never put real production keys, real database passwords, or real Keycloak client secrets there. The file’s contents are effectively public to anyone with developer-level access. ADR-017 documents this explicitly. -
The integration test suite must work with fake values. Existing integration tests gate on
infrastructure_available(); they exercise sign-then-verify round-trips, which work with any valid keypair (real or fake). -
Migration is opt-in per service per ADR-012. If Step 11 reveals a problem with canopy-rules, pause that step’s MR, fix the loader, resume. Steps 12+ keep waiting.
-
Don’t skip the per-step verification gate. Each step’s
cargo xtask validate+cargo xtask e2erun is the load-bearing check that the migration didn’t regress runtime behaviour. -
Pre-commit Q1-Q8 protocol: every commit gets the token-gated reflection. Q5 (issues) means closing the corresponding step issue with the MR. Q6 (improvements) — file an issue only for genuinely-blocked work; fix encountered improvements in the current MR if scope-feasible.
-
Pre-push hook environmental failures: if
cargo xtask validatefails on a known-environmental issue (docker bridge networking, kernel module not loaded), diagnose and fix root cause before pushing. Do not bypass with--no-verifyunless explicitly authorized.