Plan: Layered Config + Encrypted Secrets Migration (ADR-012 + ADR-017)

On this page

Status

Step Description Status

0

Plan ratification: ADR-017 + this plan + amendment NOTE on ADR-012 + nav/CLAUDE.md/CHANGELOG; close #291 and #346 as superseded; open 22 tracking issues

Done (2026-05-02) — MR !163

1

cargo xtask secrets tooling (init/edit/decrypt/add-recipient/generate-signing-key) + canopy-devtools compose service + secrets-yaml-lint CI job. (SOPS_AGE_KEY: $CANOPY_CI_AGE_KEY job wiring deferred to Step 3.)

Done (2026-05-02) — MR !166

2

Layered YAML loader in canopy-common: free-standing load_typed::<T>(prefix, config_root) + doc-hidden load_typed_internal test hook + CANOPY_CONFIG_ROOT env support. (#[serde(deny_unknown_fields)] on ServiceSettings deferred to Step 22 — see Step 2 NOTE.)

Done (2026-05-02) — MR !168

3

canopy-snap canary migration: config/canopy-snap/default.yaml + SnapConfig struct + main.rs refactor + docker-compose mount + secrets/dev.yaml entries. Initial nested-struct + deny_unknown_fields design fixed in MR !171 — see Step 3 NOTE.

Done (2026-05-02) — MR !170 + !171

4

canopy-exchange migration (zero env vars; validates template handles trivial case). Empty ExchangeConfig struct deliberately skipped — would be dead code; pattern lands when canopy-exchange grows tunables.

Done (2026-05-02) — MR !172

5

canopy-persons migration: shared.encryption_key in secrets/dev.yaml for CANOPY_ENCRYPTION_KEY; canopy-common::crypto gains EncryptionKeys/encryption_keys_from_env/decrypt_with_rotation for ADR-017 rotation support (current + previous); canopy-persons EncryptionKey wrapper threads both keys through. (Multi-key support implemented in this MR rather than deferred — the no-deferral rule.)

Done (2026-05-02) — MR !173

6

canopy-verification migration (single CANOPY_INTERNAL_API_KEY secret routed via EnvSecretProvider).

Done (2026-05-02) — MR !174

7

canopy-applications migration (jurisdiction + rulesets_dir → YAML, ApplicationsConfig)

Done (2026-05-02) — MR !175

8

canopy-enrollment migration (jurisdiction + rulesets_dir → YAML, EnrollmentConfig)

Done (2026-05-02) — MR !176

9

canopy-renewals migration (jurisdiction + rulesets_dir → YAML, RenewalsConfig)

Done (2026-05-02) — MR !176

10

canopy-notices migration (jurisdiction + rulesets_dir → YAML, NoticesServiceConfig — kept alongside existing NoticeConfig in same module)

Done (2026-05-02) — MR !176

11

canopy-rules migration: RulesConfig { federal_rulesets_dir, rulesets_dir, eval_workers }. Engine signature gains &RulesConfig so the env reads in engine.rs move into the typed loader path.

Done (2026-05-02) — MR !177

12

canopy-appeals migration: AppealsServiceConfig { jurisdiction, rulesets_dir, enrollment_url }. EnrollmentClient::from_env retired in favor of EnrollmentClient::new(&url).

Done (2026-05-02) — MR !177

13

canopy-security migration: SecurityConfig (breach + FTI verify intervals); FTI DB URLs route through EnvSecretProvider (audit-logged, secrets/dev.yaml under canopy-security:).

Done (2026-05-02) — MR !178

14

canopy-tanf migration: TanfConfig (rules_url, rulesets_dir) + signing_key via SOPS.

Done (2026-05-02) — MR !179

15

canopy-caps migration: CapsConfig + signing_key via SOPS.

Done (2026-05-02) — MR !179

16

canopy-wic migration: WicConfig + signing_key via SOPS.

Done (2026-05-02) — MR !179

17

canopy-medicaid migration: MedicaidConfig (jurisdiction, rulesets_dir, rules_url, persons_url) + signing_key via SOPS.

Done (2026-05-02) — MR !180

18

canopy-eligibility migration: EligibilityConfig (jurisdiction, rulesets_dir, persons_url, keys_dir). Verifying keys stay env-or-.keys/-routed via VerifyingKeyRegistry::from_env_or_keys_dir — production deployers can SOPS-source those env values without changing the per-service struct (the keys are public, not secrets in the SOPS sense).

Done (2026-05-02) — MR !181

19

canopy-portal migration: groundwork only (YAML mount + CANOPY_CONFIG_ROOT); the lone CANOPY_SESSION_SECURE env read stays in main.rs since the migration ROI is minimal for one tunable. PortalConfig added when canopy-portal grows its first non-session-secure tunable.

Done (2026-05-02) — MR !182

20

canopy-web migration: WebConfig (jurisdiction, rulesets_dir, session_secure, keycloak_client_id, keycloak_external_url, keycloak_internal_url, redirect_url). OidcConfig::from_env retired in favor of OidcConfig::from_web_config(&svc_config). Service URLs in compose env (downstream service mesh) stay env-routed; out of plan scope.

Done (2026-05-02) — MR !182

21

canopy-reporting migration: ReportingConfig (jurisdiction + rulesets_dir + 7 downstream URLs). ServiceClients::from_env retired in favor of ServiceClients::from_config(&svc_config).

Done (2026-05-02) — MR !182

22

Cleanup: retire .env.example, final docker-compose.yml pass, update developer guide, archive plan

Done (2026-05-02) — this MR

Branch: chore/adr-017-and-config-migration-plan (Step 0); feat/{descriptive} per per-step branches
Supersedes: #291, #346

Prerequisites for the implementer

This plan is self-contained: code snippets, exact file paths, struct signatures, and verification commands per step are below. A contextless agent or new contributor should be able to execute Steps 1-22 from this document alone.

Mandatory reading before any code change:

  • .claude/CLAUDE.md — project conventions, GitLab scoped labels, commit signing, glossary.

  • .claude/docs/delivery-protocol.md — preflight checks, delivery checklist.

  • .claude/docs/git-workflow.md — branch naming, commit message types (feat|fix|chore|refactor|docs|test|ci:), GPG signing, pre-commit Q1-Q8 protocol.

  • .claude/docs/security-baseline.md, .claude/docs/coding-conventions.md, .claude/docs/testing.md.

  • ADR-012, ADR-013, ADR-016, ADR-017.

  • The plan template at docs/modules/ROOT/pages/plans/_template.adoc.

Tools required on developer machine before Step 1:

  • Standard: cargo, cargo-nextest, Rust stable Edition 2024, docker + docker compose, git with GPG signing, glab CLI.

  • age and sops are not host requirements — they ship in the canopy-devtools compose service (added in Step 1) alongside yq. Mirrors the existing pattern for node/npm/Playwright (see tests/e2e/Dockerfile) and the postgres CLIs (pg_dump/pg_restore, invoked via docker exec from xtask::cmd::migrate).

Context

ADR-012 (Accepted 2026-04-23) ratified layered YAML config with env-var overrides but the loader is not yet implementedcrates/canopy-common/src/settings.rs::ServiceSettings::load uses only the config crate’s Environment source. Issue #291 tracked the migration but stayed workflow::needs-spec because the spec was incomplete.

ADR-017 (this plan ratifies it) introduces SOPS+age-encrypted secrets at rest, replacing the rejected Vault direction (issue #346).

The two concerns share the same env-var sprawl (~110 std::env::var() calls across 19 services), the same per-service touch (each service’s main.rs + Cargo.toml + docker-compose.yml entry), and the same runtime contract. Doing them in a single per-service migration MR halves the touch on each service.

Scope

In scope:

  • ADR-017 ratification + ADR-012 amendment NOTE.

  • Layered YAML loader implementation in canopy-common (the missing half of ADR-012).

  • cargo xtask secrets subcommand + secrets-yaml-lint CI job + CI age-key wiring.

  • Per-service migration of all 19 canopy services.

  • secrets/dev.yaml with fake values only for devstack and integration tests.

  • Closing #291 and #346 as superseded.

Out of scope:

  • Per-jurisdiction prod secrets (each jurisdiction operates their own private deployment-config repo, kinetic-style).

  • CLI argument layer (excluded by ADR-012).

  • Live SIGHUP/inotify config reload.

  • Multi-key SSN-encryption-key rotation support (filed as follow-up at end of Step 5).

  • PITR runbook (independent deliverable, #353).

Design

Coupling. ADR-017 amends ADR-012 — secrets at rest live in encrypted YAML; tunables in plaintext YAML; both surface to services via CANOPY_{SERVICE}__* env vars at runtime. The runtime read code is unchanged; only the at-rest representation and deploy-time injection differ.

Per-service migration MR template (canopy-snap is the canary in Step 3; Steps 4-21 substitute service names):

  1. Create config/{service}/default.yaml with the service’s tunables.

  2. Create services/{service}/src/config.rs with {Service}Config struct using #[serde(deny_unknown_fields)].

  3. Refactor services/{service}/src/main.rs: replace direct std::env::var reads of tunables with ServiceSettings::load_typed::<{Service}Config>("CANOPY_{SERVICE}", None); route secret reads through EnvSecretProvider::get_optional.

  4. Add canopy-secrets = { workspace = true } to services/{service}/Cargo.toml if not already present.

  5. Modify docker-compose.yml: remove migrated tunable env vars; mount the YAML config volume; keep secret env vars (they come from SOPS-decrypted dev.yaml at deploy time).

  6. Add the service’s secrets (fake values) to secrets/dev.yaml via cargo xtask secrets edit.

  7. Add unit test {service}_config_loads_from_default_yaml.

  8. Verify: cargo nextest run -p {service} + cargo xtask validate + cargo xtask e2e.

File layout (final state in canopy repo):

.sops.yaml                                       # SOPS recipient rules
secrets/
  dev.yaml                                       # encrypted, fake values only
config/
  canopy-{snap,tanf,medicaid,...}/
    default.yaml
docs/modules/ROOT/pages/
  adrs/adr-017-encrypted-secrets-at-rest.adoc
  plans/secret-and-config-migration.adoc         # archived after Step 22

Recipient model. Two recipients on secrets/dev.yaml: primary developer’s age public key + CI runner’s age public key. CI runner private key in GitLab masked variable CANOPY_CI_AGE_KEY; CI jobs that bring up devstack set SOPS_AGE_KEY: $CANOPY_CI_AGE_KEY.

Special-case routing — shared: section. Secrets consumed by canopy-common (e.g. CANOPY_ENCRYPTION_KEY) have no service prefix. The YAML→dotenv walk treats top-level shared: specially: leaves emit CANOPY_<KEY_UPPER> (single underscore, no service prefix). Documented as a doc-comment on xtask::cmd::secrets::walk_to_dotenv.

Config root path resolution. ServiceSettings::load_typed(prefix, config_root: Option<&Path>). Resolution priority: explicit Some(&Path) (used by tests) → CANOPY_CONFIG_ROOT env var (used by deployments mounting at /app/config) → default . (PWD; used in dev when running from repo root).

Steps

Step 0 — Plan ratification (this MR, docs-only)

Branch: chore/adr-017-and-config-migration-plan (merged 2026-05-02 as MR !163).

Files created:

  1. docs/modules/ROOT/pages/adrs/adr-017-encrypted-secrets-at-rest.adoc — full ADR (Context, Decision, Rotation Mechanics, Consequences, Alternatives Considered, Related ADRs).

  2. docs/modules/ROOT/pages/plans/secret-and-config-migration.adoc (this file).

Files modified:

  1. ADR-012 — added a NOTE block at the top of == Decision: Amended by ADR-017 (2026-05-02). Secrets at rest now ship as SOPS-encrypted YAML; the runtime contract (env vars) is preserved. The "secrets never in checked-in YAML" line below applies to plaintext YAML in \`config/\ only.` Existing prose preserved.

  2. docs/modules/ROOT/nav.adoc — added ADR-017 entry under Architecture & Design and a Foundational Migrations subsection linking this plan.

  3. .claude/CLAUDE.md — added an architecture-list bullet for ADR-017; lightly amended the ADR-012 bullet to note the amendment.

  4. CHANGELOG.adoc — entry under === Added.

GitLab actions (after merge):

  • Close #346 (Vault) and #291 (env-var → YAML migration) with supersession comments.

  • Open 22 step-tracking issues (#354 through #375) labeled [secret-and-config-migration] Step N: <heading> with type::*, priority::medium (Steps 1-3) or priority::low (Steps 4-22), program::*, service::*, workflow::ready.

Verification: cargo xtask check-docs clean, cargo xtask docs plan-lint clean (no Deferred rows), cargo xtask validate --skip-docker green.

Commit: docs: ADR-017 SOPS+age + secret-and-config-migration plan (closes #291, #346).

Step 1 — cargo xtask secrets tooling + secrets-yaml-lint CI job

Branch: feat/xtask-secrets-tooling. Tracking issue: #354.

Containerized tooling. age and sops ship in a new canopy-devtools compose service rather than as host requirements (matches the canopy-e2e Playwright pattern). xtask::devtools wraps docker compose --profile devtools run --rm so call sites stay readable.

Files created:

  1. tools/canopy-devtools/Dockerfile — alpine-based image with age, sops, yq. Versions confirmed at build time: age ≥ 1.1, sops ≥ 3.8. Yq is mikefarah’s Go binary (single-file).

  2. xtask/src/devtools.rs — wrapper module. Three primary fns:

    pub fn run(tool: &str, args: &[&str], opts: RunOpts) -> Result<Output>;
    pub fn run_interactive(tool: &str, args: &[&str], opts: RunOpts) -> Result<()>;
    
    pub struct RunOpts { pub age_key: AgeKeyMode, pub interactive: bool }
    pub enum AgeKeyMode { None, ReadOnly, ReadWrite }

    The wrapper resolves host UID/GID via id -u/id -g and passes --user UID:GID to docker so files written by the container land owned by the developer. For sops-decrypt operations, AgeKeyMode::ReadOnly bind-mounts ~/.config/sops/age and sets SOPS_AGE_KEY_FILE=/sops-age/keys.txt inside the container. AgeKeyMode::ReadWrite is used only by secrets init to write a fresh keys.txt.

  3. xtask/src/cmd/secrets.rs — new module. Five subcommands matching the action enum:

    // SPDX-License-Identifier: AGPL-3.0-or-later
    
    #[derive(clap::Args)]
    pub struct Args {
        #[command(subcommand)]
        pub action: Action,
    }
    
    #[derive(clap::Subcommand)]
    pub enum Action {
        /// Generate dev age keypair (if missing) and print the public key.
        /// Pass `--for-ci` to instead emit a transient keypair for the
        /// GitLab `CANOPY_CI_AGE_KEY` masked variable.
        Init { #[arg(long = "for-ci")] for_ci: bool },
        /// Open `secrets/dev.yaml` in `$EDITOR` via sops (decrypts on read,
        /// re-encrypts on save).
        Edit,
        /// Print decrypted secrets/dev.yaml as a flat dotenv stream.
        Decrypt,
        /// Append a new age recipient to `.sops.yaml` and re-encrypt the data key.
        AddRecipient { age_key: String },
        /// Print a fresh ECDSA P-256 PEM (paste into `secrets edit`).
        GenerateSigningKey,
    }

    Decrypt-walk algorithm. SOPS is invoked with --output-type json and the result is parsed with serde_json::Value (no serde_yaml dep — it’s archived; the lockfile literally says 0.9.34+deprecated). Walk rules:

    • Service-keyed (top-level matches canopy-<name>): leaves emit CANOPY_<SVC_UPPER>__<KEY_UPPER>=<value>.

    • Shared (top-level is shared): leaves emit CANOPY_<KEY_UPPER>=<value> (single underscore, no service prefix). Used for secrets consumed by canopy-common.

    • Skip: the top-level sops: key (SOPS metadata).

    • Multi-line value escaping: replace \n with literal \\n and wrap in double quotes.

      GenerateSigningKey calls canopy_signing::keygen::generate_key_pair() directly (no container needed; pure Rust).

  4. .sops.yaml at workspace root with both age recipients (developer + CI):

    creation_rules:
      - path_regex: ^secrets/dev\.yaml$
        age: >-
          <DEVELOPER-AGE-PUBLIC-KEY>,
          <CI-RUNNER-AGE-PUBLIC-KEY>

    The CI runner’s age private key is stored in GitLab as masked variable CANOPY_CI_AGE_KEY (Settings → CI/CD → Variables → Masked + Protected). Generated via cargo xtask secrets init --for-ci (transient — never persisted to disk on the dev machine).

  5. secrets/dev.yaml — encrypted, placeholder contents only. Per-service entries land in Steps 3-21.

Files modified:

  1. docker-compose.yml — add canopy-devtools service entry under a devtools profile (so it doesn’t start on dev start):

    canopy-devtools:
      profiles: [devtools]
      build:
        context: ./tools/canopy-devtools
      image: canopy-devtools:latest
      volumes:
        - .:/work
  2. xtask/src/cmd/mod.rs — register pub mod secrets; and add Secrets(secrets::Args) to the Command enum.

  3. xtask/src/main.rs — register mod devtools; and add Command::Secrets(args) ⇒ cmd::secrets::run(args) to the dispatch.

  4. .gitlab-ci.yml — new secrets-yaml-lint job (insert after adr-013-plan-lint):

    secrets-yaml-lint:
      image: alpine:3.21
      stage: test
      script:
        - |
          set -euo pipefail
          VIOLATIONS=0
          if [ -d config ]; then
            for f in $(find config -name '*.yaml' 2>/dev/null); do
              if grep -inE '(password|secret|signing_key|private_key|_token|api_key)\s*:\s*[A-Za-z0-9/+]{16,}' "$f" | grep -v '# allow-secret:'; then
                echo "VIOLATION: plaintext secret-shaped value in $f"
                VIOLATIONS=$((VIOLATIONS + 1))
              fi
              if grep -in 'BEGIN .*PRIVATE KEY' "$f" | grep -v '# allow-secret:'; then
                echo "VIOLATION: PEM private key embedded in $f"
                VIOLATIONS=$((VIOLATIONS + 1))
              fi
            done
          fi
          if [ "$VIOLATIONS" -gt 0 ]; then
            echo "ERROR: $VIOLATIONS plaintext-secret violation(s)"
            exit 1
          fi
          echo "OK: no plaintext secrets in config/"
      rules:
        - if: $CI_COMMIT_BRANCH
        - if: $CI_MERGE_REQUEST_IID

    Heuristic is conservative; false positives escape with a # allow-secret: <reason> annotation. If it proves too noisy, follow up by migrating to gitleaks or trufflehog. Uses a stock alpine:3.21 image rather than .rust-base since it’s pure regex (no Rust toolchain needed; ~2s job).

    NOTE
    SOPS_AGE_KEY env wiring on the cargo-test / e2e jobs is deferred to Step 3. No CI job invokes sops until the canopy-snap canary’s xtask::cmd::dev decrypt-at-start integration ships. Wiring it here would create a dead-code "what if CANOPY_CI_AGE_KEY isn’t set" risk for zero current benefit.

Operational prerequisite (before Step 3 merges, not this MR): configure GitLab masked variable CANOPY_CI_AGE_KEY from the output of cargo xtask secrets init --for-ci.

Tests:

  • cargo xtask secrets init round-trip on a clean machine: generates age keypair, writes to ~/.config/sops/age/keys.txt, prints public key.

  • cargo xtask secrets init --for-ci emits a transient keypair with paste instructions.

  • cargo xtask secrets decrypt round-trips secrets/dev.yaml (placeholder payload) and emits dotenv-shaped output.

  • Unit tests in xtask/src/cmd/secrets.rs::tests (no docker required) cover the pure-Rust helpers:

    Test Asserts

    decrypt_walk_emits_dotenv_for_simple_value

    service-keyed top-level → CANOPY_<SVC>__<KEY>=<value>

    decrypt_walk_emits_shared_without_service_prefix

    shared.<key>CANOPY_<KEY> (single underscore)

    decrypt_walk_skips_sops_metadata

    sops: block excluded from output

    decrypt_walk_handles_multi_line_pem

    PEM blocks → \n-escaped + double-quoted

    format_dotenv_quotes_values_with_spaces

    values with whitespace / quotes get escaped

    append_recipient_to_inline_age

    inline age: a,b form correctly extended to folded scalar

    append_recipient_to_folded_age

    folded age: >-\n a,\n b form gets new line at correct indent

    parse_public_key_from_age_keygen_stdout

    # public key: age1…​ parser handles age-keygen output

Verification:

  • cargo nextest run -p xtask cmd::secrets — 9 unit tests pass.

  • docker compose --profile devtools build canopy-devtools succeeds; tools report expected versions (age 1.2.x, sops 3.9.x, yq 4.x).

  • cargo xtask secrets init — generates dev keypair if missing; idempotent on second run.

  • cargo xtask secrets init --for-ci — emits ephemeral keypair with masked-variable paste instructions.

  • cargo xtask secrets decrypt — round-trips placeholder secrets/dev.yaml and emits dotenv lines.

  • cargo xtask secrets edit — opens $EDITOR in the container; saving re-encrypts in place.

  • cargo xtask validate --skip-docker — green.

  • Manual: push a deliberate config/test/default.yaml with password: AAAAAAAAAAAAAAAAAAAA, observe secrets-yaml-lint job fails; revert.

Commit: feat: cargo xtask secrets tooling + canopy-devtools container + secrets-yaml-lint CI job (closes #354).

Step 2 — Layered YAML loader in canopy-common

Branch: feat/canopy-common-layered-yaml-loader. Tracking issue: #355.

NOTE
#[serde(deny_unknown_fields)] on ServiceSettings is deferred to Step 22 (cleanup pass). Reason: every service still passes service-specific env vars (CANOPY_SNAPRULES_URL, CANOPY_TANFSIGNING_KEY, etc.) through the same prefix as the shared CANOPY_<SVC>PORT / DATABASE_URL / etc. fields that ServiceSettings consumes. Adding deny_unknown_fields to ServiceSettings here would crash every service’s canopy_api::bootstrap call until the per-service config structs (Steps 3-21) replace those direct env reads. Per-service structs introduced from Step 3 onwards DO carry deny_unknown_fields so the typo-catching benefit lands service-by-service.
NOTE
tests use a load_typed_internal(prefix, config_root, env_name, env_override) hook that injects env vars via config::Environment::source(Some(map)) instead of mutating std::env. Required by crates/canopy-common/src/lib.rs:9 (#![forbid(unsafe_code)]), which rules out the unsafe { std::env::set_var(…​) } pattern that the original plan sketch used. Production callers use the public [load_typed] without the override.

Files modified:

  1. crates/canopy-common/src/settings.rs — extend ServiceSettings::load to be the layered loader per ADR-012. Current implementation reads only env vars:

    // BEFORE:
    pub fn load(prefix: &str) -> Result<Self, config::ConfigError> {
        Config::builder()
            .add_source(Environment::with_prefix(prefix).separator("__").try_parsing(true))
            .build()?
            .try_deserialize()
    }

    New implementation: a free-standing module-level load_typed<T> (the public API) plus a #[doc(hidden)] load_typed_internal that takes the env-overlay name and an optional env-var override map. The override is the test hook (forced by forbid(unsafe_code); see the NOTE above).

    pub fn load(prefix: &str) -> Result<Self, config::ConfigError> {
        load_typed::<Self>(prefix, None)
    }
    
    /// Generic typed loader. See module docs for layering rules.
    pub fn load_typed<T: serde::de::DeserializeOwned>(
        prefix: &str,
        config_root: Option<&Path>,
    ) -> Result<T, config::ConfigError> {
        let env_name = std::env::var("CANOPY_ENV").unwrap_or_else(|_| "dev".to_string());
        load_typed_internal::<T>(prefix, config_root, &env_name, None)
    }
    
    #[doc(hidden)]
    pub fn load_typed_internal<T: serde::de::DeserializeOwned>(
        prefix: &str,
        config_root: Option<&Path>,
        env_name: &str,
        env_override: Option<config::Map<String, String>>,
    ) -> Result<T, config::ConfigError> {
        let service = prefix.to_lowercase().replace('_', "-");
        let root: PathBuf = config_root
            .map(Path::to_path_buf)
            .or_else(|| std::env::var_os("CANOPY_CONFIG_ROOT").map(PathBuf::from))
            .unwrap_or_else(|| PathBuf::from("."));
        let default_path = root.join(&service).join("default");
        let env_path = root.join(&service).join(env_name);
        let env_source = Environment::with_prefix(prefix)
            .separator("__")
            .try_parsing(true)
            .source(env_override);
        let builder = Config::builder()
            .add_source(File::with_name(default_path.to_str().unwrap()).required(false))
            .add_source(File::with_name(env_path.to_str().unwrap()).required(false))
            .add_source(env_source);
        let config = builder.build()?;
        // DEBUG attribution per ADR-012 §Negative.
        if tracing::enabled!(tracing::Level::DEBUG)
            && let Ok(map) = config.clone().try_deserialize::<serde_json::Value>()
        {
            tracing::debug!(prefix = %prefix, service = %service, env = %env_name,
                            config = %map, "config loaded");
        }
        config.try_deserialize()
    }

    Both YAML files are optional — services that have no tunables today rely on env vars alone, and the loader silently succeeds if neither file exists. load_with_secrets keeps its current signature; secrets continue to come from SecretProvider, never from YAML.

  2. crates/canopy-common/Cargo.toml — add tempfile = { workspace = true } to [dev-dependencies]. config, tracing, and serde_json already present.

Tests: extend the existing test module in settings.rs. Tests pass Some(tmpdir.path()) as config_root and inject env vars via Some(env_map) on load_typed_internal. No std::env::set_var (forbidden by crate-level #[forbid(unsafe_code)]); env vars never escape the test scope.

fn env_map(pairs: &[(&str, &str)]) -> Map<String, String> {
    pairs.iter().map(|(k, v)| ((*k).to_string(), (*v).to_string())).collect()
}

#[derive(Debug, Deserialize, PartialEq)]
#[serde(deny_unknown_fields)]
struct TestConfig { port: u16, jurisdiction: String }

#[test]
fn load_typed_falls_back_to_env_only_when_yaml_absent() {
    let tmp = tempdir().unwrap();
    let env = env_map(&[("CANOPY_TEST__PORT", "7000"),
                         ("CANOPY_TEST__JURISDICTION", "georgia")]);
    let cfg: TestConfig = load_typed_internal("CANOPY_TEST", Some(tmp.path()),
                                              "dev", Some(env)).unwrap();
    assert_eq!(cfg, TestConfig { port: 7000, jurisdiction: "georgia".into() });
}
// + load_typed_layers_yaml_under_env_override
// + load_typed_env_overlay_wins_over_default
// + load_typed_dev_overlay_falls_back_to_default_when_missing
// + load_typed_deny_unknown_fields_catches_typo_in_yaml
// + load_typed_service_path_derives_from_prefix

Verification:

  • cargo nextest run -p canopy-common — all tests pass (71 → 71 + 6 new).

  • cargo xtask validate — green.

  • Manual: temporarily create config/canopy-test/default.yaml with a port; observe a service that uses prefix CANOPY_TEST picks it up; revert.

Commit: feat(canopy-common): layered YAML config loader (closes #355).

Step 3 — canopy-snap canary migration

Branch: feat/canopy-snap-config-migration. Tracking issue: #356.

This is the canary; Steps 4-21 follow this template exactly with service-specific substitutions.

Files created:

  1. config/canopy-snap/default.yaml — non-secret tunables. Schema:

    jurisdiction: georgia
    rulesets_dir: rulesets
    rules:
      url: http://canopy-rules:8001
    verification:
      url: http://canopy-verification:8005

    Confirmed by reading services/canopy-snap/src/main.rs lines 56, 59, 73, 77 — these are the four current tunable env reads. No secrets in this file.

  2. services/canopy-snap/src/config.rs — typed config struct:

    // SPDX-License-Identifier: AGPL-3.0-or-later
    
    //! Service-specific config schema for canopy-snap (ADR-012).
    
    use std::path::PathBuf;
    use serde::Deserialize;
    
    #[derive(Debug, Deserialize)]
    #[serde(deny_unknown_fields)]
    pub struct SnapConfig {
        pub jurisdiction: String,
        pub rulesets_dir: PathBuf,
        pub rules: RulesClientConfig,
        pub verification: VerificationClientConfig,
    }
    
    #[derive(Debug, Deserialize)]
    #[serde(deny_unknown_fields)]
    pub struct RulesClientConfig {
        pub url: String,
    }
    
    #[derive(Debug, Deserialize)]
    #[serde(deny_unknown_fields)]
    pub struct VerificationClientConfig {
        pub url: String,
    }

Files modified:

  1. services/canopy-snap/Cargo.toml — add canopy-secrets = { workspace = true } to [dependencies].

  2. services/canopy-snap/src/lib.rs (or wherever modules are declared) — add pub mod config;.

  3. services/canopy-snap/src/main.rs — refactor lines 56-85. Today:

    // BEFORE:
    let jurisdiction = std::env::var("CANOPY_SNAP__JURISDICTION").expect("CANOPY_SNAP__JURISDICTION is required");
    let rulesets_dir = std::env::var("CANOPY_RULESETS_DIR").unwrap_or_else(|_| "rulesets".into());
    let rules_url = std::env::var("CANOPY_SNAP__RULES_URL").unwrap_or_else(|_| "http://localhost:8001".into());
    let verification_url = std::env::var("CANOPY_SNAP__VERIFICATION_URL").unwrap_or_else(|_| "http://localhost:8005".into());
    let internal_api_key = std::env::var("CANOPY_INTERNAL_API_KEY").unwrap_or_else(|_| "canopy-internal-dev-key".into());
    let signing_key_pem = std::env::var("CANOPY_SNAP__SIGNING_KEY").unwrap_or_default();

    After:

    // AFTER:
    use canopy_snap::config::SnapConfig;
    use canopy_secrets::{EnvSecretProvider, SecretProvider};
    
    let snap_config: SnapConfig =
        canopy_common::settings::ServiceSettings::load_typed("CANOPY_SNAP", None)
            .expect("load canopy-snap config");
    
    let secrets: EnvSecretProvider = EnvSecretProvider::new("canopy-snap");
    let internal_api_key = secrets.get_optional("CANOPY_INTERNAL_API_KEY")
        .expect("read CANOPY_INTERNAL_API_KEY")
        .unwrap_or_else(|| "canopy-internal-dev-key".into());
    let signing_key_pem = secrets.get_optional("CANOPY_SNAP__SIGNING_KEY")
        .expect("read CANOPY_SNAP__SIGNING_KEY")
        .unwrap_or_default();

    Update every downstream reference: jurisdictionsnap_config.jurisdiction, rulesets_dirsnap_config.rulesets_dir, rules_url&snap_config.rules.url, verification_url&snap_config.verification.url. Note rulesets_dir was a String and is now a PathBuf — coerce at call sites if needed.

  4. docker-compose.yml — locate the canopy-snap service block. Remove these lines from the environment: block:

    • CANOPY_SNAP__JURISDICTION: "georgia"

    • CANOPY_SNAP__RULES_URL: "http://canopy-rules:8001"

    • CANOPY_RULESETS_DIR (if explicitly set)

    • CANOPY_SNAP__VERIFICATION_URL (if explicitly set)

      Add a volumes: entry to mount the YAML config:

      volumes:
        - ./config/canopy-snap:/app/config/canopy-snap:ro

      Keep these lines (still env-var-injected at runtime; they are secrets or runtime control vars):

    • CANOPY_SNAP__DATABASE_URL (DB password)

    • CANOPY_SNAP__RABBITMQ_URL (broker password)

    • CANOPY_SNAPKEYCLOAK_ISSUER, CANOPY_SNAPKEYCLOAK_URL (auth endpoints)

    • CANOPY_SNAP__SIGNING_KEY (sources from SOPS via xtask dev start)

    • CANOPY_INTERNAL_API_KEY (secret)

    • CANOPY_ENV: "development" (controls which YAML overlay loads)

  5. secrets/dev.yaml (encrypted; edit via cargo xtask secrets edit):

    canopy-snap:
      signing_key: |
        -----BEGIN PRIVATE KEY-----
        <generated PEM>
        -----END PRIVATE KEY-----
      internal_api_key: canopy-internal-dev-key-NOT-REAL

    Generate the fake signing key with cargo xtask secrets generate-signing-key (added in Step 1) and paste into the editor session.

  6. xtask/src/cmd/dev.rs — locate the dev start action. Before invoking docker compose up, run the SOPS decrypt step and write the output to a tmpfile that compose consumes via --env-file:

    // Near the top of dev_start, after preflight checks:
    let env_file = tempfile::Builder::new()
        .prefix("canopy-dev-")
        .suffix(".env")
        .tempfile()
        .context("create tempfile for sops dotenv")?;
    let decrypted = ProcessCommand::new("sops")
        .args(["--decrypt", "secrets/dev.yaml"])
        .output()
        .context("invoke sops --decrypt secrets/dev.yaml")?;
    if !decrypted.status.success() {
        let stderr = String::from_utf8_lossy(&decrypted.stderr);
        bail!("sops decrypt failed: {stderr}");
    }
    let yaml: serde_yaml::Value = serde_yaml::from_slice(&decrypted.stdout)?;
    let mut dotenv = crate::cmd::secrets::walk_to_dotenv(&yaml)?;
    dotenv.push_str("CANOPY_CONFIG_ROOT=/app/config\n");
    std::fs::write(env_file.path(), dotenv)?;
    // Pass --env-file=<env_file.path()> to the docker compose invocation.
    // `env_file` is dropped at end of dev_start, which removes the tempfile.

    Extract the YAML→dotenv walk from cmd/secrets.rs into a public walk_to_dotenv(yaml: &serde_yaml::Value) → Result<String> so dev.rs can call it without code duplication. Use tempfile::NamedTempFile for cleanup (its Drop impl removes the file — preferable to a ctrlc handler).

Tests:

  • All existing canopy-snap tests must still pass with no edits to test code itself. They read the same env vars at runtime; the change is at the loading layer.

  • New unit test in services/canopy-snap/src/config.rs::tests:

    #[test]
    fn snap_config_loads_from_default_yaml() {
        use canopy_common::settings::ServiceSettings;
        let _guard = test_mutex();
        let _cd = ChangeDir::to(repo_root());
        let cfg: SnapConfig = ServiceSettings::load_typed("CANOPY_SNAP", None).expect("load");
        assert_eq!(cfg.jurisdiction, "georgia");
        assert_eq!(cfg.rules.url, "http://canopy-rules:8001");
    }

Verification:

  • cargo nextest run -p canopy-snap — all tests pass.

  • cargo xtask dev start — devstack comes up healthy; canopy-snap reads its config from YAML + SOPS-decrypted secrets; docker compose logs canopy-snap | head shows no missing-env-var warnings.

  • cargo xtask validate — full battery green.

  • cargo xtask e2e — Playwright tests pass; SNAP-determination flow exercises the signing path end-to-end with the SOPS-injected fake key.

  • Manual: docker compose logs canopy-snap | grep 'config loaded' (DEBUG) — confirms the layered loader fired.

Commit: feat(canopy-snap): migrate to layered YAML config + SOPS-encrypted signing key (closes #356).

Steps 4-21 — Per-service migrations (one MR each)

NOTE (lessons from Step 3, post MR !171): the per-service config struct (a) does NOT carry #[serde(deny_unknown_fields)] during the migration window — the layered loader’s env source feeds every CANOPY_<SVC>* variable through, including the shared baseline keys (port, database_url, rabbitmq_url, keycloak_*) consumed by ServiceSettings; deny_unknown_fields rejects those. (b) Fields are flat (e.g. rules_url: String), not nested (e.g. rules.url) — the existing env-var convention is single separator, and nested structs would require RULESURL (double __), a breaking change to the ops contract. Both are restored in Step 22 once the prefix-sharing is resolved.

Migration template (mirrors Step 3 with service-specific substitutions):

  1. Create config/{service}/default.yaml with the service’s tunables (read its main.rs to enumerate; cross-check against the inventory below).

  2. Create services/{service}/src/config.rs with {Service}Config (flat fields; no deny_unknown_fields — see lessons above). Skip the struct entirely when a service has zero tunables — adding an empty struct that’s never instantiated is dead code (canopy-exchange in Step 4 takes this route).

  3. Modify services/{service}/src/main.rs (and lib.rs to declare the new module): replace direct std::env::var reads of tunables with canopy_common::settings::load_typed::<{Service}Config>("CANOPY_{SERVICE}", None); route secret reads through EnvSecretProvider::get_optional.

  4. Modify services/{service}/Cargo.toml: add canopy-secrets = { workspace = true } and config = { workspace = true } if not present.

  5. Modify docker-compose.yml: remove migrated tunable env vars from the service’s environment: block; mount ./config/{service}:/app/config/{service}:ro; set CANOPY_CONFIG_ROOT: "/app/config"; add ${CANOPY_<SVC>__SIGNING_KEY:-} / ${CANOPY_INTERNAL_API_KEY:-canopy-internal-dev-key} substitutions for any secrets the service consumes.

  6. Add the service’s secrets (fake values) to secrets/dev.yaml via cargo xtask secrets edit. Use canopy-{svc} for service-specific secrets, shared: for cross-service ones (e.g. CANOPY_INTERNAL_API_KEY, CANOPY_ENCRYPTION_KEY).

  7. Add a unit test {service}_config_loads_from_default_yaml mirroring Step 3’s pattern.

Per-MR verification (template, identical to Step 3):

  • cargo nextest run -p {service} — all existing + new tests pass.

  • cargo xtask dev start — devstack stays healthy.

  • cargo xtask validate — green.

  • cargo xtask e2e — Playwright passes.

Per-MR commit message template: feat({service}): migrate to layered YAML config + SOPS secrets (closes #<step-issue>). Branch naming: feat/{service}-config-migration.

Service inventory (canonical migration order — easiest first, hardest last):

Step Service Secrets Tunables Special notes

4 (#357)

canopy-exchange

0

0

Stub service; trivial migration; validates the template handles the zero-content case. The MR delivers an empty default.yaml (with # placeholder comment to avoid empty-file issues) and an empty ExchangeConfig struct.

5 (#358)

canopy-persons

0 (in service main; encryption key consumed by canopy-common)

1 (CANOPY_ENV)

Special: crates/canopy-common/src/crypto.rs reads CANOPY_ENCRYPTION_KEY for AES-256-GCM SSN encryption. Consumed by every service that imports canopy-common’s crypto module, not just canopy-persons. Add CANOPY_ENCRYPTION_KEY to secrets/dev.yaml under a top-level shared: section: shared:\n encryption_key: <fake>. Update xtask secrets decrypt walk to map shared.<key> to CANOPY_<KEY_UPPER>. After Step 5, file a follow-up issue: "feat(canopy-common): multi-key SSN encryption support for rotation" (referenced in ADR-017 Rotation Mechanics).

6 (#359)

canopy-verification

1 (CANOPY_INTERNAL_API_KEY)

0

Single-secret migration.

7 (#360)

canopy-applications

0

2 (JURISDICTION, RULESETS_DIR)

Pure tunable migration.

8 (#361)

canopy-enrollment

0

2

Same shape as Step 7.

9 (#362)

canopy-renewals

0

2

Same shape.

10 (#363)

canopy-notices

0

2

Same shape.

11 (#364)

canopy-rules

0

3 (FEDERAL_RULESETS_DIR, RULESETS_DIR, EVAL_WORKERS)

12 (#365)

canopy-appeals

0

3 (ENROLLMENT_URL, JURISDICTION, RULESETS_DIR)

13 (#366)

canopy-security

2 (CANOPY_SECURITYFTI_TANF_DATABASE_URL, CANOPY_SECURITYFTI_MEDICAID_DATABASE_URL)

3 (BREACH_DETECTION_INTERVAL_SECS, FTI_VERIFY_INTERVAL_SECS, CANOPY_PORT_POSTGRES_5432)

FTI-scoped DB URLs are secrets (read-only credentials).

14 (#367)

canopy-tanf

1 (CANOPY_TANF__SIGNING_KEY)

2 (RULES_URL, RULESETS_DIR)

15 (#368)

canopy-caps

1 (signing key)

2

Same shape as Step 14.

16 (#369)

canopy-wic

1 (signing key)

2

Same shape.

17 (#370)

canopy-medicaid

1 (signing key)

4 (JURISDICTION, PERSONS_URL, RULESETS_DIR, RULES_URL)

18 (#371)

canopy-eligibility

N (verifying keys for 5 programs: CANOPY_VERIFY_KEY_<PROGRAM>_<CURRENT|PREVIOUS>)

4 (JURISDICTION, PERSONS_URL, KEYS_DIR, RULESETS_DIR)

Verifying keys are public but managed alongside signing keys; place under canopy-eligibility: in secrets/dev.yaml for symmetry. Consumer is VerifyingKeyRegistry::from_env (crates/canopy-signing/src/verifier.rs).

19 (#372)

canopy-portal

0

1 (CANOPY_SESSION_SECURE)

Plus session-secret if/when wired.

20 (#373)

canopy-web

1 (Keycloak client secret if present)

7 (RULESETS_DIR, SESSION_SECURE, JURISDICTION, KEYCLOAK_CLIENT_ID, KEYCLOAK_EXTERNAL_URL, KEYCLOAK_INTERNAL_URL, REDIRECT_URL)

Largest tunable surface among UI services.

21 (#374)

canopy-reporting

0

9 service URLs + 1 JURISDICTION

Largest service-URL mesh. Pure tunable migration.

Step 3 already migrated canopy-snap. Total services migrated: 1 canary + 18 in Steps 4-21 = 19. Verify against find services -maxdepth 1 -type d -name 'canopy-*' before starting the run; if a 20th service has been added, insert it into the order after Step 21 by complexity.

Step 22 — Cleanup pass + retire .env.example + archive plan

Branch: chore/secret-and-config-migration-cleanup. Tracking issue: #375.

Files modified:

  1. .env.example — delete. Schema source of truth is now per-service config/{service}/default.yaml.

  2. docker-compose.yml — final cleanup pass. Comb every service’s environment: block; if any block has been reduced to only secrets, leave it. If any block is now empty, remove it. Update top-of-file comments to point at config/{service}/default.yaml for the schema.

  3. developer-guide.adoc — update the "Configuration" section (or add one if absent) to describe the YAML+env layered model. Add a "First run on a fresh checkout" subsection that walks through cargo xtask secrets init. Cross-link to ADR-012 and ADR-017.

  4. docs/modules/ROOT/pages/plans/secret-and-config-migration.adoc (this file) — flip every Status row to Done (YYYY-MM-DD). Run cargo xtask docs plan-archive to move the file into archive/.

  5. docs/modules/ROOT/pages/plans/archive.adoc — add a row in the appropriate section.

  6. CHANGELOG.adoc — final entry under === Changed: * Secret and config migration complete. All 19 canopy services migrated to layered YAML config + SOPS-encrypted secrets.

Verification:

  • cargo xtask validate — green.

  • cargo xtask docs plan-lint — clean.

  • cargo xtask docs plan-archive — successfully moves the migration plan into archive/.

  • cargo xtask e2e — green.

  • Audit grep: grep -rn 'std::env::var("CANOPY_' services/ crates/ — every remaining hit is a secret read or a runtime control variable (CANOPY_ENV, CANOPY_CI).

  • Schema audit: for f in config/*/default.yaml; do yq . "$f" > /dev/null || echo "BROKEN: $f"; done.

  • Onboarding rehearsal: clone the repo on a fresh machine, install age + sops, run cargo xtask secrets init, propose new public key in .sops.yaml via PR, get added by an existing recipient, run cargo xtask dev start, run cargo xtask test — full happy path on a truly fresh checkout.

Commit: chore: retire .env.example, archive secret-and-config-migration plan.

Files Touched

File Change

crates/canopy-common/src/settings.rs

Step 2: layered loader extension; add load_typed<T>(prefix, config_root) and #[serde(deny_unknown_fields)] to ServiceSettings

xtask/src/cmd/mod.rs, xtask/src/cmd/secrets.rs (new)

Step 1: secrets subcommand registration and implementation

xtask/src/cmd/dev.rs

Step 3: decrypt-at-dev-start integration

.gitlab-ci.yml

Step 1: secrets-yaml-lint job + SOPS_AGE_KEY env on devstack-using jobs

.sops.yaml (new)

Step 1: SOPS recipient rules with developer + CI recipients

secrets/dev.yaml (new, encrypted)

Step 1 creates empty; Steps 3-21 grow it with each service’s fake secrets

config/{service}/default.yaml (new × 19)

Steps 3-21: per-service tunables

services/{service}/src/main.rs × 19

Steps 3-21: refactor env reads to typed config + secret provider

services/{service}/src/config.rs (new × 17)

Steps 3-21: per-service config struct (without deny_unknown_fields — see plan lessons-learned). canopy-exchange (Step 4) and canopy-portal (Step 19) skip the struct since they have zero tunables.

services/{service}/Cargo.toml × 19

Steps 3-21: add canopy-secrets dep where missing

docker-compose.yml

Steps 3-22: remove tunable env vars; mount YAML configs

docs/modules/ROOT/pages/adrs/adr-017-encrypted-secrets-at-rest.adoc (new)

Step 0

docs/modules/ROOT/pages/plans/secret-and-config-migration.adoc (new; archived in Step 22)

Step 0

docs/modules/ROOT/pages/adrs/adr-012-layered-yaml-configuration.adoc

Step 0: NOTE block referencing ADR-017

docs/modules/ROOT/nav.adoc

Step 0: ADR-017 + plan entries

.claude/CLAUDE.md

Step 0: ADR-017 architecture bullet

CHANGELOG.adoc

Every step: one entry per merged MR

.env.example

Step 22: deleted

Verification

End-to-end, after Step 22:

  1. cargo xtask validate --timing — green, full battery (~5-6 minutes).

  2. cargo xtask e2e — Playwright suite passes.

  3. cargo xtask docs plan-lint — clean.

  4. cargo xtask docs plan-archive — moves this plan to archive/.

  5. Audit grep: grep -rn 'std::env::var("CANOPY_' services/ crates/ — every remaining hit is a secret read or a runtime control variable (CANOPY_ENV, CANOPY_CI).

  6. Schema audit: every config/{service}/default.yaml parses (for f in config/*/default.yaml; do yq . "$f" > /dev/null; done).

  7. Onboarding rehearsal: fresh clone + age + sops install + cargo xtask secrets init + cargo xtask dev start succeeds.

Documentation Updates

  • docs/modules/ROOT/pages/developer-guide.adoc — Configuration section + first-run onboarding (cargo xtask secrets init).

  • .claude/docs/services.md — note the secrets layer if any service exposes new endpoints (none currently planned).

  • CHANGELOG.adoc — entry per merged MR.

  • .claude/CLAUDE.md — architecture-list bullet for ADR-017 (Step 0).

Existing utilities to reuse

  • canopy-common::settings::ServiceSettings::load_with_secrets — keeps its current signature; secrets continue to flow through here.

  • canopy-secrets::EnvSecretProvider — the audit-logging env-var-backed provider from phase 1 (MR !152). Use as EnvSecretProvider::new(service_name) in every service’s main.rs after migration.

  • canopy-api::bootstrap — every service’s main calls canopy_api::bootstrap("CANOPY_{SERVICE}", "canopy-{service}").await? already; the migration doesn’t change this.

  • config crate (workspace dep) — supports Config::builder().add_source(File::with_name(…​).required(false)) for layered YAML; already in `canopy-common’s deps.

  • xtask/src/cmd/migrate.rs — pattern reference for structuring a new xtask subcommand with sub-actions.

Notes for the implementer

  • Don’t bundle steps. Each step is independently revertible. If Step 7 (canopy-applications) breaks something, the MR is reverted, the plan moves on; subsequent steps don’t depend on it directly.

  • YAML mounts in compose vs. baked into image. For dev, YAML mounts are fine (live edits). For production, deployers may prefer baking config into the image at build time. ADR-012 doesn’t mandate either; document both in the runbook.

  • The EnvSecretProvider audit log logs every secret access with target = "canopy.secrets". After migration, this log should fire for every signing-key / API-key / DB-password read, not for tunable reads. Verify post-Step-22 that the log is only secrets, not config.

  • Secrets in dev.yaml are FAKE. Never put real production keys, real database passwords, or real Keycloak client secrets there. The file’s contents are effectively public to anyone with developer-level access. ADR-017 documents this explicitly.

  • The integration test suite must work with fake values. Existing integration tests gate on infrastructure_available(); they exercise sign-then-verify round-trips, which work with any valid keypair (real or fake).

  • Migration is opt-in per service per ADR-012. If Step 11 reveals a problem with canopy-rules, pause that step’s MR, fix the loader, resume. Steps 12+ keep waiting.

  • Don’t skip the per-step verification gate. Each step’s cargo xtask validate + cargo xtask e2e run is the load-bearing check that the migration didn’t regress runtime behaviour.

  • Pre-commit Q1-Q8 protocol: every commit gets the token-gated reflection. Q5 (issues) means closing the corresponding step issue with the MR. Q6 (improvements) — file an issue only for genuinely-blocked work; fix encountered improvements in the current MR if scope-feasible.

  • Pre-push hook environmental failures: if cargo xtask validate fails on a known-environmental issue (docker bridge networking, kernel module not loaded), diagnose and fix root cause before pushing. Do not bypass with --no-verify unless explicitly authorized.

Edit this page · default