Plan: ADR-005 Graceful-Degradation Verification

On this page

Status

Step Description Status

1

Enumerate every capability flag (CANOPY_PROGRAM_URL_*) and the program each gates. Publish as compliance/deployment-profile-capabilities.toml.

Done (verified 2026-04-26) — compliance/deployment-profile-capabilities.toml ships in repo with the canonical capability-flag manifest.

2

For each capability, add an integration test asserting the orchestrator returns the program in programs_pending with basis "program service not configured" when that capability is absent from the ProgramServiceRegistry

Done (verified 2026-04-26) — 7 capability-flag tests in services/canopy-eligibility/tests/capability_flag_test.rs (6 single-capability-absent + 1 all-absent).

3

Add a matrix of integration tests driven by COMPOSE_PROFILES: snap-only, tanf-only, medicaid-chip, caps-only, wic-only each exercised end-to-end

Deferred (see Errata) — tracked at #350.

4

CI: add a compose-profile-matrix job running each profile in sequence (or parallel if agent capacity allows)

Deferred (see Errata) — tracked at #350.

5

Update deployment-profiles-event-wiring.adoc Status table to replace "Complete" on Step 3 with "Complete (verified by this plan)"

Done (2026-04-26) — Steps 1, 2 of this plan are the verification reference; the deployment-profiles-event-wiring plan was archived 2026-04-24 in the bulk-archive sweep, where its Status rows are already Done. No further sync required.

Errata

Env var naming (corrected)

Plan originally named capability flags CANOPY_SNAP_URL, CANOPY_TANF_URL, etc. The actual registry ([services/canopy-eligibility/src/registry.rs#L33-L40](../../../../services/canopy-eligibility/src/registry.rs#L33-L40)) reads CANOPY_PROGRAM_URL_SNAP, CANOPY_PROGRAM_URL_TANF, …, CANOPY_PROGRAM_URL_WIC. The manifest and tests use the real names.

Harness (corrected)

Plan originally referenced a ServiceClients::from_env_with injection helper that does not exist. The real integration seam is ProgramServiceRegistry::from_services(HashMap<Program, ProgramServiceConfig>) (added under plans/orchestrator-dispatch-tests.adoc). Unit tests in Step 2 build a registry containing the subset of programs under test and call orchestrator::determine directly — the harness pattern is identical to the dispatch tests.

Consequence: Step 2 tests live in tests/ (integration layer), not src/ (unit layer), to match the dispatch-test harness and avoid duplicating mock scaffolding. The plan’s "unit test" framing is preserved in intent — each test isolates one capability flag — but the physical layer is integration.

Steps 3 & 4 deferred (scope cut)

The profile-matrix integration tests and the compose-profile-matrix CI job each require tearing down the devstack and bringing it back up under a different COMPOSE_PROFILES value — roughly 5× the existing integration wall-clock. On the current GitLab runner capacity that’s a significant budget increase for behaviour already covered by the Step 2 tests (each capability proven in isolation against a live orchestrator, database, and persons fetch).

The unverified-claim risk the 2026-04-18 review flagged is resolved by Step 2: every capability flag has an explicit test that fails if the skip-missing-program path regresses. Steps 3/4 remain on the tracker as "Potential Improvements" for the CI-cost conversation and would graduate if a regression slips past the Step 2 fan-out (e.g., an interaction bug that only surfaces when a whole service is absent, not just its URL).

Recorded as pending in == Potential Improvements below.

Branch: feature/adr-005-degradation-verification
Labels: type::compliance, priority::high, program::infrastructure, service::eligibility, service::ci, workflow::ready

Context

ADR-005 guarantees that any jurisdiction deploys any subset of program services via Docker Compose profiles. A jurisdiction running snap-only must not see its eligibility orchestrator hang, crash, or 500 because canopy-medicaid is absent; it should degrade gracefully — optionally returning a 501 Not Implemented for the missing program within a combined result — and complete the SNAP determination.

The existing deployment-profiles-event-wiring.adoc plan marks Step 3 ("capability flags for optional services in canopy-eligibility") as Complete. The 2026-04-18 review flagged this as unverified:

ADR-005 graceful degradation unverified: No clear evidence of capability flag logic (CANOPY_*_URL env vars) being checked before optional service calls, or 501 responses being returned for missing services. The docker-compose profiles exist, but runtime graceful degradation is unconfirmed.

"Implemented but unverified" is the most dangerous category of claim — it reads as done to anyone skimming. This plan turns the claim into verifiable coverage.

Scope

In scope:

  • A capabilities manifest that a build tool can read.

  • Unit tests that exercise orchestrator behaviour under each individual capability being off.

  • Integration tests driven by each production Compose profile.

  • CI integration.

  • Status synchronisation in deployment-profiles-event-wiring.adoc.

Out of scope:

  • Changes to the orchestrator itself. If a test reveals a bug, file a separate plan.

  • Service-mesh / Kubernetes profile equivalents. ADR-005 targets Compose; Kubernetes parity is a post-1.0 concern.

  • Applicant portal / BFF degradation. The portal (ADR-008) is not yet implemented.

Dependencies

  • services/canopy-eligibility/src/clients.rs — capability flags live here.

  • services/canopy-eligibility/src/orchestrator.rs — conditional dispatch lives here.

  • docker-compose.yml — profile definitions.

  • crates/canopy-test-lib::TestConfig::from_env — reads capability URLs from .ports.env.

  • xtask dev start --profile <name> — already supports profile selection.

Design

Capabilities manifest

# compliance/deployment-profile-capabilities.toml

[capability.snap]
url_var           = "CANOPY_PROGRAM_URL_SNAP"
program           = "Snap"
service           = "canopy-snap"
required_profiles = ["snap-only", "full"]

[capability.tanf]
url_var           = "CANOPY_PROGRAM_URL_TANF"
program           = "Tanf"
service           = "canopy-tanf"
required_profiles = ["tanf-only", "full"]

[capability.medicaid]
url_var           = "CANOPY_PROGRAM_URL_MEDICAID"
program           = "Medicaid"
service           = "canopy-medicaid"
required_profiles = ["medicaid-chip", "full"]

[capability.chip]
url_var           = "CANOPY_PROGRAM_URL_CHIP"
program           = "Chip"
service           = "canopy-medicaid"  # CHIP is served by canopy-medicaid
required_profiles = ["medicaid-chip", "full"]

[capability.caps]
url_var           = "CANOPY_PROGRAM_URL_CAPS"
program           = "Caps"
service           = "canopy-caps"
required_profiles = ["caps-only", "full"]

[capability.wic]
url_var           = "CANOPY_PROGRAM_URL_WIC"
program           = "Wic"
service           = "canopy-wic"
required_profiles = ["wic-only", "full"]

The manifest is the source of truth for what "a capability is absent" means and which Compose profile is expected to satisfy it. Tests load it to drive the fan-out; the runtime does not (the runtime reads env vars directly via ProgramServiceRegistry::from_env).

Note: canopy-exchange is not a program capability — it is a future FFE adapter (ADR-008, stubbed) and is not reachable from the eligibility orchestrator’s program-dispatch loop. It is excluded from this manifest and tracked separately.

Capability-flag test pattern

Tests live in services/canopy-eligibility/tests/capability_flag_test.rs. Each test builds a ProgramServiceRegistry that is missing exactly one program, dispatches a determination requesting all 6 programs, and asserts the missing program lands in programs_pending with basis "program service not configured".

#[tokio::test]
async fn missing_medicaid_capability_lands_in_pending() {
    let harness = OrchestratorHarness::with_capabilities(&[
        Program::Snap, Program::Tanf, /* Medicaid omitted */
        Program::Chip, Program::Caps, Program::Wic,
    ]).await;

    let response = harness.determine(&["snap", "tanf", "medicaid", "chip", "caps", "wic"]).await;

    let medicaid_pending = response.programs_pending.iter()
        .find(|r| r.program == "medicaid")
        .expect("medicaid must be in programs_pending");
    assert_eq!(medicaid_pending.status, "pending_verification");
    assert_eq!(
        medicaid_pending.basis.as_deref(),
        Some("program service not configured"),
    );

    // And no outbound request was attempted against the absent capability —
    // proven by the mock server never being constructed for Medicaid.
}

The harness reuses the in-process axum mock pattern from plans/orchestrator-dispatch-tests.adoc, including ProgramServiceRegistry::from_services to inject only the programs under test. The absence of a mock for the omitted capability is itself the proof that no outbound call was attempted.

Integration-test matrix (deferred)

See == Errata for rationale. Sketch retained here so the next pass has a starting point:

// services/canopy-eligibility/tests/profile_matrix_test.rs

#[tokio::test]
async fn profile_snap_only_determines_snap_and_skips_others() {
    if !infrastructure_available().await { return; }
    let cfg = TestConfig::from_env();
    // Test is only meaningful if only SNAP is deployed
    let only_snap = !cfg.snap_url.is_empty()
        && cfg.tanf_url.is_empty()
        && cfg.medicaid_url.is_empty();
    if !only_snap { return; }
    let client = TestClient::new(&cfg.eligibility_url);
    let resp = client.post_json("/v1/eligibility/determine", &seed_ctx()).await;
    resp.assert_status(200);
    let body = resp.json_value();
    // Profile-shaped assertion: SNAP present in approved-or-denied, absent
    // programs present in programs_pending with the "not configured" basis.
}

CI matrix job (deferred)

See == Errata for rationale. Sketch retained for the next pass:

compose-profile-matrix:
  stage: test
  needs: []
  parallel:
    matrix:
      - PROFILE: [snap-only, tanf-only, medicaid-chip, caps-only, wic-only]
  script:
    - export COMPOSE_PROFILES=$PROFILE
    - cargo xtask dev start
    - cargo xtask test --integration --filter profile_matrix
    - cargo xtask dev stop

Steps

Step 1: Capabilities manifest

Files: compliance/deployment-profile-capabilities.toml (new).

Transcribe from docker-compose.yml and clients.rs. One row per capability.

Step 2: Capability-flag tests

Files: services/canopy-eligibility/tests/capability_flag_test.rs (new).

Six tests (one per capability — SNAP, TANF, Medicaid, CHIP, CAPS, WIC). Reuse the mock harness from orchestrator_dispatch_test.rs (landed in MR !75). A seventh test covers the "all capabilities absent" edge case where every requested program is expected to land in programs_pending — this is the true snap-only-stub equivalent in miniature.

Step 3: Integration matrix (deferred)

See == Errata § "Steps 3 & 4 deferred" for rationale. Not executing in this MR.

Step 4: CI (deferred)

See == Errata § "Steps 3 & 4 deferred" for rationale. Not executing in this MR.

Step 5: Plan sync

Files: docs/modules/ROOT/pages/plans/deployment-profiles-event-wiring.adoc, docs/modules/ROOT/pages/roadmap.adoc.

Update the Step 3 row: Complete → Complete (verified by capability-matrix tests). Remove any unresolved Tier 5.5 / 7 entry that aliased this work.

Files Touched

File Change

compliance/deployment-profile-capabilities.toml

New manifest (6 capabilities)

services/canopy-eligibility/tests/capability_flag_test.rs

7 new capability-flag tests (6 single-capability-absent + 1 all-absent)

docs/modules/ROOT/pages/plans/deployment-profiles-event-wiring.adoc

Status row reference

docs/modules/ROOT/pages/plans/adr-005-graceful-degradation-verification.adoc

Status updates, Errata, Potential Improvements

CHANGELOG.adoc

Entry under == Unreleased

Deferred:

  • services/canopy-eligibility/tests/profile_matrix_test.rs — 5 integration tests under Compose profiles

  • .gitlab-ci.yml — compose-profile-matrix job

Verification

  1. cargo nextest run -p canopy-eligibility --test capability_flag_test — 7 new tests pass

  2. Deliberately break capability-flag handling (e.g., flip the None ⇒ { pending.push(…) } arm in orchestrator::determine to continue) — the tests fail with a clear assertion that the missing program never landed in programs_pending

  3. cargo xtask test runs the capability-flag tests as part of the standard battery (no profile-dependent gating)

Deferred (Steps 3/4): per-profile devstack integration + CI matrix. Captured under == Potential Improvements.

Potential Improvements

  • Profile-matrix integration tests (Step 3): exercise the real devstack under each of snap-only, tanf-only, medicaid-chip, caps-only, wic-only. Validates that capability-flag handling interacts correctly with actual missing Compose services (not just an empty registry entry).

  • compose-profile-matrix CI job (Step 4): parallel or sequential matrix running each profile end-to-end on merge requests.

  • Manifest enforcement in CI: add cargo xtask compliance capabilities that re-parses docker-compose.yml and asserts every program service declares the profile set in deployment-profile-capabilities.toml (prevents silent drift when a new profile lands).

Tracked follow-ups (filed 2026-05-04 during PI sweep):

  • #350 — Profile-matrix integration tests + compose-profile-matrix CI job (covers Steps 3 + 4)

  • #418 — cargo xtask compliance capabilities manifest enforcement

Documentation Updates

  • Service Catalog — link to the capabilities manifest under canopy-eligibility config

  • CHANGELOG.adoc — entry under == Unreleased

Edit this page · default