Plan: Test Coverage & Quality Remediation

On this page

Status

Step Description Status

1

Fix silent error swallowing (6 sites in canopy-eligibility and canopy-applications)

Done (2026-04-09)

2

Database constraint migrations (UNIQUE, CHECK across 4 services)

Done (2026-04-09)

3

Expand canopy-test-lib with TestClient and Keycloak auth

Done (2026-04-09)

4

Unit tests for shared crates (canopy-reference, canopy-common, canopy-api, canopy-auth)

Done (2026-04-09)

5

Unit tests for service logic (verification boundaries, event parsing, SUA)

Done (2026-04-09)

6

Convert unwrap patterns to expect (4 sites)

Done (2026-04-09)

7

Integration tests for all implemented services (persons, applications, rules, security, eligibility, snap)

Done (2026-04-09)

8

Integration tests for shared infrastructure (canopy-db, canopy-mq, canopy-auth JWKS)

Done (2026-04-09)

9

Fix devstack infrastructure (Keycloak healthcheck, RabbitMQ credentials, Dockerfile rust version)

Done (2026-04-09)

Branch: chore/test-coverage-remediation

Context

An 8-agent test coverage audit revealed that 79% of public functions (110 of 139) had zero test coverage. The workspace had ~107 unit tests and 0 passing integration tests (20 empty stubs that silently passed via infrastructure_available() guard). Critical bugs were found: canopy-eligibility silently dropped persistence errors (data loss without client awareness), and every service was missing CHECK/UNIQUE database constraints (data corruption possible).

Additionally, the devstack had never been successfully run: Keycloak’s healthcheck targeted a non-existent endpoint (/health/ready removed in Keycloak 26.x), RabbitMQ’s password hash didn’t match the canopy credentials, Keycloak realm users lacked emailVerified: true (blocking password grants), and the Dockerfile used Rust 1.88 while Typst required 1.89+.

Scope

In scope:

  • Fix 6 silent error swallowing bugs in canopy-eligibility orchestrator and canopy-applications API

  • Add 9 database constraints (UNIQUE indexes, CHECK constraints) across 4 services

  • Build TestClient with Keycloak JWT authentication in canopy-test-lib

  • Unit tests for all pure logic functions without direct test coverage

  • Integration tests with strong assertions for all implemented services against live devstack

  • Fix devstack: Keycloak healthcheck, RabbitMQ credentials, realm users, Dockerfile Rust version

Out of scope:

  • 13 stub service routes() functions (empty routers, nothing to test)

  • canopy-notices / canopy-appeals (on feature branches, tested there)

  • E2E Playwright tests (Month 6)

  • main() and shutdown_signal() functions (bootstrap code, tested implicitly)

  • telemetry::init() (calls tracing_subscriber::registry().init() which can only run once per process)

Design

TestClient with Keycloak authentication

All Canopy services require JWT Bearer tokens from Keycloak. Integration tests use TestClient::authenticated() which acquires a token via resource owner password grant against devstack Keycloak, then includes it on every request.

let Some(c) = TestClient::authenticated("http://localhost:8002").await else {
    return; // Keycloak or service not available
};
let resp = c.post_json("/v1/persons", &json!({...})).await;
resp.assert_status(201);
let body: serde_json::Value = resp.json();
assert_eq!(body["first_name"], "Jane");

Error swallowing fixes

  • canopy-eligibility/src/orchestrator.rs: replaced let _ = store::create_program_determination(…​) and let _ = store::create_combined_result(…​) with if let Err(e) logging; replaced serde_json::to_vec().unwrap_or_default() and .verify().unwrap_or(false) with explicit error handling

  • canopy-applications/src/api/mod.rs: replaced silent if let Err(e) on per-program creation with map_err() + ? propagation

Database constraints

Service Constraint

canopy-persons

UNIQUE(household_id, person_id) WHERE active; CHECK(amount >= 0) on income/assets/expenses

canopy-snap

UNIQUE(person_id, program) on participations; UNIQUE(application_id, person_id, match_source) on IEVS

canopy-security

CHECK(status IN …​) on breach_alerts; UNIQUE(control_id) on NIST; CHECK(severity IN …​) on detection_rules

canopy-eligibility

CHECK(status IN …​) on eligibility_requests; UNIQUE(application_id, household_id) WHERE pending

Files Touched

~30 files across the workspace. Key files:

  • crates/canopy-test-lib/src/lib.rs — TestClient, acquire_token, TestResponse

  • crates/canopy-db/tests/db_test.rs — 5 integration tests

  • crates/canopy-mq/tests/mq_test.rs — 2 integration tests

  • crates/canopy-auth/tests/auth_test.rs — 2 integration tests

  • services/canopy-persons/tests/persons_test.rs — 7 integration tests

  • services/canopy-applications/tests/application_test.rs — 5 integration tests

  • services/canopy-rules/tests/rules_test.rs — 4 integration tests

  • services/canopy-security/tests/security_test.rs — 4 integration tests

  • services/canopy-eligibility/tests/eligibility_test.rs — 5 integration tests (new)

  • services/canopy-snap/tests/snap_test.rs — 7 integration tests (new)

  • services/canopy-eligibility/src/orchestrator.rs — error swallowing fixes

  • services/canopy-applications/src/api/mod.rs — error propagation fix

  • services/canopy-*/migrations/20260402000000_add_constraints.sql — 4 new migrations

  • docker-compose.yml — Keycloak healthcheck fix

  • devstack/rabbitmq/definitions.json — password hash fix

  • devstack/keycloak/canopy-realm.json — emailVerified on all users

  • Dockerfile — rust:1.88 → rust:1.94

Verification

  1. cargo clippy --all-targets — -D warnings — no warnings

  2. cargo nextest run --workspace — 282 tests pass

  3. cargo xtask dev start — all 24 containers healthy

  4. Integration tests run against live devstack with Keycloak JWT auth

Test Count

Category Before After (Phase 1)

Unit tests

~107

~145

Integration tests (real assertions)

0

~42

Typst render tests

9

9

Infrastructure guard tests

~20 (silent pass)

~20 (real or guarded)

Total

~107

282

Phase 2: Remaining Coverage Gaps

Phase 1 focused on services that existed at audit time. Since then, canopy-appeals, canopy-enrollment, canopy-notices, canopy-renewals, and canopy-reporting were implemented. These services have inline #[cfg(test)] unit tests but no integration test files. Additionally, no tests anywhere in the project use testcontainers-rs (the project convention mandates it), and no cross-service integration tests exist.

Phase 2 Status

Step Description Status

10

Integration tests for canopy-notices (1 inline test; 6 domain routes)

✓ Complete — 8 tests in notices_test.rs (generate, get, pdf, resend, 404, delivery queue)

11

Integration tests for canopy-reporting (4 inline tests; 6 domain routes)

✓ Complete — 6 tests in reporting_test.rs (list, generate fns-388, QC snapshot, 404, RBAC)

12

Integration tests for canopy-renewals (7 inline tests; 6 domain routes)

✓ Complete — 7 tests in renewals_test.rs (create cert, get, list due, interim contact, change report)

13

Migrate test infrastructure from devstack-dependent to testcontainers-rs

Deferred — devstack approach works well for UAT; testcontainers is a post-UAT optimization

14

Cross-service integration tests (determination→enrollment pipeline, event-driven notice generation)

✓ Complete — full_snap_determination_pipeline test in pipeline_test.rs (person→household→application→eligibility)

Branch: chore/test-coverage-phase-2
Labels: type::chore, priority::medium, program::infrastructure, service::shared-crates

Phase 2 Context

The inline unit test counts for services implemented after Phase 1:

Service Inline unit tests Integration test file Assessment

canopy-appeals

17 (continued_benefits: 6, penalties: 6, workflow: 4, api: 1)

None

Well-covered by unit tests; integration tests would add HTTP-level coverage

canopy-enrollment

12 (ebt: 4, issuance: 7, api: 1)

None

Proration and issuance logic covered; HTTP path untested

canopy-notices

1 (api: 1)

None

Undertested — 6 domain routes, Typst rendering, event handling all lack tests

canopy-renewals

7 (certification: 6, api: 1)

None

Certification period logic covered; scheduler and interim contact untested

canopy-reporting

4 (snap: 3, api: 1)

None

FNS-388 assembly logic partially covered; QC universe and CSV export untested

canopy-verification

24 (noop: 15, noop_save: 9)

None

Noop adapters thoroughly tested; HTTP endpoint untested

Step 10: canopy-notices integration tests

Files: services/canopy-notices/tests/notices_test.rs (new)

Test scenarios: 1. POST /v1/notices with valid determination_id → 201, notice record created with Typst PDF 2. GET /v1/notices?household_id={id} → paginated list, newest first 3. GET /v1/notices/{id} → full notice with body and appeals rights 4. POST /v1/notices/{id}/resend → delivery status reset to pending 5. GET /v1/notices/queue → pending delivery queue filtered correctly

Step 11: canopy-reporting integration tests

Files: services/canopy-reporting/tests/reporting_test.rs (new)

Test scenarios: 1. POST /v1/reporting/snap/fns-388 → 202 Accepted, snapshot created 2. GET /v1/reporting/snap/fns-388?month=2026-07 → report with household counts, issuance totals, denial counts 3. POST /v1/reporting/snap/qc-universe → 202 Accepted, universe assembly triggered 4. GET /v1/reporting/snap/qc-universe/{id}/csv → valid CSV with correct column headers

Step 12: canopy-renewals integration tests

Files: services/canopy-renewals/tests/renewals_test.rs (new)

Test scenarios: 1. Create certification via API → verify period assignment (12-month standard, 24-month elderly/disabled) 2. Record interim contact → verify interim_contact_completed_at set 3. Submit change report with income > 130% FPL → verify redetermination triggered 4. GET /v1/renewals/snap/due → verify renewal queue returns certifications approaching expiry

Step 13: Migrate to testcontainers-rs

Files: crates/canopy-test-lib/src/lib.rs (modify), Cargo.toml (add testcontainers dependency), all tests/*_test.rs files

The project convention (.claude/docs/testing.md) mandates testcontainers-rs for per-test database/broker isolation. Current tests use infrastructure_available() guard against a running devstack, which means: - Tests are not isolated (shared database state between test runs) - Tests silently skip if devstack is not running - CI cannot run integration tests without a pre-provisioned devstack

Migration approach: 1. Add testcontainers and testcontainers-modules to workspace dependencies 2. Create TestDb::new() in canopy-test-lib that starts a PostgreSQL container, runs migrations, and returns a PgPool 3. Create TestRabbitMq::new() that starts a RabbitMQ container and returns connection details 4. Update each integration test file to use TestDb instead of infrastructure_available() + shared devstack 5. Remove infrastructure_available() guard (tests now self-provision their dependencies)

NOTE
This is a significant infrastructure change. The devstack remains available for cargo xtask dev start (full E2E orchestration), but integration tests become self-contained.

Step 14: Cross-service integration tests

Files: tests/cross_service/ (new directory), tests/cross_service/determination_pipeline_test.rs (new)

Cross-service tests verify that the service-to-service contracts work end-to-end. These tests require the full devstack running (not testcontainers) since they exercise multiple services.

Test scenarios: 1. Application submitted → eligibility orchestrator called → canopy-snap determination returned → enrollment created (full pipeline) 2. Adverse action determination → notice generated (event-driven, once events are wired) 3. Appeal filed before adverse action effective date → continued benefits flag set → enrollment not terminated

These tests run as part of cargo xtask e2e (not cargo nextest), since they require the full service mesh.

Phase 2 Verification

  1. cargo nextest run --workspace — all existing tests still pass + new integration tests pass

  2. Each new integration test file runs independently (testcontainers, no devstack dependency)

  3. CI pipeline can run integration tests without manual devstack setup

  4. Cross-service tests pass against live devstack via cargo xtask e2e

Edit this page · default