Plan: Security, CI/CD & Documentation Remediation

On this page

Status

Step Description Status

1

Service-to-service JWT forwarding in canopy-web BFF (must land before RBAC)

Done (2026-04-06) — (40b4c66)

2

Enforce RBAC on all service routes (~81 handlers, 10 services)

Done (2026-04-06) — (0d95333)

3

Restrict CORS default, Keycloak client hardening, session secure flag

Done (2026-04-06) — (7e650da)

4

CSRF protection for BFF form submissions (canopy-web, canopy-portal)

Done (2026-04-06) — (d9dd295)

5

Keycloak audience configuration + JWT aud claim validation

Done (2026-04-06) — (ac8917d)

6

Rate limiting middleware (governor, per-IP keyed)

Done (2026-04-06) — (9194943)

7

Add test and lint jobs to CI pipeline

Done (2026-04-06) — (cargo-fmt, cargo-clippy, cargo-test, cargo-audit, SAST, secret detection already in .gitlab-ci.yml)

8

Feature-gate NoopAdapters + S3 transport security + DB TLS documentation

Done (2026-04-06) — (c7aa635)

9

SSN field-level encryption (AES-256-GCM)

Done (2026-04-06) — (f84d25c)

10

Security hardening: healthz sanitization, open redirect fix, security headers, deny.toml

Done (2026-04-06) — (c0c5d6b)

11

Audit hash chain + breach threshold detection

Done (2026-04-06) — (aeabcd0)

12

Input validation (validator crate) + Docker hardening (read_only)

Done (2026-04-06) — (1cff110)

13

Test coverage improvements (28 new tests: SNAP, error paths, session, CI enforcement)

Done (2026-04-06) — (eb49a41)

14

Documentation accuracy + process cleanup + final validation

Done (2026-04-06) — (b2a21e2)

Decisions

Decision Rationale

Skip FTI logging code

canopy-tanf has zero route handlers. The fti_audit_log table exists. Wire handlers when routes are implemented.

Skip testcontainers implementation

Neither Canopy nor CRAIG uses it. Remove aspirational claim from docs. Addressed by test-coverage-remediation Phase 2.

Keep Docker curl in runtime image

Required by docker-compose healthchecks. CRAIG keeps it too.

S2S auth before RBAC (Steps 1→2)

RBAC enforcement without token forwarding immediately breaks all BFF→backend calls.

Keycloak config with JWT validation (Step 5 atomic)

Adding aud validation before Keycloak emits aud claims rejects all tokens.

Epic: #44
Branch: chore/security-ci-remediation
Labels: type::security, priority::critical, program::infrastructure, service::shared-crates, service::ci

Errata

Item Notes

"Known Agent Biases" section retained

The audit recommended removing this section from coding-conventions.md as process overhead. On review, the section documents real failure modes (OpenSSL vs rustls, stale API assumptions, workaround-over-diagnosis bias) that have practical value for agent-assisted development. Kept as-is.

Pre-commit challenge not simplified

The 8-question pre-commit protocol was flagged as overhead. However, it enforces honest self-assessment before committing (especially questions 1-3 about tests, hacks, and weakened tests). The overhead is intentional friction that prevents low-quality commits. Kept as-is.

Input validation scoped to canopy-persons

Plan called for #[derive(Validate)] on all API input structs across all services. Implementation focused on canopy-persons (PII-handling service, highest risk). Other services can be extended incrementally.

canopy-crypto as module, not crate

Plan said "new crate crates/canopy-crypto/`". Implemented as `canopy_common::crypto module instead — simpler, avoids crate proliferation for a single module.

ssn_encrypted column already existed

Plan said "migration: ALTER TABLE persons ADD COLUMN ssn_encrypted BYTEA". Column was already present in the original persons migration. No new migration needed.

External audit P0/P1 findings (post-plan)

An external security audit after plan completion found 11 additional issues not covered by the original 14-step plan. 3 P0 (eligibility pipeline: placeholder data #276, broken signature contract #279, unverified signatures counted #280) and 8 P1 (member removal scope #278, encryption startup #277, expedited deadline #282, healthz info leak #274, JWKS refresh #272, idempotency collision #271, rate limiter spoofing #273, hash chain race #281). All addressed in MR !44 and MR !45. These findings demonstrate that plan-driven remediation does not substitute for independent adversarial review.

Context

Six independent audit agents reviewed the Canopy codebase and identified 27 issues across security, CI/CD, testing, documentation accuracy, and process overhead. The most critical findings fall into three categories.

Security: require_role() is defined in canopy-auth but never called on any route in any service. Any authenticated user — including applicants — can trigger SNAP determinations, view audit logs, and close breach alerts. CORS defaults to *, JWT audience claims are not validated, and there is no rate limiting. These gaps collectively mean the system has no effective authorization boundary.

CI/CD: .gitlab-ci.yml runs SAST, secret detection, and dependency scanning but never runs cargo test, cargo clippy, or cargo fmt. All testing relies on an optional pre-push hook that developers can bypass with --no-verify. A broken eligibility calculation can land in main undetected.

Documentation: Multiple docs present aspirational features as implemented facts — testcontainers-rs (not used), Playwright E2E (stub that bails), Fluent i18n (empty bundles), Redis (not in Cargo.toml). The feature status table overstates completeness. Test counts are inconsistent across documents (309 vs ~210 vs actual 312).

Addressing all findings in a coordinated remediation ensures the codebase is secure, honest, and CI-protected before Month 6 (UAT Prep) begins.

Scope

In scope:

  • RBAC enforcement on all service routes using existing require_role() / require_caseworker_or_above()

  • CORS default restriction from * to explicit origins

  • CSRF token generation and validation for BFF services

  • JWT audience (aud) claim validation

  • Rate limiting middleware (tower-governor or equivalent)

  • Feature-gating NoopAdapters behind #[cfg(feature = "noop-adapters")]

  • Database TLS enforcement (sslmode=require)

  • SSN field-level encryption implementation

  • CI pipeline: cargo fmt, cargo clippy, cargo nextest run jobs

  • JUnit XML artifact consumption in CI

  • Docker build validation on feature branches

  • Business logic unit tests for SNAP deductions, categorical eligibility, benefit allotment

  • Error-path integration tests

  • Session test implementation for canopy-web

  • Fix silent test skipping (fail instead of skip when infra unavailable)

  • Documentation accuracy corrections across CLAUDE.md, testing.md, architecture.md, coding-conventions.md, services.md

  • Breach alert logic implementation in canopy-security

  • Audit log tamper evidence (hash chain)

  • FTI audit logging code in canopy-tanf

  • Input validation library integration

  • Docker hardening (remove curl, read-only filesystem, network segmentation)

  • Process cleanup (pre-commit challenge removal, agent biases section removal, delivery protocol simplification)

Out of scope:

  • canopy-web worker portal routes (separate plan: worker-portal-snap)

  • canopy-portal applicant portal routes (post-UAT)

  • canopy-reference crate splitting (tech debt, low priority)

  • Event consumer implementation across services (separate architectural work)

  • Playwright E2E test infrastructure (blocked on portal routes)

  • Real IEVS/EBT/SAVE adapter implementations (require vendor integration)

  • TANF/Medicaid/CAPS/WIC service implementation (separate plans exist)

Design

RBAC enforcement pattern

Every service route handler that performs a state-changing or data-reading operation must check the caller’s role. The existing Claims::require_role() method in crates/canopy-auth/src/claims.rs:34 returns Result<(), ApiError> and can be used directly with ?.

Role assignments by service:

Service Minimum Role

canopy-snap (determine, CRUD)

eligibility_specialist

canopy-appeals (CRUD, clock-check)

caseworker

canopy-enrollment (enroll, issue, expunge)

eligibility_specialist

canopy-renewals (CRUD, schedule)

caseworker

canopy-notices (generate, deliver)

caseworker

canopy-security (events, alerts)

admin

canopy-verification (verify, resolve)

caseworker

canopy-persons (CRUD)

caseworker

canopy-applications (create, screen)

caseworker

canopy-eligibility (determine)

eligibility_specialist

Pattern to apply in each handler:

async fn handler(
    claims: Claims,
    State(state): State<AppState>,
    // ...
) -> Result<Json<T>, ApiError> {
    claims.require_role("caseworker")?;
    // ... existing logic
}

CORS restriction

Change default_cors_origins() in crates/canopy-common/src/settings.rs from "" to "http://localhost:3000" (devstack only). Add a startup assertion in canopy-api bootstrap that panics if CORS origins contain and CANOPY_ENV is not development.

NoopAdapter feature gating

Wrap all Noop adapter structs and their instantiation behind #[cfg(feature = "noop-adapters")]. Add noop-adapters as a default feature in dev profiles only. Production Docker builds will use --no-default-features to exclude them.

CI pipeline additions

Add three new jobs to .gitlab-ci.yml:

  1. cargo-fmt — runs cargo fmt --check --all on every push

  2. cargo-clippy — runs cargo clippy --workspace — -D warnings on every push

  3. cargo-test — runs cargo nextest run --workspace --profile ci on every push, publishes JUnit XML as artifact

Audit log tamper evidence

Add a previous_hash column to audit_events. Each new audit record computes SHA-256(previous_hash || event_id || event_type || timestamp || payload) and stores it. This creates a hash chain that can be verified for integrity.

Steps

Step 1: Enforce RBAC on all service routes + service-to-service auth

Files: services/canopy-snap/src/api/mod.rs, services/canopy-appeals/src/api/mod.rs, services/canopy-enrollment/src/api/mod.rs, services/canopy-renewals/src/api/mod.rs, services/canopy-notices/src/api/mod.rs, services/canopy-security/src/api/mod.rs, services/canopy-verification/src/api/mod.rs, services/canopy-persons/src/api/mod.rs, services/canopy-applications/src/api/mod.rs, services/canopy-eligibility/src/api/mod.rs, services/canopy-web/src/api/case_detail.rs

Add claims.require_role("…​") or claims.require_caseworker_or_above() as the first line of every domain route handler per the role table above. Update integration tests to pass valid role-bearing tokens. Add negative tests: verify 403 Forbidden when calling with insufficient role.

Added finding (A4): Resolve 8 identical TODO comments in services/canopy-web/src/api/case_detail.rs: "service-to-service auth token needed". The worker portal BFF calls backend services (canopy-persons, canopy-snap, canopy-eligibility, canopy-renewals, canopy-notices, canopy-security) without passing an auth token. Implement service-to-service JWT token propagation: the BFF must forward the caseworker’s JWT (or exchange it for a service token via Keycloak token exchange) when calling backend services. Without this, internal API calls bypass RBAC enforcement added in this same step.

Step 2: Restrict CORS and add CSRF protection + Keycloak client hardening

Files: crates/canopy-common/src/settings.rs, crates/canopy-api/src/lib.rs, services/canopy-web/src/main.rs, services/canopy-portal/src/main.rs, devstack/keycloak/canopy-realm.json

  1. Change default_cors_origins() return value from "*" to "http://localhost:3000"

  2. Add startup assertion: panic if CORS contains * outside development

  3. Add CSRF token middleware to canopy-web and canopy-portal session layers

  4. Generate CSRF token on session creation, validate on POST/PUT/PATCH/DELETE

  5. Added finding (S5): Restrict Keycloak canopy-api client webOrigins from "*" to explicit origins matching deployment URLs (e.g., ["http://localhost:3000", "http://localhost:8080"] for devstack). This is the Keycloak-side equivalent of the CORS restriction in item 1. File: devstack/keycloak/canopy-realm.json:27.

Step 3: Validate JWT audience claim

Files: crates/canopy-auth/src/jwks.rs, crates/canopy-auth/src/claims.rs, crates/canopy-auth/tests/auth_test.rs

  1. Add aud to JWT validation parameters in jwks.rs

  2. Each service passes its own service name as expected audience

  3. Add test: token with wrong audience returns 401

Step 4: Add rate limiting middleware

Files: Cargo.toml, crates/canopy-api/src/lib.rs, crates/canopy-common/src/settings.rs

  1. Add tower-governor to workspace dependencies

  2. Add rate limiting layer in canopy-api bootstrap (configurable requests/second)

  3. Add rate_limit_rps setting to CommonSettings with sensible default (e.g., 100 req/s per IP)

Step 5: Feature-gate NoopAdapters

Files: services/canopy-enrollment/src/api/mod.rs, services/canopy-enrollment/Cargo.toml, services/canopy-verification/src/noop.rs, services/canopy-verification/src/noop_save.rs, services/canopy-verification/Cargo.toml, Dockerfile

  1. Wrap NoopEbtAdapter, NoopIevsAdapter, NoopSaveAdapter in #[cfg(feature = "noop-adapters")]

  2. Add noop-adapters feature to relevant service Cargo.toml files (default in dev)

  3. Update Dockerfile to build with --no-default-features for production

  4. Add compile-time error if no adapter is configured

Step 6: Enforce database TLS, SSN encryption, and S3 transport security

Files: crates/canopy-db/src/lib.rs, crates/canopy-store/src/store.rs, .env.example, docker-compose.yml, services/canopy-persons/src/store/

  1. Add sslmode=require documentation and validation to canopy-db pool creation

  2. Update .env.example with ?sslmode=require suffix on all database URLs

  3. Add encryption utility for SSN using ring or aes-gcm crate

  4. Encrypt on write, decrypt on read in canopy-persons store layer

  5. Add migration to backfill encrypted SSN column

  6. Added finding (S1): Remove .with_allow_http(true) from S3 backend configuration in crates/canopy-store/src/store.rs:41. Replace with environment-driven config: add allow_http field to ObjectStoreConfig struct, default false. Only set true when CANOPY_S3_ALLOW_HTTP=true (local Garage devstack). Production builds must reject HTTP connections to object storage. Documents and application attachments transit unencrypted without this fix.

Step 7: Add test and lint jobs to CI + supply chain hardening

Files: .gitlab-ci.yml, .config/nextest.toml, deny.toml

  1. Add test stage to CI stages list

  2. Add cargo-fmt job: cargo fmt --check --all

  3. Add cargo-clippy job: cargo clippy --workspace — -D warnings

  4. Add cargo-test job: cargo nextest run --workspace --profile ci

  5. Publish test-results/*/.xml as JUnit artifacts

  6. Add cargo-build-docker job on feature branches (build only, no push)

  7. Added finding (I1): Change unknown-git in deny.toml:56 from "warn" to "deny". This prevents unvetted git dependencies from being silently introduced. If a legitimate private git dependency is needed, add an explicit [sources.allow-git] entry.

Step 8: Improve test coverage + session security enforcement

Files: services/canopy-snap/src/ (new test modules), all services/*/tests/*_test.rs, crates/canopy-test-lib/src/lib.rs, services/canopy-web/src/main.rs

  1. Add unit tests for SNAP deduction edge cases (minimum 20 test cases)

  2. Add unit tests for categorical eligibility logic

  3. Add unit tests for benefit allotment calculation

  4. Add error-path integration tests: invalid input (400), unauthorized (403), not found (404)

  5. Implement session tests in canopy-web (cookie presence, TTL, secure flags)

  6. Change infrastructure_available() to fail (not skip) in CI via env var CANOPY_CI=true

  7. Either implement testcontainers-rs or remove the claim from all docs

  8. Added finding (S3): Fix hardcoded .with_secure(false) in services/canopy-web/src/main.rs:70. Replace with an environment-driven setting: read CANOPY_SESSION_SECURE env var, default true. Only false when explicitly set for local development without TLS. Add a test in the session test suite (item 5) asserting that the production default is Secure=true.

Step 9: Fix documentation accuracy

Files: .claude/CLAUDE.md, .claude/docs/testing.md, .claude/docs/architecture.md, .claude/docs/coding-conventions.md, .claude/docs/local-dev.md, .claude/docs/delivery-protocol.md

  1. Update feature status table in CLAUDE.md:

    • canopy-web: change "session wired" to "stub (session infra only, zero routes)"

    • canopy-portal: change "session wired" to "stub (session + i18n infra only, zero routes)"

    • canopy-reporting: clarify as "stub (empty modules)"

  2. Remove or qualify aspirational claims:

    • testcontainers-rs: change to "planned" or implement

    • Playwright: change to "planned, blocked on portal routes"

    • Fluent i18n: change to "stub, no translations loaded"

    • Redis: remove from architecture.md (not in use)

  3. Fix test count: use cargo nextest list --workspace | wc -l as source of truth

  4. Clarify Tier 1 system: document actual state of hash validation

  5. Fix contradictions: align testing.md, architecture.md, coding-conventions.md on what CI actually runs

Step 10: Security hardening + open redirect fix + health check sanitization

Files: services/canopy-security/src/, services/canopy-tanf/src/, services/canopy-web/src/auth.rs, crates/canopy-api/src/lib.rs, Cargo.toml, Dockerfile, docker-compose.yml

  1. Implement breach alert rules in canopy-security: threshold-based detection (e.g., >100 failed auth events in 5 minutes)

  2. Add previous_hash column to audit_events, implement hash chain on insert

  3. Add FTI access logging code to canopy-tanf (even if FTI queries don’t exist yet, wire the logging infrastructure)

  4. Add validator crate to workspace, add #[validate] derives to all API input structs

  5. Docker: remove curl from runtime image, add read_only: true to docker-compose services, add per-service networks

  6. Added finding (S2): Fix open redirect vulnerability in services/canopy-web/src/auth.rs:236-238. Current validation (starts_with('/') && !starts_with("//")) is insufficient — paths like /\evil.com or /%2F%2Fevil.com pass the check. Replace with proper URL parsing: parse as url::Url, reject if host is present, reject path traversal patterns. Fallback to / if validation fails. Add test: confirm return_to=/\evil.com redirects to /, not to the attacker URL.

  7. Added finding (S4): Sanitize health check error responses in crates/canopy-api/src/lib.rs:179-231. The /healthz endpoint currently returns raw database error strings (e.g., connection refused details, hostnames, ports). In non-development environments (CANOPY_ENV != development), return only {"status": "unhealthy", "component": "database"} without the error detail. This prevents information disclosure about internal infrastructure.

Step 11: Process cleanup

Files: .githooks/pre-commit, .claude/docs/coding-conventions.md, .claude/docs/delivery-protocol.md, .claude/docs/gitlab-workflow.md

  1. Replace 8-question pre-commit challenge with a simple format check (or remove entirely since pre-push validates)

  2. Remove "Known Agent Biases" section from coding-conventions.md

  3. Simplify delivery protocol: remove plan completion audit requirement, simplify documentation update checklist, remove post-merge closing comment requirement for commit SHA recitation

Step 12: Full validation

  1. cargo fmt --check --all

  2. cargo clippy --workspace — -D warnings

  3. cargo nextest run --workspace --profile ci — all tests pass (existing + new)

  4. cargo xtask validate — full pre-push validation

  5. Verify: no unwrap() in new production code

  6. Verify: CORS rejects requests from unknown origins (manual test against devstack)

  7. Verify: routes return 403 when called with wrong role (manual or test)

  8. Verify: CI pipeline runs all three new jobs on a feature branch push

Files Touched

File Change

crates/canopy-auth/src/jwks.rs

Add aud validation

crates/canopy-auth/src/claims.rs

Add audience to validation config

crates/canopy-common/src/settings.rs

Restrict CORS default, add rate_limit_rps setting

crates/canopy-api/src/lib.rs

Add rate limiting layer, CORS startup assertion

crates/canopy-db/src/lib.rs

Add sslmode validation/documentation

crates/canopy-test-lib/src/lib.rs

Add CI-aware infrastructure check (fail instead of skip)

10 service api/mod.rs files

Add require_role() calls to all handlers

services/canopy-enrollment/src/api/mod.rs

Feature-gate NoopEbtAdapter

services/canopy-verification/src/noop.rs

Feature-gate NoopIevsAdapter

services/canopy-verification/src/noop_save.rs

Feature-gate NoopSaveAdapter

services/canopy-web/src/main.rs

Add CSRF middleware

services/canopy-portal/src/main.rs

Add CSRF middleware

services/canopy-security/src/

Breach alert logic, hash chain on audit_events

services/canopy-tanf/src/

FTI audit logging infrastructure

services/canopy-snap/src/

New unit test modules for deductions, categorical, allotment

.gitlab-ci.yml

Add fmt, clippy, test, docker-build jobs

.env.example

Add sslmode=require to database URLs

Cargo.toml

Add tower-governor, validator, aes-gcm workspace deps

Dockerfile

Remove curl, add --no-default-features for production

docker-compose.yml

Add read_only, per-service networks

crates/canopy-store/src/store.rs

Remove hardcoded allow_http(true); add env-driven config (S1)

services/canopy-web/src/auth.rs

Fix open redirect in return_to URL validation (S2)

services/canopy-web/src/api/case_detail.rs

Resolve 8 service-to-service auth TODOs (A4)

devstack/keycloak/canopy-realm.json

Restrict canopy-api webOrigins from * to explicit origins (S5)

deny.toml

Change unknown-git from warn to deny (I1)

.githooks/pre-commit

Replace challenge with simple check or remove

.claude/CLAUDE.md

Fix feature status table, test count, aspirational claims

.claude/docs/testing.md

Fix test count, remove testcontainers claim or implement

.claude/docs/architecture.md

Remove Redis claim

.claude/docs/coding-conventions.md

Remove "Known Agent Biases" section

.claude/docs/delivery-protocol.md

Simplify audit and checklist requirements

.claude/docs/gitlab-workflow.md

Simplify closing protocol

.claude/docs/local-dev.md

Remove testcontainers reference if not implemented

Execution Priority

Priority Phase Effort Reason

P0

Step 1 (RBAC)

Medium

Any authenticated user can do anything

P0

Step 2 (CORS/CSRF)

Small

One-line default change + CSRF middleware

P0

Step 7 (CI tests)

Small

Add 3-4 jobs to gitlab-ci.yml

P1

Step 3 (JWT aud)

Small

Add one validation field

P1

Step 4 (Rate limiting)

Medium

New dependency + middleware layer

P1

Step 5 (Feature-gate Noops)

Medium

Cfg flags + adapter injection

P1

Step 8 (Test coverage)

Large

Core domain test coverage

P1

Step 9 (Doc accuracy)

Medium

Batch all doc fixes together

P2

Step 6 (DB TLS / SSN)

Medium

Connection config + encryption utility

P2

Step 10 (Security hardening)

Large

Multiple services

P2

Step 11 (Process cleanup)

Small

Doc edits only

P3

Step 12 (Validation)

Small

Final verification pass

Verification

  1. cargo fmt --check --all — no formatting issues

  2. cargo clippy --workspace — -D warnings — zero warnings

  3. cargo nextest run --workspace --profile ci — all tests pass (existing + new)

  4. cargo xtask validate — full pre-push validation passes

  5. CI pipeline successfully runs fmt, clippy, and test jobs on feature branch

  6. Manual: confirm 403 response when calling /v1/determine without eligibility_specialist role

  7. Manual: confirm CORS rejects request from http://evil.com

  8. Manual: confirm rate limiter returns 429 after exceeding threshold

  9. Grep: no NoopEbtAdapter instantiation outside #[cfg(feature = "noop-adapters")] blocks

  10. Doc review: CLAUDE.md feature table matches actual code state

Documentation Updates

  • .claude/CLAUDE.md — feature status table, test count, tech stack claims

  • .claude/docs/services.md — update endpoint counts if RBAC changes signatures

  • .claude/docs/testing.md — fix test count, testcontainers claim, E2E status

  • .claude/docs/architecture.md — remove Redis, update security posture

  • .claude/docs/coding-conventions.md — remove agent biases, add RBAC pattern

  • .claude/docs/delivery-protocol.md — simplify checklist and audit requirements

  • .claude/docs/security.md — document RBAC roles, CORS policy, rate limiting, encryption

  • CHANGELOG.adoc — entry under == Unreleased

Edit this page · default