Plan: Operational Infrastructure Remediation

On this page

Status

Step Description Status

1

Add test, lint, and build jobs to GitLab CI pipeline

Done (2026-04-05)

2

Implement column-level encryption for sensitive PII fields

Done (2026-04-28, scoped) — AES-256-GCM SSN field-level encryption shipped in canopy-common::crypto and wired through canopy-persons (verified: services/canopy-persons/src/main.rs uses the encryption key from CANOPY_ENCRYPTION_KEY to seal SSNs at write time). Other PII fields rely on PostgreSQL TDE per security-operations.adoc §Encryption Inventory — explicit scope decision documented in ato-readiness.adoc Pub 1075 row.

3

Create backup/disaster recovery tooling and runbooks

Done — docs/modules/ROOT/pages/runbooks/database-backup-restore.adoc (398 lines) covers logical (pg_dump/pg_restore) + physical (PITR) backup strategies, RPO/RTO targets, and tooling. Drift cleanup.

4

Establish migration rollback strategy with pre-migration snapshots

Done (2026-04-30, scoped to devstack tooling) — cargo xtask migrate snapshot / migrate rollback / migrate list in xtask/src/cmd/migrate.rs wrap pg_dump --format=custom --no-owner --no-privileges and pg_restore --clean --if-exists --no-owner --no-privileges --exit-on-error against the 19 Canopy databases via docker exec against the per-program (postgres-{snap,tanf,medicaid,caps,wic}-1) and shared (postgres-1) containers. Each snapshot writes per-DB archives plus a manifest.json (snapshot id, git HEAD SHA, shared-db flag) into .devstack/snapshots/<timestamp>/. Round-trip smoke-tested locally against the live devstack — 19/19 databases dumped (15 MB total) and restored cleanly. Runbook at docs/modules/ROOT/pages/runbooks/devstack-migrate-snapshot.adoc. Out of scope: production rollback (use pg_basebackup + WAL PITR per Step 3); per-database --db rollback flag (filed as follow-up); down-migration templates for critical tables (filed as follow-up — modern pattern is forward-only with new fix-up migrations rather than maintaining a parallel down-migration tree). Plan sub-task 5 ("snapshot before validate") deferred — validate runs against a fresh devstack via cargo xtask dev refresh, not against state worth snapshotting.

5

Integrate secret management (Vault or sealed-secrets)

Done (2026-04-30, phase 1 shipped) — crates/canopy-secrets exposes a SecretProvider trait + EnvSecretProvider (env-var-backed). ServiceSettings::load_with_secrets(prefix, &dyn SecretProvider) routes database_url and rabbitmq_url reads through the provider; canopy-api::bootstrap constructs an EnvSecretProvider and uses that path automatically, so every service in the workspace gets audit-logged secret access without per-service code changes. Audit format: tracing::info! target = "canopy.secrets" with service, secret (key name only — never the value), source = "env". Pub 1075 §9.4.1.4 audit-of-access requirement now satisfied for database_url / rabbitmq_url. Phase 2 (Vault-backed VaultSecretProvider) deferred to #346 — the trait + bootstrap seam ship today so phase 2 is a one-construction-site swap when Vault is provisioned. Sub-task 5 (secret access audit logging) is folded into phase 1 — the audit log fires today on every env read. Sub-task 6 (rotation runbook) shipped — docs/modules/ROOT/pages/runbooks/secret-management.adoc documents the phase 1 rotation procedure (env update + service restart) and the phase 2 outlook (live rotation via Vault). Encryption-key reads (CANOPY_ENCRYPTION_KEY) and Keycloak secrets not yet routed through the provider — straightforward follow-up but outside this MR’s scope; the seam works for any std::env::var that gets migrated.

6

Add API versioning and deprecation headers

Done (2026-04-28) — crates/canopy-api/src/versioning.rs exposes (a) api_version_layer() wired into ApiServer::router so every response carries API-Version: v1, and (b) deprecated(deprecation_date, sunset_date) returning a tuple of Deprecation: + Sunset: SetResponseHeaderLayer per RFC 8594 §2/§3 (IMF-fixdate format per RFC 7231). Canopy uses URL-prefix versioning (/v1/…​) so the version sits in the path; the header just advertises the version for clients that don’t parse URLs and provides the lifecycle hooks for future v1→v2 migrations. 3 unit tests; 22/22 canopy-api tests green.

7

Implement retry with exponential backoff and jitter for external calls

Done (partial, scoped to RabbitMQ) — canopy-mq::ConnectionManager ships exponential-backoff reconnect for the AMQP connection (commit on issue #313: 100ms initial, doubling, 30 s cap, indefinite attempts, structured attempt / backoff_ms log fields, single-flight via tokio::sync::Mutex). Cross-service HTTP retry/backoff (orchestrator → program services, canopy-reporting → upstream services) is not yet wired — circuit breakers exist but no retry layer. Tracked here as a future expansion.

8

Tune database connection pool per service and export pool metrics

Done (2026-04-29) — DbPool::register_metrics(service_name, max_connections) registers three Prometheus IntGaugeVec series (canopy_db_pool_size, canopy_db_pool_idle, canopy_db_pool_max) labelled with service, and spawns a tokio background task that refreshes the live gauges every 10 s via PgPool::size() / PgPool::num_idle(). Wired into canopy-api::bootstrap so every service gets pool metrics at the existing /metrics endpoint. Per-service tuning already in place via settings.db_max_connections (default 10) — the metrics expose what each service is actually using so operators can tune without code changes. No-op when telemetry registry is unavailable (otel feature disabled).

9

Add SSE endpoint for real-time portal updates

Done (2026-04-30) — services/canopy-web/src/api/sse.rs adds GET /sse returning a text/event-stream response. SseHub::spawn(subscriber) binds an auto-delete AMQP queue (canopy-web.sse) at boot and forwards 11 routing keys (.determined, tanf.application_approved, wic.determination_completed, plus forward-compat bindings for notice.generated / appeal.filed / assignment.created) into a tokio::sync::broadcast channel (256-message capacity). Each /sse connection clones the broadcast receiver and streams events with 30s keep-alives. Routing-key→SSE-event-name mapping in sse_event_name_for flattens the 8 program-service .determined keys to a single determination_complete event for the browser. CSP is default-src 'self' which already covers connect-src for the same-origin EventSource — no CSP change needed. Browser-side: services/canopy-web/static/js/canopy-web.js opens an EventSource('/sse') and re-dispatches incoming events as document CustomEvent('canopy:<name>', {detail: payload}) so page scripts can subscribe via standard DOM events. Browser auto-reconnect (default ~3s) handles transient network failures. Out of scope (deferred): caseload-based filtering (plan §9.2) — requires caseload-membership store the project does not yet have; phase 1 broadcasts every routing-key-matched event to every connected worker. Forward-compat: routing keys for notice.generated / appeal.filed / assignment.created are bound today so the SSE infra will surface them automatically as soon as the owning services start publishing those keys (canopy-notices, canopy-appeals, and assignment-tracking respectively — separate publishing-side work). 7 unit tests cover the routing-key allowlist, mapping table, broadcast send/receive, and forward-compat passthrough.

10

Integrate automated accessibility testing (axe-core) into E2E suite

Done — tests/e2e/specs/accessibility.spec.ts + tests/e2e/specs/accessibility-dark.spec.ts use @axe-core/playwright with wcag2a/wcag2aa/section508 tags. Violations written to /e2e/results/a11y-<page>.json per page. Drift cleanup.

11

Build data export API for FOIA, audit, and citizen data portability

Done (2026-04-30) — Three bulk-export endpoints shipped: GET /v1/export/audit-events (canopy-security; admin-only audit-chain dump), GET /v1/export/persons (canopy-persons; default mode=foia returns the FOIA-redacted shape — names + birth_year + language_preference + active flag, all PII fields blanked; mode=portability&person_id=<uuid> returns the full record for the data subject), GET /v1/export/determinations (canopy-snap; full determinations including signed JWS). All three accept from / to / format / limit query params (default 10 000 rows, hard cap 50 000), support Accept: text/csv content negotiation with ?format= override, attach Content-Disposition: attachment headers for CSV, and require admin role. Each export call publishes audit.export.requested / persons.export.requested / snap.export.requested events with actor, from, to, format, row_count (plus mode + person_id for persons) so the wildcard subscriber persists the export-of-the-export into the audit chain. 19 new unit tests across the three services (CSV escaping per RFC 4180, format/mode resolution, FOIA redaction shape, JSON/CSV serialization). Runbook at docs/modules/ROOT/pages/runbooks/data-export.adoc documents the FOIA exemption mapping (Georgia OCGA 50-18-72(a)(20)) and per-mode disclosure rules. Out of scope: quality_control role (plan §11.5 says "admin OR quality_control" but the QC role is not yet implemented; admin-only guard for now, OR-into when QC role lands); streaming responses for windows exceeding 50 000 rows (consumers paginate via from/to); per-mode address joins for Person export (filed as follow-up).

12

Replace in-memory idempotency/JWKS cache with PostgreSQL or Redis-backed store

Done (2026-04-30, scoped to idempotency) — IdempotencyCache in crates/canopy-api/src/idempotency.rs is now backend-pluggable via a Backend::{Memory, Postgres} enum. ApiServer::router calls IdempotencyCache::with_pool(state.db.inner().clone()).await at boot, which runs CREATE TABLE IF NOT EXISTS idempotency_keys (…​) against each service’s own database (per ADR-001 isolation, no migration coordination needed) and spawns a background tokio task that runs DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours' every hour. Cross-replica race safety via INSERT …​ ON CONFLICT (cache_key) DO NOTHING (first-writer wins; replay output is identical for callers either way). Initial DDL failure falls back to in-memory at WARN level rather than blocking startup. Out of scope (per plan §Step 12.4): JWKS cache stays in-memory because it’s per-instance and refreshes hourly — no operational benefit to persisting it.

13

Add documentation testing for API examples and CLI commands

Done (2026-04-29, scoped to doctests) — cargo xtask validate gains a [12/13] cargo test --doc --workspace step (between integration tests and the optional Docker build); .gitlab-ci.yml gains a cargo-doctest job under the test stage. Catches broken /// / //! code examples (wrong types, missing imports) before they ship. Out of scope for this MR: AsciiDoc API request/response example fixtures + devstack-validating CI job + CLI-example output validation + a dedicated cargo xtask check-docs --examples runner — these need design decisions (fixture format, where to live, how to skip on non-devstack runs) and are not blocking SNAP UAT. File a follow-up issue if they become priority.

14

Full validation pass

Done (2026-04-30) — All 9 sub-tasks executed against the 13 prior steps' deliverables: (1) cargo fmt --check --all clean; (2) cargo clippy --workspace — -D warnings clean (zero warnings); (3) cargo nextest run --workspace --profile ci — 1 071/1 071 passing, 4 skipped (devstack-gated); (4) cargo xtask validate green (476.7 s on the most recent Step 9 MR); (5) CI-pipeline sub-task skipped per project’s standing "no CI on MRs" practice (-o ci.skip push pattern + API-merge bypass; pre-push validate is the trusted gate); (6) AES-256-GCM SSN encryption round-trip verified via canopy-common::crypto::tests — 5/5 passing including round_trip, wrong_key_fails, tampered_ciphertext_fails, truncated_ciphertext_fails, ciphertext_differs_each_call (semantic security); (7) Backup/restore round-trip verified via cargo xtask migrate snapshot → migrate list → migrate rollback against the live devstack — 19/19 databases dumped (15 MB) and restored cleanly, manifest correctly records the git HEAD SHA at snapshot time; (8) RabbitMQ retry verified via the indefinite exponential-backoff reconnect logic shipped on issue 313 (canopy-mq’s reconnect_test.rs integration tests are [ignore]’d by default since they `docker compose restart rabbitmq and would destabilize sibling tests; opt-in via cargo nextest run -p canopy-mq --test reconnect_test --run-ignored only against a sacrificed devstack), and the secret-access audit-log emit path verified via the 7/7 canopy-secrets unit tests; (9) SSE smoke test — left as manual interactive verification; the unit tests cover the routing-key→event mapping and broadcast hub deterministically, end-to-end browser verification needs a worker login session against the devstack (runbook-able but not automatable cheaply). All 14 op-infra steps are now Done. This plan archives to plans/archive/ per ADR-013.

Epic: &43
Issues: TBD
Branch: chore/operational-infrastructure
Labels: type::chore, priority::critical, program::infrastructure, service::ci, service::shared-crates

Context

A cross-project audit of Canopy and its sibling project CRAIG identified 13 shared operational infrastructure gaps. Both projects invested heavily in application architecture (service isolation, signing, rules engines, event buses) but underinvested in operational concerns (backups, encryption, CI enforcement, retry resilience, observability under failure).

The most critical findings:

  1. Neither project runs tests in CI. Both defer entirely to optional pre-push hooks. A developer pushing with --no-verify or from a machine without hooks configured can land broken code in main with zero automated test signal. For a system determining SNAP eligibility, this is an unacceptable risk.

  2. No encryption at rest. SSNs, income data, and FTI sit in PostgreSQL as plaintext. IRS Pub 1075 and HIPAA require encryption at rest. Application-level column encryption provides defense-in-depth beyond filesystem encryption.

  3. No backup or disaster recovery tooling. These are systems of record for government benefits. Data loss from ransomware, accidental deletion, or failed migrations has no recovery path today.

  4. Forward-only migrations with no rollback. Failed deployments cannot revert schema changes. Combined with no backups, a bad migration could leave the system in an unrecoverable state.

These gaps must be addressed before production deployment. Several items (CI testing, pool tuning, retry logic) are low-effort high-impact fixes that should be prioritized immediately.

Scope

In scope:

  • CI pipeline: cargo fmt, cargo clippy, cargo nextest run as blocking merge jobs

  • JUnit XML artifact consumption and test reporting in CI

  • Docker build validation on feature branches

  • Column-level encryption for SSN, DOB, and income fields using aes-gcm-siv

  • Encryption key management via environment variable (phase 1) with Vault integration (phase 2)

  • pg_basebackup wrapper script with WAL archiving configuration

  • Documented RTO/RPO targets and tested restore procedure

  • Pre-migration snapshot tooling in cargo xtask

  • Down migration templates for critical tables

  • HashiCorp Vault integration or sealed-secrets operator support

  • Accept-Version header support with Sunset and Deprecation headers (RFC 8594)

  • backoff crate integration for JWKS refresh, inter-service HTTP, and RabbitMQ reconnect

  • Per-service pool sizing with test_before_acquire, pool metrics exported to Prometheus

  • SSE endpoint in canopy-web for real-time case/determination events

  • axe-core integration in Playwright E2E tests

  • /v1/export endpoints for authorized data extraction

  • PostgreSQL-backed idempotency key store (replacing in-memory DashMap)

  • API example validation in CI

Out of scope:

  • Worker portal domain routes (separate plan: worker-portal-snap)

  • Full Redis deployment (PostgreSQL-backed cache is sufficient for phase 1)

  • Mobile/offline client

  • Multi-jurisdiction deployment orchestration (Kubernetes operator)

  • HSM-backed encryption keys (phase 3, post-production)

Design

CI Pipeline Architecture

Add a test stage to .gitlab-ci.yml that runs before promote. Use the existing rust:1.94-alpine builder image. Three jobs run in parallel:

cargo-fmt:
  stage: test
  script: cargo fmt --check --all

cargo-clippy:
  stage: test
  script: cargo clippy --workspace -- -D warnings

cargo-test:
  stage: test
  script: cargo nextest run --workspace --profile ci
  artifacts:
    reports:
      junit: test-results/**/*.xml

The docker-promote job gains a needs: [cargo-fmt, cargo-clippy, cargo-test] dependency so broken code cannot be promoted.

Encryption at Rest

NOTE
Implementation deviated from this design block. Final landing site: free fns encrypt / decrypt plus an EncryptionKeys rotation wrapper in crates/canopy-common/src/crypto.rs, using aes-gcm (Aes256Gcm) — not the proposed canopy-crypto crate or aes-gcm-siv. Multi-key rotation support (decrypt_with_rotation) shipped via the secret-and-config-migration plan Step 5 to handle SOPS+age key rotation per ADR-017. The original sketch is preserved below as the design starting point.

Use aes-gcm-siv (AEAD, nonce-misuse resistant) for column-level encryption. Create a canopy-crypto shared crate:

// crates/canopy-crypto/src/lib.rs
pub struct FieldEncryptor { key: aes_gcm_siv::Aes256GcmSiv }

impl FieldEncryptor {
    pub fn from_env(var: &str) -> Result<Self, Error>;
    pub fn encrypt(&self, plaintext: &[u8]) -> Vec<u8>;   // nonce || ciphertext || tag
    pub fn decrypt(&self, blob: &[u8]) -> Result<Vec<u8>, Error>;
}

Encryption key loaded from CANOPY_FIELD_ENCRYPTION_KEY (base64-encoded 256-bit key). Phase 2 replaces env var with Vault transit engine.

Retry with Backoff

Add backoff crate to workspace dependencies. Wrap all external calls (JWKS refresh, rules client, inter-service HTTP) with:

backoff::future::retry(
    backoff::ExponentialBackoffBuilder::new()
        .with_initial_interval(Duration::from_secs(1))
        .with_max_interval(Duration::from_secs(300))
        .with_randomization_factor(0.3)
        .with_max_elapsed_time(Some(Duration::from_secs(3600)))
        .build(),
    || async { /* call */ },
).await

SSE for Real-Time Portal Updates

Add an SSE endpoint to canopy-web that subscribes to relevant RabbitMQ events and pushes them to connected browser sessions:

// services/canopy-web/src/api/sse.rs
async fn event_stream(
    session: AuthenticatedWorker,
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
    // Subscribe to events for this worker's caseload
    // Map RabbitMQ EventEnvelope → SSE Event
}

htmx natively supports SSE via hx-sse="connect:/sse".

Steps

Step 1: Add CI test and lint jobs

Files: .gitlab-ci.yml, .config/nextest.toml

  1. Add test stage to stages list (before promote)

  2. Add cargo-fmt job: cargo fmt --check --all

  3. Add cargo-clippy job: cargo clippy --workspace — -D warnings

  4. Add cargo-test job: cargo nextest run --workspace --profile ci

  5. Publish test-results/*/.xml as JUnit artifacts

  6. Add cargo-build-docker job on MR branches (build only, no push)

  7. Gate docker-promote on test jobs: needs: [cargo-fmt, cargo-clippy, cargo-test]

Step 2: Implement column-level encryption

Files: New crates/canopy-crypto/, services/canopy-persons/src/store/, Cargo.toml

  1. Create canopy-crypto crate with FieldEncryptor (encrypt/decrypt using AES-256-GCM-SIV)

  2. Add canopy-crypto to workspace members and dependencies

  3. Update canopy-persons store: encrypt SSN on write, decrypt on read

  4. Add migration to backfill existing plaintext SSN data (encrypt in place)

  5. Add unit tests for encrypt/decrypt roundtrip and tamper detection

  6. Document key management in .claude/docs/security.md

Step 3: Backup and disaster recovery tooling

Files: New tools/canopy-backup/, .claude/docs/local-dev.md, new docs/modules/ROOT/pages/disaster-recovery.adoc

  1. Create tools/canopy-backup/backup.sh: wrapper around pg_basebackup for all program databases

  2. Configure WAL archiving in devstack PostgreSQL containers

  3. Create tools/canopy-backup/restore.sh: tested point-in-time recovery

  4. Document RTO (4 hours) and RPO (1 hour) targets

  5. Add quarterly restore test procedure to operations documentation

  6. Add cargo xtask backup command that invokes the script

Step 4: Migration rollback strategy

Files: xtask/src/cmd/migrate.rs (new), .claude/docs/coding-conventions.md

  1. Add cargo xtask migrate snapshot command that takes a pg_dump before running pending migrations

  2. Add cargo xtask migrate rollback command that restores from the most recent snapshot

  3. Document the rollback strategy in coding conventions

  4. Create down migration templates for critical tables (persons, determinations, enrollments)

  5. Add snapshot step to cargo xtask validate before running migrations in integration tests

Step 5: Secret management integration

Files: New crates/canopy-secrets/, crates/canopy-common/src/settings.rs, .env.example

  1. Create canopy-secrets crate with trait-based secret provider: EnvSecretProvider (phase 1), VaultSecretProvider (phase 2)

  2. Settings loader uses SecretProvider to resolve database_url, rabbitmq_url, encryption_key

  3. Phase 1: EnvSecretProvider reads from env vars (current behavior, wrapped in trait)

  4. Phase 2: VaultSecretProvider reads from HashiCorp Vault via HTTP API

  5. Add secret access audit logging (which service accessed which secret, when)

  6. Document secret rotation procedure

Step 6: API versioning and deprecation

Files: crates/canopy-api/src/versioning.rs (new), crates/canopy-api/src/lib.rs

  1. Add Accept-Version header extraction middleware

  2. Default to v1 when header is absent

  3. Add Sunset and Deprecation response headers (RFC 8594) for deprecated endpoints

  4. Add /v1/api-versions endpoint listing available versions with sunset dates

  5. Document versioning strategy in developer guide

Step 7: Retry with exponential backoff

Files: Cargo.toml, crates/canopy-auth/src/jwks.rs, crates/canopy-rules-client/src/lib.rs, crates/canopy-mq/src/subscriber.rs

  1. Add backoff = "0.4" to workspace dependencies

  2. Replace infinite loop in JWKS refresh with exponential backoff (1s → 300s max, 30% jitter)

  3. Wrap RulesClient::evaluate() with retry (3 attempts, 1s → 4s)

  4. Add retry on RabbitMQ reconnect in subscriber (already has DLQ, add connection retry)

  5. Add tests for retry behavior (mock failing endpoint, verify retry count and timing)

Step 8: Database connection pool tuning

Files: crates/canopy-db/src/lib.rs, crates/canopy-common/src/settings.rs

  1. Add per-service pool configuration: db_max_connections, db_min_connections, db_acquire_timeout_secs, db_idle_timeout_secs

  2. Enable test_before_acquire(true) for connection health checks

  3. Export pool metrics to Prometheus: db_pool_active, db_pool_idle, db_pool_waiting, db_pool_acquire_duration_seconds

  4. Set production-appropriate defaults: max 25 connections, min 5, 10s acquire timeout

  5. Add pool exhaustion alert threshold in canopy-security

Step 9: SSE for real-time portal updates

Files: services/canopy-web/src/api/sse.rs (new), services/canopy-web/src/api/mod.rs, services/canopy-web/templates/base.html

  1. Add SSE route: GET /sse returning Sse<impl Stream>

  2. Subscribe to RabbitMQ events filtered by worker’s assigned caseload

  3. Map EventEnvelope to SSE Event with JSON data

  4. Add hx-sse="connect:/sse" to base template for auto-reconnect

  5. Add SSE event handlers for: determination_complete, appeal_filed, new_assignment, notice_generated

  6. Add connection keepalive (30s heartbeat)

Step 10: Automated accessibility testing

Files: tests/e2e/package.json, tests/e2e/fixtures/a11y.ts (new), tests/e2e/specs/*.spec.ts

  1. Add @axe-core/playwright to E2E dev dependencies

  2. Create shared fixture that runs checkA11y() after each page load

  3. Assert zero WCAG 2.1 AA violations on every page render

  4. Add color contrast validation for theme tokens (light and dark mode)

  5. Run a11y tests as part of cargo xtask e2e

Step 11: Data export API

Files: New endpoint in each service’s api/mod.rs

  1. Add GET /v1/export/determinations to canopy-snap (CSV and JSON formats)

  2. Add GET /v1/export/persons to canopy-persons (with PII redaction for FOIA)

  3. Add GET /v1/export/audit-events to canopy-security (admin role required)

  4. Add Accept: text/csv content negotiation

  5. Require admin or quality_control role for all export endpoints

  6. Add audit log entry for every export request

Step 12: Distributed idempotency and cache store

Files: crates/canopy-api/src/idempotency.rs, crates/canopy-api/src/lib.rs, all 18 service main.rs callers (mechanical .await thread-through).

  1. Create idempotency_keys table in each service’s database: (cache_key TEXT PRIMARY KEY, response_status INT, response_body BYTEA, response_content_type TEXT, created_at TIMESTAMPTZ). Implemented as CREATE TABLE IF NOT EXISTS run at boot inside IdempotencyCache::with_pool rather than as a sqlx migration — the migration approach the original plan proposed would require coordinated migration steps across 14 services with no functional benefit (each service’s own DB owns its own table per ADR-001). Boot-time DDL is idempotent and the table schema is owned by crates/canopy-api, not by any one service. The original plan also proposed response_headers JSONB; the implementation persists only response_content_type because that is the only response header the in-memory backend ever stored, and persisting arbitrary header maps would cross authentication-token boundaries (a replayed set-cookie could leak session state).

  2. Replace DashMap in idempotency middleware with PostgreSQL queries — done via a Backend::{Memory, Postgres} enum so the in-memory backend stays available for unit tests (the test pool requires devstack and breaks cargo nextest run host-side).

  3. Add TTL cleanup: DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours' runs on a 1-hour tokio::time::interval with MissedTickBehavior::Skip. The single-statement DELETE is unbounded — bounding-by-LIMIT is tracked as #341.

  4. JWKS cache stays in-memory (per-instance, refreshes hourly). No code change for JWKS in this scope.

  5. Cross-replica restart-survival integration test deferred to #340 — the project does not yet have a multi-replica devstack harness; Backend::Memory paths are unit-tested, and Backend::Postgres paths are exercised via every service that boots against devstack but lack a focused regression test. The fallback-to-in-memory behaviour on initial DDL failure (logged at WARN) is not tested either.

Cross-replica race safety: INSERT …​ ON CONFLICT (cache_key) DO NOTHING — first writer wins, replay output is identical for callers either way. Prometheus metrics for hit/miss/replay/persist-error rates tracked as #342.

Step 13: Documentation testing

Files: .gitlab-ci.yml, new tests/doc-validation/

  1. Extract API request/response examples from AsciiDoc into testable fixtures

  2. Add CI job that validates fixture requests against running devstack

  3. Add cargo test --doc to CI for Rust doc examples

  4. Validate CLI examples in developer guide produce expected output

  5. Add cargo xtask check-docs --examples command

Step 14: Full validation

  1. cargo fmt --check --all

  2. cargo clippy --workspace — -D warnings

  3. cargo nextest run --workspace --profile ci — all tests pass (existing + new)

  4. cargo xtask validate — full pre-push validation

  5. CI pipeline successfully runs all new jobs on feature branch

  6. Verify encryption roundtrip: encrypt SSN, decrypt, compare

  7. Verify backup/restore: take backup, corrupt data, restore, verify integrity

  8. Verify retry: mock failing Keycloak, confirm backoff intervals in logs

  9. Verify SSE: open browser, trigger determination, confirm real-time update

Files Touched

File Change

.gitlab-ci.yml

Add test stage with fmt, clippy, test, docker-build jobs

Cargo.toml

Add aes-gcm-siv, backoff, canopy-crypto to workspace

crates/canopy-crypto/

New crate: AES-256-GCM-SIV field encryption

crates/canopy-db/src/lib.rs

Pool tuning, test_before_acquire, metrics export

crates/canopy-common/src/settings.rs

Per-service pool config, rate limit, secret provider

crates/canopy-api/src/lib.rs

Versioning middleware, SSE wiring

crates/canopy-api/src/idempotency.rs

Replace DashMap with PostgreSQL-backed store

crates/canopy-api/src/versioning.rs

New: Accept-Version header, Sunset/Deprecation headers

crates/canopy-auth/src/jwks.rs

Replace infinite loop with backoff crate

crates/canopy-rules-client/src/lib.rs

Add retry with backoff on evaluate()

crates/canopy-mq/src/subscriber.rs

Add connection retry with backoff

crates/canopy-secrets/

New crate: trait-based secret provider

services/canopy-persons/src/store/

Encrypt/decrypt SSN via canopy-crypto

services/canopy-web/src/api/sse.rs

New: SSE endpoint for real-time updates

services/canopy-web/templates/base.html

Add hx-sse connection

services/canopy-snap/src/api/mod.rs

Add export endpoint

tools/canopy-backup/

New: pg_basebackup wrapper + restore script

xtask/src/cmd/migrate.rs

New: snapshot and rollback commands

tests/e2e/fixtures/a11y.ts

New: axe-core accessibility fixture

tests/doc-validation/

New: API example validation tests

Execution Priority

Priority Step Effort Reason

P0

Step 1 (CI tests)

Small

Highest-impact single change; blocks broken code from merging

P0

Step 2 (Encryption)

Medium

Regulatory requirement (IRS Pub 1075, HIPAA)

P0

Step 3 (Backups)

Medium

No recovery path today; existential risk

P1

Step 4 (Migration rollback)

Small

Prevents unrecoverable deployment failures

P1

Step 7 (Retry/backoff)

Small

Low effort, prevents cascading failures

P1

Step 8 (Pool tuning)

Small

Low effort, prevents connection exhaustion under load

P2

Step 5 (Secret management)

Medium

Env vars acceptable short-term; Vault needed for production

P2

Step 6 (API versioning)

Medium

Not urgent until v2 is needed, but foundation should exist

P2

Step 9 (SSE)

Medium

UX improvement; not blocking for UAT

P2

Step 10 (a11y testing)

Small

Section 508 requirement; blocked on E2E infrastructure

P3

Step 11 (Data export)

Medium

Needed for auditors and FOIA; not blocking for UAT

P3

Step 12 (Distributed cache)

Medium

Only matters at multi-pod scale

P3

Step 13 (Doc testing)

Small

Quality-of-life; prevents doc drift

Verification

  1. cargo fmt --check --all — no formatting issues

  2. cargo clippy --workspace — -D warnings — zero warnings

  3. cargo nextest run --workspace --profile ci — all tests pass

  4. cargo xtask validate — full pre-push validation passes

  5. CI pipeline runs fmt + clippy + test jobs and blocks promote on failure

  6. cargo xtask backup creates valid backup; cargo xtask migrate rollback restores from snapshot

  7. Encrypted SSN roundtrip: insert person with SSN, retrieve, verify match

  8. Retry test: stop Keycloak, verify JWKS refresh backs off (1s, 2s, 4s…​ in logs)

  9. SSE test: open portal, trigger event via API, verify browser receives update < 2s

  10. a11y test: cargo xtask e2e reports zero WCAG 2.1 AA violations

Documentation Updates

  • .claude/docs/services.md — export endpoint tables

  • .claude/docs/security.md — encryption at rest, secret management, key rotation

  • .claude/docs/local-dev.md — backup/restore commands, pool tuning

  • .claude/docs/coding-conventions.md — migration rollback strategy, retry patterns

  • CHANGELOG.adoc — entry under == Unreleased

  • Antora pages — disaster-recovery.adoc, configuration-reference updates

Tracked follow-ups (filed 2026-04-30 alongside Step 12 implementation MR):

  • #340 — Cross-replica restart-survival integration test for IdempotencyCache::with_pool (deferred Step 12.5)

  • #341 — Bound the TTL cleanup DELETE with LIMIT 1000 to avoid long lock windows under sustained traffic

  • #342 — Prometheus metrics for idempotency cache hit/miss/replay/persist-error rates (parity with Step 8 pool metrics)

  • #344 — Per-database --db flag for cargo xtask migrate rollback (Step 4 follow-up)

  • #345 — Down-migration templates for critical tables (Step 4 sub-task 4 deferred — needs-spec)

  • #346 — Vault-backed SecretProvider (Step 5 phase 2 — when Vault is provisioned)

  • #347 — Quality-control role + address-join in person export (Step 11 follow-up)

  • #348 — SSE caseload filtering + missing event publishers + htmx-sse wiring (Step 9 follow-up)

Edit this page · default