Security Operations & Runbooks

On this page
Contents

Incident Classification

Severity Definition Response Target Examples

P0 — Critical

Active data breach, system compromise, service outage affecting all users

15 minutes (acknowledge), 4 hours (contain)

FTI data exposure, authentication bypass, complete service failure, ransomware

P1 — High

Service degradation affecting multiple users, security vulnerability actively exploited

1 hour (acknowledge), 24 hours (resolve)

Database connection exhaustion, partial service failure, privilege escalation attempt

P2 — Medium

Non-critical security finding, single-user impact

24 hours (acknowledge), 7 business days (resolve)

Dependency CVE (high severity), individual auth failure, non-critical data exposure

P3 — Low

Informational finding, best practice deviation

Next sprint

Dependency CVE (medium/low), configuration improvement, documentation gap

Incident Response Procedure

1. DETECT

  • Automated: canopy-security breach detection rules fire alerts (checked every 60 seconds)

  • Monitoring: Health check failures, error rate spikes, unusual traffic patterns

  • User reports: Caseworker reports unexpected behavior

  • Audit: Review canopy-security audit events API (GET /v1/security/events)

2. CLASSIFY

Assign severity (P0–P3) based on the classification table above. Consider: * Is restricted federal data (FTI, IEVS, HIPAA PHI) at risk? * Is the system available to caseworkers? * How many users/households are affected?

3. CONTAIN

  • Isolate affected service: Scale to 0 replicas or block inbound traffic

  • Rotate compromised credentials: See Key Rotation Runbook below

  • Preserve evidence: Do NOT delete logs. Archive audit events via POST /v1/security/archive

  • Notify on-call team: Per agency escalation procedures

4. ERADICATE

  • Identify root cause (vulnerability, misconfiguration, insider threat)

  • Patch the vulnerability or fix the configuration

  • Deploy fix via standard deployment pipeline (build → test → staging → production)

5. RECOVER

  • Restore service (scale replicas back up, unblock traffic)

  • Verify data integrity via hash chain — POST /v1/security/chain/verify (family-full) + poll the job, then confirm GET /v1/security/chain/status (#1205; GET /v1/security/verify-chain is deleted). The FTI family tasks exist too (#1206 MR-3): family=fti&service=canopy-{tanf,medicaid} runs one verifier task per configured program DB, and degradation is PER FAMILY — one program DB outage degrades that family’s status only, never the audit family or the process. DORMANCY CAVEAT: until the #1279 cutover the verifiers are off — status reports unknown → 503 by design (a latched legacy v1 FTI breach still surfaces as breached/legacy_breach_latched) and verify returns 503 verifier_unavailable, so pre-cutover integrity verification during recovery remains a manual DBA procedure (bounded ad-hoc re-hash of the affected window)

  • Verify determination signatures are intact (canopy-eligibility JWS verification)

  • Monitor for recurrence (24-hour watch period)

6. NOTIFY

  • Federal partners per Breach Notification Chain (below)

  • Affected individuals per state law requirements

  • Agency leadership and legal counsel

  • Document incident in post-incident review

Breach Notification Chain

Data Type Federal Authority Notification Window Contact

FTI (Federal Tax Information)

IRS Office of Safeguards

24 hours

safeguards@irs.gov; (202) 803-9449

IEVS / SNAP Data

USDA FNS Regional Office (Southeast)

48 hours

FNS Southeast Regional Office, Atlanta

HIPAA PHI (Medicaid/CHIP)

HHS Office for Civil Rights

60 days (individuals); 60 days (HHS if >500 affected)

ocrportal.hhs.gov/ocr/breach

General PII

State Attorney General

Per state breach notification law (Georgia: without unreasonable delay)

Georgia AG Consumer Protection Division

Key Rotation Runbook

ECDSA Signing Keys (per program service)

Program services sign determinations with ECDSA P-256 keys.

NOTE
T2-6 / ADR-036 changed key rotation fundamentally. There is no more CANOPY_VERIFY_KEY_{PROGRAM}_PREV slot and no 30-day grace window. Each program service derives its signing kid from its public key and self-registers that public key into canopy-security’s signing_key_history on boot; the orchestrator’s verifier lazy-loads any since-retired key from there (via the per-program JWKS) on a cache-miss. So a determination signed with a rotated-out key stays verifiable indefinitely, with no operator-managed previous-key window — you rotate by deploying a new key, and the old key remains in the retention store forever.

Routine rotation (quarterly recommended):

  1. Generate a new key pair:

    cargo xtask gen-signing-keys --program snap

    Produces .keys/snap-private.pem (PKCS#8) and .keys/snap-public.pem (SPKI).

  2. Set the new keys (the public key is the new current verifier key; the private key is the new signer):

    CANOPY_VERIFY_KEY_SNAP = <new public key PEM>
CANOPY_SNAP__SIGNING_KEY = <new private key PEM>

  3. Rolling restart canopy-snap (and canopy-eligibility to pick up the new current verifier key). On boot canopy-snap registers the new public key into signing_key_history under its key-derived kid. New determinations are signed with the new key.

  4. No PREV step, no grace-period cleanup. Determinations signed with the old key continue to verify: the verifier misses the new in-memory key, then lazy-loads the old key from signing_key_history (registered while it was active) and verifies. The old key is retained permanently — appeals/QC can run for years.

Emergency rotation (compromised key):

  1. Immediately set CANOPY_SNAP__SIGNING_KEY + CANOPY_VERIFY_KEY_SNAP to a new key pair and restart all services.

  2. Audit all determinations signed during the compromise window.

  3. To distrust the compromised key (so its signatures stop verifying), tombstone its signing_key_history row in the blessed maintenance window AND ensure it is not the current CANOPY_VERIFY_KEY_SNAP — a dedicated revocation surface is a tracked follow-up (the store currently retains, never revokes).

Registrant attribution + the standing audit (#1261, post-#1259): every registration records registrant_service_id — the authenticated service identity the #1259 authorization gate ran against (NULL = pre-attribution row, never backfilled: attribution is evidence). The battery-run standing audit (signing_key_history_every_row_is_key_derived_and_attributable, canopy-security integration tests) re-applies the production gate to every attributed row and content-verifies pre-attribution rows (known program, valid SPKI key, key-derived kid) in every environment it touches; a violation fails the battery and the row must be purged through the guarded canopy.signing_key_maintenance window — never normalized.

Portal Applicant-Claim Key (#1442 / #1569)

The portal signs per-request applicant ownership claims with a dedicated P-256 key; every claim-verifying origin (canopy-applications, canopy-eligibility, canopy-enrollment, canopy-persons, …) loads the public key at boot via applicant_verifier_from_env. Since #1569 the verifier accepts multiple concatenated PEM blocks in the env value, so rotation is a zero-downtime overlap, never a hard cut of live citizen sessions:

  1. Generate the incoming pair (cargo xtask keygen conventions); do NOT deploy the private key yet.

  2. Append the incoming PUBLIC pem to every origin’s CANOPY_<SERVICE>__APPLICANT_VERIFYING_KEY value (concatenate the two exports byte-for-byte — each block’s kid is derived over the exact export bytes) and roll the origins. Dev boxes stage the incoming key as .keys/portal-applicant-public-next.pem instead.

  3. Once every origin verifies both keys, swap the portal’s PRIVATE key to the incoming one and roll the portal. In-flight claims signed by the old key stay verifiable.

  4. After the longest claim TTL has elapsed, remove the retired public block from the origins' env values and roll again.

Emergency (portal key compromise): skip the overlap — replace both halves immediately and accept the hard cut; every live citizen session re-auths through lookup.

Keycloak JWKS Keys

Keycloak manages its own key rotation internally. The JwksProvider in canopy-auth auto-refreshes the JWKS cache hourly and force-refreshes on unknown kid (with 30-second debounce).

Emergency (Keycloak key compromise):

  1. Log into Keycloak admin console

  2. Navigate to Realm Settings → Keys → Providers

  3. Add a new RSA key provider (higher priority than current)

  4. Keycloak starts signing with new key immediately

  5. canopy-auth auto-detects the new kid within 1 hour (or immediately on next unknown-kid JWT)

  6. Disable the compromised key provider after grace period

AES-256-GCM Encryption Key

Used for SSN field-level encryption in canopy-persons.

WARNING
Rotating this key requires re-encrypting ALL encrypted fields in the database. This is a data migration, not just a config change.

  1. Generate new key: openssl rand -base64 32

  2. Write a database migration that:

    1. Reads each encrypted field with the old key

    2. Re-encrypts with the new key

    3. Updates the row

  3. Set CANOPY_ENCRYPTION_KEY to the new key

  4. Run the migration (service must be restarted after)

NOTE
Re-encryption migration tooling is not yet built. This is tracked as a future enhancement.

chain-v2 Genesis & Migration-Job Runbook (#1246)

The chain-v2 substrate (ADR-014 Amendment 6; the MR-2 migrations) lands DORMANT. Two operator procedures exist ahead of the #1279 cutover:

Empty-genesis install (cargo xtask chain-genesis)

# Devstack (the TLS gate fails closed unless development is declared):
CANOPY_ENV=development cargo xtask chain-genesis --service canopy-tanf
# Production: URLs via env, never argv.
CANOPY_CHAIN_GENESIS__TARGET_URL=…  CANOPY_CHAIN_GENESIS__ANCHOR_URL=… \
  cargo xtask chain-genesis --service canopy-tanf [--shard-count N]

  • Two-phase and crash-resumable: Phase A installs the registry on the target database in one transaction (instance, epoch 0 in installing, topology, source binding, pre-created heads); Phase B records the genesis anchor via chain_anchor_append on canopy_security (idempotent CAS); Phase C re-fetches everything, independently REBUILDS the manifest and byte-compares it, then validates the full genesis shape. Reruns: crash-after-A resumes at B (after proving the heads untouched); a completed install re-validates and reports "already installed"; any supplied parameter differing from installed state is a hard error.

  • The whole run holds a per-(database, family) advisory lock — concurrent installers refuse.

  • Credentials: the TARGET URL needs a migration/owner-capable principal (genesis writes owner-role-owned tables); the ANCHOR URL needs EXECUTE on chain_anchor_append. The command hard-compares the anchor database name to canopy_security independent of CANOPY_ENV (the ADR-001 guard alone is warn-only in development).

  • Wiped-target caveat: if a target database is recreated while canopy_security retains the family’s prior anchors, a fresh install mints a NEW instance — the prior instance’s anchors remain recorded (instance history is immutable by design; anchor sequences are per-instance). The orphaned prior chain is evidence, not garbage: never delete it.

  • The chain is NOT appendable after install: epoch 0 stays installing until the cutover operator runs chain_epoch_activate (under the migration/owner credential — EXECUTE is granted to no runtime or verify role). chain-genesis is never legitimate against an activated chain.

Deploy-time migration job (cargo xtask migrate apply)

After the #1279 cutover the three chain services run with CANOPY_<SVC>SKIP_MIGRATIONS=true: bootstrap applies nothing and the runtime environment holds no migration-capable credential (a closed pool does not un-know a credential — the job model removes it entirely). Migrations then run as a deploy step: cargo xtask migrate apply --service canopy-tanf (URL via CANOPY_MIGRATEDATABASE_URL), or the devstack chain-migration-split compose one-shots. Until cutover, NOTHING enables this — bootstrap migrates as today, now over a dedicated short-lived connection that closes before the app pool exists.

Migration-credential requirements + the rotation wrinkle: the production migration credential needs CREATEROLE and schema CREATE (create-then- transfer ownership). A ROTATED credential lacks ADMIN OPTION on roles the old one created — the substrate migrations' reconcile step surfaces this loudly (refusing, not half-applying); re-grant the chain owner roles to the new migrator before re-running.

Reporting Credential-Cutover Runbook (#1456, ADR-004 A8b)

canopy-reporting’s restricted holdings are owned by canopy_reporting_owner (NOLOGIN) and the runtime connects as the restricted canopy_reporting_app login; migrations 20261111000000/20261111000001 create the roles (DO-guarded), transfer ownership (catalog-driven ALTER … OWNER loop — REASSIGN OWNED cannot be used: the historical devstack owner is the pinned bootstrap superuser, which Postgres refuses to reassign from), apply the per-object grant matrix, and move the generation janitor’s deletes behind the SECURITY DEFINER reporting_janitor_reap fn. Canonical design: the A8b child plan.

Cutover (per environment)

  1. Provision the login out of source — canopy_reporting_app must be LOGIN with a secret-managed password BEFORE the runtime URL flips. Devstack: devstack/postgres/init.sql does this on a FRESH volume only — an existing devstack adopts via the documented destructive reset (cargo xtask dev clean --confirm && cargo xtask dev start). Production: create the role manually (CREATE ROLE canopy_reporting_app LOGIN PASSWORD '<vault>') or pre-create NOLOGIN and flip.

  2. Split the credentials: set CANOPY_REPORTINGMIGRATION_DATABASE_URL to the owner-capable migrator URL and point CANOPY_REPORTINGDATABASE_URL at the canopy_reporting_app login. Bootstrap migrates on the former over a dedicated pool (closed before the app pool exists) and serves on the latter.

  3. One-time transfer identity: the cutover migration’s ownership loop runs ALTER … OWNER, which requires the executor to OWN the objects (or be a superuser) and hold SET-membership in canopy_reporting_owner (the migration grants itself that membership). Run the first post-cutover migration as the historical migrator identity or a superuser; every later migration works under an ordinary CREATEROLE non-superuser member of the owner role.

  4. Verify: boot logs show no ALLOW_BROAD_DB_ROLE WARN; the #1456 boot guard passes (the session is the app identity, unprivileged); services/canopy-reporting/tests/least_privilege_role_test.rs proves the matrix on a real app-login connection.

Rollback

Point CANOPY_REPORTINGDATABASE_URL back at the broad credential and set CANOPY_REPORTINGALLOW_BROAD_DB_ROLE=true — the boot guard then WARNs loudly instead of refusing (the accountable override; ADR-041 doctrine: fail-closed default + explicit per-control override). Ownership/grants need no rollback: the broad credential is a superuser or owner-member and is not bound by the matrix. Remove the override as soon as the restricted login is restored — every boot under it is an auditable WARN.

Standing convention — every future reporting migration

Objects created by a later migration are owned by the MIGRATOR until transferred: end every new reporting migration that creates tables or functions with ALTER … OWNER TO canopy_reporting_owner + the app grants its runtime SQL needs. Skipping it silently regresses to migrator-owned objects the app cannot reach (fails loud in the devstack battery — the runtime IS the restricted login). The catalog-loop in 20261111000000 is the one-time backstop for pre-cutover objects only, not a recurring sweep.

chain-v2 Staging Ops & the Unpark Runbook (#1207)

The append-transport staging queue (chain_append_staging, canopy_security) parks a row when the drainer classifies a DETERMINISTIC per-row fault: prevalidation failure (digest mismatch, uncastable field), a substrate Row-class refusal, or a divergent replay (same event id, different content — already staged or already chained). Parked rows are an auditor-visible quarantine: the staging health snapshot degrades on ANY parked row, and they are excluded from every claim. Parking is single-shot by design — retrying a deterministic refusal is theater.

Unpark is an OPERATOR action under the maintenance/migration credential — never a runtime API (the app role cannot clear a quarantine it created):

  1. Inspect the quarantine: SELECT event_id, park_reason, attempts, staged_at, parked_at FROM chain_append_staging WHERE parked_at IS NOT NULL ORDER BY parked_at;

  2. Adjudicate the cause. A divergent replay park is potentially a Pub 1075 §9 signal (same identity, different content) — treat as an incident input, never a silent unpark. A prevalidation/refusal park usually means a builder/substrate version skew — fix the deployment first.

  3. Record a ticket reference for the decision (the park row itself is the evidence; do not delete it).

  4. Restore the row to draining: UPDATE chain_append_staging SET parked_at = NULL, park_reason = NULL, attempts = 0 WHERE event_id = '<id>'; The next drainer pass reclaims it. A row that reparks with the same reason confirms the fault is still live — escalate, don’t loop.

Residual staging while the flag is OFF (a trace of an aborted trial) stays visible: the stats sampler runs even when the drainer is dormant, and any staged or parked rows degrade the snapshot until drained or adjudicated.

chain-v2 Incident-Resolution Runbook (#1205, ADR-014 Amendment 9)

When a chain-v2 verifier detects an integrity finding it LATCHES a chain_incidents row: the family’s status goes breached (503), the family’s verification loops halt, and the state stays latched until the procedure below clears it — a clean scheduled pass NEVER clears a breach (scheduled runs are structurally barred from resolution; only a manual, job-linked revalidation run can). Design: plan chain-v2 verifiers D7.

Resolution is a guarded manual flow — there is no UI and no unguarded write path (the resolve fn enforces every precondition in SQL):

  1. Inspect the evidence under the incident-admin credential. Evidence and resolution text are readable ONLY by canopy_chain_incident_admin (the verify role and _app see evidence-free views): SELECT id, chain_family, chain_epoch, shard_id, kind, detected_loop_kind, detected_at, evidence FROM chain_incidents WHERE state = 'latched' ORDER BY detected_at; Evidence is positions + expected/got hex — enough to adjudicate whether this is data corruption, an operational fault, or tampering (a genuine break is Pub 1075 §9 reportable — see the Breach Notification Chain).

  2. Open a ticket. The resolution records an evidence_ref; the ticket is that reference. Never resolve without one — the fn refuses an empty ref.

  3. Trigger the manual revalidation job: POST /v1/security/chain/verify with {"family": …, "incident_id": "<id>"} (plus service for an FTI incident). An incident_id job bypasses the family halt gate and must run the incident’s DETECTED loop — stored at latch, never inferred — or family-full (the enqueue fn refuses a mismatched loop at the door). CLI: canopy security chain-verify --family … [--service …] [--wait].

  4. Confirm the outcome is ok. Poll GET /v1/security/chain/verify-jobs/{id} until state = done and the linked run’s outcome = ok. A rejected outcome means the finding is still live — the incident stays latched; back to step 1.

  5. Resolve under the incident-admin credential: SELECT chain_incident_resolve('<incident-id>', '<reason>', '<ticket-ref>', '<run-id>'); The fn records actor := session_user (enforced in SQL, never caller-supplied) and refuses unless the revalidation run is manual-mode
    ok, job-linked, of the STORED detected loop, scope-covering (whole-family or the incident’s shard), same instance/family, and finished AFTER the detection. On success the family’s halt gate reopens on the next pass.

Credential provisioning: the three chain-v2 roles — canopy_chain_verify (the background verifier + job claim protocol), canopy_chain_incident_admin (evidence read + resolve), and canopy_chain_anchor_emitter (anchor append + emitter transitions, #1278) — are NOLOGIN until the #1279 cutover provisions login carriers. Until then this runbook is not executable in production by construction (dormant verifier ⇒ no latches to resolve); rehearse it on a devstack with the test login carriers.

Audit Program-Scope Posture (#1519, epic &78 Part D)

Audit rows carry the authoritative programs set (audit_events.programs TEXT[], both twins): NULL = no assertion (the row is INVISIBLE to every scoped worker’s audit surfaces — role-agnostic, ADR-044), '{}' = asserted program-neutral (visible to all), {…​} = the named storage slugs.

Backfill posture — forward-only, deliberately. Rows written before the 20261128000000 migration have programs = NULL and DROP from the worker portal’s audit page, CSV export, panels and citations. There is NO synthetic backfill: inventing a program for a historical row would be an unverifiable authorization statement (ADR-016 forward-only; the household_id precedent). Operational access to pre-#1519 history:

  • Devstack: wipe/reseed (cargo xtask dev clean --confirm + seed) — the seed stamps every generated row.

  • Service-level access: canopy-security’s GET /v1/security/events with an EMPTY programs filter (service callers and direct admin API use) remains unscoped — the pre-#1519 contract — so history is reachable for incident response without a worker-portal disclosure surface.

  • Production: none exists (pre-UAT). If that changes before this paragraph is deleted, a deliberate backfill decision (derive from the hashed event_type/metadata, verified row-by-row) must be ruled on first — do not enable a worker-facing unscoped view instead.

Chain posture: the column is OUTSIDE the frozen v1 hash (dedup_key precedent; the 20261128000000 migration header records the tamper-evidence reasoning); the dormant chain-v2 tables deliberately do NOT carry it yet — the #1279 cutover rules on its v2 treatment.

Startup-Guard Inventory (fail-closed defaults + accountable overrides)

Doctrine (#1265, mirroring ADR-041): canopy never hard-denies a deployment’s choice. Each guard below refuses to boot outside CANOPY_ENV=development by default; the per-control override env var lets the operator proceed anyway, producing a loud, auditable per-boot WARN naming the accepted risk — the deployment owns it. Absence of a flag ALWAYS leaves its guard fail-closed; accepted syntax varies by loading path: the three #1412 flags and CANOPY_WEB__DEADLINE_OVERRIDE are read directly from the environment and accept exactly the literal true (lowercase — 1, TRUE, on are ignored), while the three serde-loaded flags (ALLOW_FABRICATED_VERIFICATION, ALLOW_INSECURE_SCANNER, ALLOW_BROAD_DB_ROLE) parse as standard config booleans (true/1/on/yes, case-insensitive). An override warns only when the guard would actually have refused (setting a flag alongside a compliant configuration logs nothing). Unset CANOPY_ENV resolves to production — omission is never permissive.

Guard (what refuses) Override Origin

Unencrypted DB connection: DATABASE_URL without sslmode=require / verify-ca / verify-full (every service, via canopy-db)

CANOPY_DB__ALLOW_UNENCRYPTED_CONNECTION

#1260, override #1412

DB-name ↔ service mismatch (ADR-001 program isolation, via canopy-db)

CANOPY_DB__ALLOW_NAME_MISMATCH

#1260, override #1412

Local (/tmp) object-store backend (data lost on restart, not shared across replicas — via canopy-store)

CANOPY_STORE__ALLOW_LOCAL_BACKEND

#1260, override #1412

Fabricated IEVS/SAVE/SSA data: a noop-adapters canopy-verification build

CANOPY_VERIFICATION__ALLOW_FABRICATED_VERIFICATION

#1265 (audit W2)

Uninspected uploads: scanner_backend=noop on canopy-applications (ADR-042)

CANOPY_APPLICATIONS__ALLOW_INSECURE_SCANNER

#1006

Broad reporting DB role (pre-cutover canopy superrole — see the credential-cutover runbook above)

CANOPY_REPORTING__ALLOW_BROAD_DB_ROLE

#1456

Out-of-range SSR deadline budgets on canopy-web (#1306 bounds)

CANOPY_WEB__DEADLINE_OVERRIDE

#1306

Not everything fail-closed carries an override — deliberately. A guard gets one only where the guarded condition is a deployment choice. A missing secret (CANOPY_ENCRYPTION_KEY — the mechanism cannot run without it) and identity admission (the ADR-044 worker program-scope claim, whose override is the IdP claim mapper, not a canopy flag) have none.

Deployment Rollback

Service Rollback

  1. Identify failing service(s) from health checks (GET /healthz returning error)

  2. Check recent deployment logs for what changed

  3. Revert container image tag — ADR-040 production refs are immutable per commit, so rollback is a redeploy of the previous known-good :<short-sha> (never "the previous image if not rebuilt", which only ever held for local from-source builds):

    # Docker Compose (prebuilt override — see the Scaling & Deployment
# runbook; the override interpolates BOTH image vars, so both must be
# set even for a service-only rollback)
export COMPOSE_FILE=docker-compose.yml:docker-compose.prebuilt.yml
export CANOPY_PREBUILT_IMAGES=true
export CANOPY_PREBUILT_SERVICE_IMAGE="$CI_REGISTRY_IMAGE:<previous-short-sha>"
export CANOPY_PREBUILT_PORTAL_IMAGE="$CI_REGISTRY_IMAGE/portal:<previous-short-sha>"
docker compose --profile snap-only up -d --pull always

# Kubernetes
kubectl rollout undo deployment/canopy-snap

  4. Verify health: GET /healthz returns {"status": "ok"}

  5. Investigate root cause before redeploying

Database Rollback

Migrations are forward-only by project convention. If a migration causes problems:

Option A (preferred): Corrective migration

Write a new migration that fixes the issue (e.g., ALTER TABLE …​ DROP COLUMN if a column was added incorrectly). This preserves the migration history and is auditable.

Option B (destructive): Restore from backup

  1. Stop the affected service

  2. Restore the PostgreSQL database from PITR backup to a point before the bad migration

  3. Restart the service with the corrected migration

  4. WARNING: This loses all data written after the backup point

Performance Troubleshooting

Slow Database Queries

  • Check PostgreSQL slow query log (log_min_duration_statement = 500 in postgresql.conf)

  • Verify indexes exist on frequently queried columns (check migration files for CREATE INDEX)

  • Check connection pool: if active = max_connections, increase db_max_connections

Event Bus Backpressure

  • Check RabbitMQ queue depth via management UI (http://rabbitmq:15672)

  • If queue depth > 10,000: check consumer services (canopy-security, canopy-notices, canopy-enrollment) for errors

  • Increase consumer count by scaling service replicas

Container Memory (OOM)

  • Check container memory limits vs actual usage

  • canopy-typst (PDF rendering) is the most memory-intensive operation

  • Increase memory limit if justified by load; 512 MB is sufficient for most services

Connection Pool Exhaustion

  • Default: 10 connections per service per database

  • Under heavy load: increase db_max_connections (but not beyond PostgreSQL’s max_connections)

  • Each service replica gets its own pool — 3 replicas × 10 connections = 30 connections per database

Archive Management

Audit events accumulate over time. The archive system (#1208, plan audit-archive-async) moves rows older than the operator-set age threshold from audit_events to audit_events_archive in bounded, per-chunk-committed passes — the single-transaction move of the original design is gone (#1245 deleted it: unbounded transaction, genesis- breaking boundary, silent loss). Each chunk is one atomic transaction (SET LOCAL statement_timeout → SET LOCAL canopy.audit_maintenance='on' → move ≤ archive_chunk_size rows with an inserted == deleted assertion → commit), the chain-head row is always retained, and progress is durable on the audit_archive_runs row after every chunk. API contract: canopy-security API › Archive Management.

Manual Archive

POST /v1/security/archive
Content-Type: application/json
Authorization: Bearer <admin-jwt>

{"archive_after_days": 2555}

Admin-only (service tokens 403). Returns 202 with a durable {run_id, poll_url} handle — the run executes asynchronously; poll GET /v1/security/archive-runs/{run_id} to completion. A 409 means a run is already active and carries that run’s handle. archive_after_days is an age threshold (domain 1..=36500), not a retention value — the archive retains rows indefinitely (retention policy = #1303).

Enabling scheduled archival (one-time procedure)

Scheduled archival ships dormant. Enablement is a three-step operator procedure; do the steps in order.

Step 1 — duplicate-wedge preflight. Verify the live table and the archive share NO row ids (a legacy partial move can leave overlap, which wedges every run):

SELECT count(*) FROM audit_events a JOIN audit_events_archive b USING (id);

The count MUST be 0. If it is not, resolve the overlap first — see Duplicate-wedge recovery below. Run this full overlap query at enablement time only (it is O(min(live, archive)); the runner’s own per-run preflight is a bounded first-chunk probe, not this).

Step 2 — pre-create the indexes on large live tables. The #1208 migration creates four indexes transactionally (CREATE INDEX IF NOT EXISTS, NOT CONCURRENTLY — the sqlx migrator’s advisory lock deadlocks with CONCURRENTLY’s snapshot wait, the 20260811000000 precedent). Fresh installs start empty and need nothing. On an already-huge live table, pre-create the four indexes `CONCURRENTLY out of band BEFORE deploying the migration, so the migration’s `CREATE`s no-op:

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_audit_events_received_at_id
    ON audit_events (received_at, id);
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_audit_events_archive_received_at_id
    ON audit_events_archive (received_at, id);
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_audit_events_archive_event_timestamp_id
    ON audit_events_archive (event_timestamp, id);
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_audit_events_archive_persons_metadata
    ON audit_events_archive USING GIN (metadata)
    WHERE source_service = 'canopy-persons';

Then verify every build completed valid (a failed CONCURRENTLY build leaves an INVALID index behind):

SELECT c.relname, i.indisvalid, i.indisready
FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname IN ('idx_audit_events_received_at_id',
                    'idx_audit_events_archive_received_at_id',
                    'idx_audit_events_archive_event_timestamp_id',
                    'idx_audit_events_archive_persons_metadata');

All rows must show indisvalid = t and indisready = t before deploying — DROP INDEX and rebuild any that do not. The migration’s DO block re-verifies name, table, validity, and definition at boot and fails the boot loudly on any mismatch.

Step 3 — configure and flip the flag. Set the CANOPY_SECURITYARCHIVE_* variables (Configuration Reference) — ARCHIVE_AFTER_DAYS is REQUIRED once the scheduler is enabled (no default: the age threshold is an operator decision, no policy is embedded) — then flip CANOPY_SECURITYARCHIVE_SCHEDULER_ENABLED=true. This flag is the accountable operator override: the deployment owns the threshold choice, and every scheduled run records requested_by = 'scheduler' with its frozen config snapshot on the run row. The manual endpoint works regardless of the flag — the runner is always spawned.

Duplicate-wedge recovery

A run finalized with error_code = duplicate_overlap (visible on GET /v1/security/archive-runs/{id} and in the canopy_security_audit_archive_run_failures_total{error_code="duplicate_overlap"} counter) means a row id exists in BOTH tables — the runner refuses to move anything past it (a chunk-level 23505 rolls the chunk back whole; committed progress from earlier chunks stands). There is deliberately NO in-code auto-reconcile (#1208 P4): automation deleting audit rows on equality heuristics is worse than a loud wedge.

Manual reconcile is permitted only when every column of the twin rows matches. Compare all 17 columns with IS NOT DISTINCT FROM (NULL-safe); the duplicate live row may then be deleted under the maintenance GUC in the same transaction:

BEGIN;
SET LOCAL canopy.audit_maintenance = 'on';
-- 1) Inspect the offending id (from the run row's error_detail):
SELECT a.id
FROM audit_events a
JOIN audit_events_archive b USING (id)
WHERE a.id = '<offending-id>'
  AND a.event_id        IS NOT DISTINCT FROM b.event_id
  AND a.event_type      IS NOT DISTINCT FROM b.event_type
  AND a.source_service  IS NOT DISTINCT FROM b.source_service
  AND a.action          IS NOT DISTINCT FROM b.action
  AND a.resource_type   IS NOT DISTINCT FROM b.resource_type
  AND a.resource_id     IS NOT DISTINCT FROM b.resource_id
  AND a.user_id         IS NOT DISTINCT FROM b.user_id
  AND a.user_role       IS NOT DISTINCT FROM b.user_role
  AND a.ip_address      IS NOT DISTINCT FROM b.ip_address
  AND a.metadata        IS NOT DISTINCT FROM b.metadata
  AND a.event_timestamp IS NOT DISTINCT FROM b.event_timestamp
  AND a.received_at     IS NOT DISTINCT FROM b.received_at
  AND a.created_at      IS NOT DISTINCT FROM b.created_at
  AND a.previous_hash   IS NOT DISTINCT FROM b.previous_hash
  AND a.event_hash      IS NOT DISTINCT FROM b.event_hash
  AND a.household_id    IS NOT DISTINCT FROM b.household_id;
-- 2) ONLY if step 1 returned the id (every column identical), delete the
--    live duplicate (the archive copy is retained):
DELETE FROM audit_events WHERE id = '<offending-id>';
COMMIT;
WARNING
If the twin rows do not match on every column, that is tamper evidence — two different rows claiming the same identity in an append-only ledger. Do NOT delete either row. Open a security incident (see Incident Response above) and preserve both rows as evidence. Never script bulk deletes over the overlap set.

Upgrade-state repair (archive non-empty, live empty)

Every run preflights this state and refuses with error_code = upgrade_state_unrepaired: with audit_events empty, the next append would re-genesis against a chain the archive already holds. Repair is a gated re-seed — copy the newest archive row back into audit_events under SET LOCAL canopy.audit_maintenance = 'on' so the next append chains from it. Purpose-built repair tooling is deferred to #1303; until it lands, treat the re-seed as an incident-class manual procedure (verify the chain afterwards via POST /v1/security/chain/verify once the verifiers are active).

Interpreting more and the backlog

more: true on a done run means the pass ended on a full chunk — more movable rows may remain (this coexists with the retained chain head). Either re-POST a continuation run (fresh Idempotency-Key) or let the scheduler’s catch-up cadence drain it: more pulls the next due time forward to archive_catchup_interval_secs (default 30s), so an enabled scheduler drains backlogs without operator action. Watch the canopy_security_audit_archive_backlog_capped gauge — movable rows remaining at the last probe, capped at 100001 (100001 = "more than 100k remain"; the probe never pays an O(table) count).

Querying Archived Events

GET /v1/security/archive?limit=50
Authorization: Bearer <admin-jwt>

Keyset-paginated (received_at DESC, id DESC): pass the last row’s (received_at, id) as before_received_at + before_id for the next page. The old offset + filter parameters are gone (#1208 decision 14).

Recommended Retention Policy

Data Type Minimum Retention Authority

FTI audit logs

7 years

IRS Publication 1075 AU-11 (ADR-004 Amendment 2)

HIPAA audit logs

6 years

45 CFR §164.530(j)

IEVS audit logs

Per state CMA

Computer Matching Agreement

General audit events

3 years

State records retention schedule

Determination records

7 years

7 CFR 272.1(f)

The archive age threshold is mechanical and independent of this table: moving a row live→archive never shortens its retention (retention = archive ∪ live, and the archive retains indefinitely). Retention policy — floors, legal hold, purge — is #1303.

Data Flow Diagrams

These diagrams show how sensitive data moves through Canopy’s service architecture. Each flow enforces the isolation requirements of ADR-001 (program service isolation) and ADR-004 (legally-scoped data tenancy).

PII Flow — SSN Encryption and Storage

SSN is the only PII field with application-layer encryption. All other PII fields rely on PostgreSQL TDE in production.

Diagram

Key files:

  • services/canopy-persons/src/store/persons.rs — encrypt_ssn() wraps canopy_common::crypto::encrypt()

  • crates/canopy-common/src/crypto.rs — AES-256-GCM implementation (12-byte random nonce, 16-byte auth tag)

  • services/canopy-persons/migrations/20260326000000_create_persons_tables.sql — ssn_encrypted BYTEA column

  • Key source: CANOPY_ENCRYPTION_KEY environment variable (base64-encoded 32-byte key)

On read, SSN is decrypted only for callers with Claims::require_caseworker_or_above(). The masked form (*--6789) is used in all other contexts.

FTI Flow — Federal Tax Information Isolation

FTI is used by TANF and Medicaid for income verification. Per IRS Publication 1075, FTI must never leave authorized program databases.

Diagram

Isolation enforcement:

  • crates/canopy-mq/src/publisher.rs — RESTRICTED_FIELDS array (27 field names: SSN, FTI, IEVS, HIPAA, PII, immigration data) validated before every publish

  • crates/canopy-db/src/lib.rs — validate_database_name() warns if a service connects to the wrong database

  • services/canopy-tanf/migrations/20260325000001_create_fti_audit_log.sql — Independent FTI audit trail (accessed_by, accessed_at, purpose_code, data_elements_accessed, originating_system)

  • canopy-medicaid has identical isolation requirements (see services/canopy-medicaid/migrations/COMPLIANCE.md) — FTI audit logging is implemented (Complete) via services/canopy-medicaid/migrations/20260326000001_create_fti_audit_log.sql (independent FTI audit trail) and services/canopy-medicaid/migrations/20260425000000_add_fti_audit_hash_chain.sql (ADR-014 SHA-256 previous_hash/event_hash chain). canopy-tanf carries the equivalent migrations.

IEVS Flow — Income Verification Data

IEVS match results are stored exclusively in the canopy-snap database per 7 USC §2025(e). The verification service is transient — it queries external sources but does not persist IEVS data.

Diagram

Key files:

  • services/canopy-snap/src/verification.rs — run_verification() orchestrates per-member IEVS queries

  • services/canopy-snap/src/verification_client.rs — HTTP client for POST /internal/v1/ievs/match

  • services/canopy-verification/src/api/ievs.rs — handle_ievs_match() dispatches to 4 data sources

  • services/canopy-verification/src/noop.rs — NoopIevsAdapter returns deterministic test data for UAT

  • services/canopy-snap/migrations/20260330000000_create_ievs_tables.sql — ievs_match_results and ievs_discrepancies tables (canopy-snap DB only)

  • Discrepancy threshold loaded from jurisdiction.toml [snap.ievs_discrepancy_threshold] (default: $100/month)

  • services/canopy-snap/src/events.rs — Events carry IDs and status only, never PII, income, or IEVS data

Determination Signing Flow (ADR-002)

Every eligibility determination is signed with ECDSA P-256 (detached JWS per RFC 7515). The orchestrator independently verifies each signature. Unverified determinations are quarantined.

Diagram

Key files:

  • services/canopy-snap/src/determine.rs:295 — Signs determination payload

  • crates/canopy-signing/src/signer.rs — SigningKey::sign_detached() (ECDSA P-256)

  • crates/canopy-signing/src/verifier.rs — VerifyingKeyRegistry with dual-key rotation support

  • services/canopy-eligibility/src/orchestrator.rs:345 — Signature verification, quarantine logic

  • services/canopy-eligibility/migrations/20260326000000_create_eligibility_tables.sql — program_determinations table with signature TEXT NOT NULL and signature_verified BOOLEAN

Encryption Inventory

Data Algorithm Where Status

SSN

AES-256-GCM (application-layer, per-field)

canopy-persons, canopy-snap

✓ Implemented

Determinations

ECDSA P-256 JWS (signing, not encryption)

Per program service → canopy-eligibility verifies

✓ Implemented

Audit chain

SHA-256 hash chain (integrity, not encryption)

canopy-security

✓ Implemented

HTTP traffic

TLS 1.2+ via rustls (OpenSSL banned)

All services

✓ Implemented

Database connections

TLS via sslmode=require

All PostgreSQL connections

✓ Configured

Message bus

TLS via amqps:// (production)

All RabbitMQ connections

✓ Configured

All PII at rest (beyond SSN)

PostgreSQL Transparent Data Encryption

Production database servers

Recommended — no code change required

Security Configuration Reference

The runbooks above describe how to respond; this section is the canonical reference for how Canopy’s security posture is configured. The universal posture (Kerckhoffs’s principle, public-visibility enforcement) is governed by the synced baseline at docs/modules/standards/pages/security-baseline.adoc and is non-negotiable across all projects — Canopy adds the project-specific configuration below.

Authentication

  • Keycloak OIDC with RS256 JWT (any RFC 6749 + OIDC-discovery-compliant provider works via config — see IdP Integration).

  • JWKS auto-refreshes hourly; force-refresh on unknown kid (30-second debounce) — see the Keycloak JWKS Keys runbook above.

  • Split issuer / fetch URLs are supported for Docker deployments (the in-cluster JWKS fetch URL can differ from the public token issuer).

Authorization

Role-based access control. Role guards live in the canopy-auth crate. Roles:

  • applicant

  • caseworker

  • eligibility_specialist

  • supervisor

  • quality_control

  • admin

Handlers enforce least privilege (e.g. Medicaid handlers require eligibility_specialist_or_above; SSN decryption requires Claims::require_caseworker_or_above()).

Federal Data Isolation (ADR-004)

Legally-scoped data tenancy isolates restricted federal data to the program services authorized to hold it (the per-data-type flows are diagrammed under Data Flow Diagrams below):

  • FTI isolated to canopy-tanf and canopy-medicaid, each with an independent audit log.

  • IEVS data isolated to canopy-snap (7 USC §2025(e)).

  • SSA SOLQ/BINDEX scoped per Computer Matching Agreement.

  • FDSH data isolated to canopy-medicaid.

  • No restricted data in event-bus payloads.

CI enforcement: cargo xtask compliance audit-data-tenancy (job compliance-data-tenancy) scans every service’s migrations and source for protected field-name patterns and fails the build if FTI, IEVS, or SSA SOLQ/BINDEX fields surface in an unauthorised service. The authorisation matrix lives at compliance/data-tenancy-authorisation.toml — one section per data class (fti / ievs / ssa_solq_bindex), each listing authorised services and the field-name patterns the scanner flags. Exceptions go through the block with a written justification.

Audit Posture

  • canopy-security subscribes to all events via the wildcard (#) routing key for system-wide audit (see Archive Management above for retention and the hash-chain integrity model).

  • FTI audit logs are maintained separately per IRS Pub 1075 (the fti_audit_log table in canopy-tanf and canopy-medicaid) and must be available for IRS on-site inspection independently of the general audit store.

Session Management

BFF services use PostgreSQL-backed sessions via tower-sessions-sqlx-store (with a Redis LRU cache). MemoryStore is banned (ADR-009).

Secret Management

  • Secrets are read from environment variables (CANOPY_{SERVICE}__*) — never hardcode secrets in source or checked-in config.

  • .env.example carries dummy/placeholder values and is checked into version control; .env is `.gitignore`d and never committed.

  • At-rest secrets are SOPS-encrypted YAML (secrets/dev.yaml, fake values only) per ADR-017; CI/CD secrets are stored as GitLab CI/CD variables (masked, protected).

Dependency Policy

  • OpenSSL is banned in deny.toml — rustls only. Containers are Alpine with musl.

  • cargo audit runs in CI as a blocking job; cargo deny checks license compatibility (AGPL-3.0-or-later allowlist) and known advisories.

  • CVE response timeline:

    Severity Patch SLA

    Critical

    Within 24 hours

    High

    Within 1 week

    Medium

    Within 1 month

    Low

    Next scheduled dependency update

Determination Signing (ADR-002)

ECDSA P-256 detached JWS for all eligibility determinations (signing config; rotation runbook is above, the end-to-end flow is diagrammed under Data Flow Diagrams):

  • Crate: crates/canopy-signing/ — SigningKey, VerifyingKey, VerifyingKeyRegistry.

  • Trait: DeterminationSigner in crates/canopy-signing/src/traits.rs — unified trait for all program services.

  • Key generation: cargo xtask gen-signing-keys --program snap generates PEM key pairs under .keys/.

  • JWS format: detached (RFC 7515 Appendix F) — header..signature with the payload omitted from the token.

  • Key format: PKCS#8 PEM (private), SPKI PEM (public).

  • Tamper detection: determinations with invalid or missing signatures are rejected (quarantined) by the canopy-eligibility orchestrator before assembling combined results.

Disqualification Screenings (7 CFR 273.11)

SNAP-specific disqualification screenings are stored in the snap_disqualification_screenings table (canopy-snap):

  • Drug felony (7 CFR 273.11(m)): jurisdiction policy in jurisdiction.toml — Georgia uses trafficking_only (partial opt-out per Georgia Code §49-4-186). Federal cutoff date: 8/22/1996.

  • Fleeing felon / probation violator (7 CFR 273.11(n)): categorical disqualification, no state variation; self-attested during application intake.

  • Striker (7 CFR 273.11(e)): pre-strike income comparison test — the household is eligible only if it would have been eligible using pre-strike income.

  • Exemption tracking: exemption_reason field with a worker audit trail (screened_by UUID).

Content Security Policy (BFF)

canopy-web and canopy-portal serve HTML and must apply a strict CSP:

default-src 'self'; script-src 'self'; style-src 'self';
img-src 'self' data:; frame-ancestors 'none'; form-action 'self'

Non-negotiable prohibitions:

  • No 'unsafe-inline' (on script-src or style-src).

  • No 'unsafe-eval'.

  • No inline <script> or <style> blocks in templates.

  • No inline style="" attributes — use utility classes in canopy-web.css.

  • No inline event handlers (onclick="", onsubmit="", @click="" with inline expressions, hx-on::*).

Implementation rules:

  • All JavaScript in external files under services/canopy-web/static/js/.

  • All CSS in external files under services/canopy-web/static/css/.

  • Dynamic theme variables served from the /theme.css endpoint.

  • Alpine.js: use the CSP build (@alpinejs/csp) with Alpine.data(name, …​) components — never inline expressions.

  • htmx: set includeIndicatorStyles: false via <meta name="htmx-config"> so it will not inject <style> at runtime.

  • Event handlers wired via addEventListener in the external JS bundle, delegated where possible.

Verification: the axe-core WCAG 2.1 AA audit in Playwright E2E must pass with zero critical violations.

Vulnerability Reporting

See SECURITY.adoc in the repository root for the public-facing vulnerability reporting process.

Edit this page · default