Incident Response

On this page

Incident Types

This runbook covers three categories of incidents. Each has distinct regulatory timelines and notification requirements.

1. FTI Breach (IRS Pub 1075 §10)

Federal Tax Information is subject to the strictest breach notification requirements. Failure to comply may result in loss of FTI access for the entire agency.

Timeline

  • Within 24 hours: Notify the IRS Office of Safeguards.

  • Within 24 hours: Contact TIGTA (Treasury Inspector General for Tax Administration) at 1-800-366-4484.

  • Within 72 hours: Submit IRS Form 2350 (Preliminary Breach Report).

Containment

  1. Immediately isolate the affected system(s) from the network. Do not power off or reboot — preserve volatile memory for forensics.

  2. Disable the compromised service account or API key.

  3. If canopy-tanf or canopy-medicaid is affected, halt their FTI audit log background tasks but do not truncate or delete any data.

Audit Log Preservation

Preserve the FTI audit trail before any remediation:

# Export FTI audit logs from canopy-tanf
curl -sf -H "Authorization: Bearer ${AUDITOR_TOKEN}" \
  https://{tanf-host}/v1/fti-audit-log > fti-audit-tanf-$(date +%Y%m%d).json

# Export FTI audit logs from canopy-medicaid
curl -sf -H "Authorization: Bearer ${AUDITOR_TOKEN}" \
  https://{medicaid-host}/v1/fti-audit-log > fti-audit-medicaid-$(date +%Y%m%d).json

Store exports on encrypted, access-controlled media. These logs are legally required evidence and must not be modified.

Notification

Prepare the IRS notification with the following information:

  • Date and time the breach was discovered.

  • Description of FTI data involved (tax return data, SSN, income).

  • Number of affected individuals.

  • Containment actions taken.

  • Remediation plan and timeline.

2. HIPAA Breach (45 CFR 164.408)

Protected Health Information (PHI) in canopy-medicaid and related services is subject to HIPAA breach notification rules.

Timeline

  • Within 60 days of discovery: Notify HHS (Department of Health and Human Services) via the HHS Breach Portal.

  • Within 60 days of discovery: Notify affected individuals by first-class mail.

  • If 500+ individuals in a single state are affected: Notify prominent local media outlets.

Risk Assessment

Conduct a four-factor risk assessment per 45 CFR 164.402(2):

  1. Nature and extent of PHI involved (diagnosis codes, Medicaid ID, SSN).

  2. Unauthorized person who used or received the PHI.

  3. Whether PHI was actually acquired or viewed (vs. opportunity alone).

  4. Extent of mitigation — what was done to reduce harm.

If the assessment demonstrates a low probability that PHI was compromised, the breach exception may apply and notification is not required. Document the assessment regardless.

Remediation

  1. Identify and patch the vulnerability or access control gap.

  2. Rotate any compromised credentials (see Signing Key Rotation if signing keys are affected).

  3. Review and tighten RBAC policies in Keycloak for Medicaid-scoped roles.

  4. Update canopy-security breach detection rules if the incident exposed a gap.

3. Service Outage

Service outages affect system availability but may not involve data breach. The goal is rapid restoration with minimal impact to caseworkers and applicants.

Health Checks

Verify the status of each service:

# Core services
curl -sf https://{rules-host}/healthz
curl -sf https://{persons-host}/healthz
curl -sf https://{applications-host}/healthz
curl -sf https://{eligibility-host}/healthz
curl -sf https://{snap-host}/healthz
curl -sf https://{tanf-host}/healthz
curl -sf https://{medicaid-host}/healthz

# Supporting services
curl -sf https://{verification-host}/healthz
curl -sf https://{enrollment-host}/healthz
curl -sf https://{renewals-host}/healthz
curl -sf https://{notices-host}/healthz
curl -sf https://{appeals-host}/healthz
curl -sf https://{reporting-host}/healthz
curl -sf https://{security-host}/healthz

# BFF services
curl -sf https://{web-host}/healthz
curl -sf https://{portal-host}/healthz

Infrastructure Connectivity

Check backing services:

# PostgreSQL connectivity
psql "${DATABASE_URL}" -c "SELECT 1;"

# RabbitMQ management API
curl -sf -u "${RABBITMQ_USER}:${RABBITMQ_PASS}" \
  https://{rabbitmq-host}:15672/api/overview

# Keycloak realm availability
curl -sf https://{keycloak-host}/realms/canopy/.well-known/openid-configuration

Recovery

  1. Restart the affected services:

    cargo xtask dev restart

  2. If a single service is affected, restart only that container:

    docker compose restart {service-name}

  3. After restart, verify recovery:

    # Re-check all health endpoints (see above)

# Run the test suite to confirm functional correctness
cargo xtask test

  4. If database migrations are suspected, verify schema state:

    cargo sqlx migrate info --source crates/{service}/migrations

Common Steps

These steps apply to all incident types.

Evidence Preservation

  1. Do not destroy evidence. Do not delete logs, restart services (unless required for containment), or modify databases until forensic collection is complete.

  2. Capture the following immediately:

    • Application logs from all affected services.

    • Database audit tables (fti_audit_log, security_audit_events).

    • RabbitMQ message traces from canopy.events exchange.

    • Keycloak authentication and admin event logs.

    • Network flow logs and firewall logs if available.

  3. Store all evidence with cryptographic timestamps and chain-of-custody documentation.

Communication Template

Use this template for initial internal notification:

Subject: [INCIDENT] {Type} - {Severity} - {Date}

Summary: {Brief description of the incident}
Discovery time: {ISO 8601 timestamp}
Affected systems: {List of services}
Affected data: {FTI / PHI / PII / None}
Estimated scope: {Number of affected individuals, if known}
Current status: {Investigating / Contained / Resolved}
Incident commander: {Name}
Next update: {ISO 8601 timestamp}

Post-Incident Review

Conduct a blameless post-incident review within 5 business days of resolution.

  1. Timeline reconstruction: Document the full sequence of events from root cause to resolution.

  2. Root cause analysis: Identify the underlying technical, process, or human factors.

  3. Impact assessment: Quantify affected users, duration, and data exposure.

  4. Action items: Create GitLab issues for each remediation task with:

    • type::security or type::bug label.

    • priority::critical or priority::high label.

    • Assigned owner and due date.

  5. Runbook updates: If this runbook was insufficient, update it as part of the remediation.

  6. Store the completed review document in docs/modules/ROOT/pages/incident-reviews/ with the naming convention {YYYY-MM-DD}-{brief-description}.adoc.

Edit this page · default