Cross-Program Alerts Scoping — Cutover & Operations (#596)

On this page

The #596 restructure makes the worker alerts feed assignment-scoped (PUB-1075 AC-6) with no unscoped fallback and no deployment override — activation is therefore governed by DATA provisioning, not a flag.

Cutover ordering

  1. Provision real household_assignments BEFORE deploying. A worker with zero active assignments sees an empty panel by design (the scoped feed returns []). The devstack seed’s partition (tools/canopy-seed, phase 15 — every household assigned to one of two caseworker fixtures) is the template; production provisioning is an operations task against canopy-applications' POST /v1/workers/{worker_id}/assignments (service-gated, supervisor-actor).

  2. Preflight coverage query (run against canopy_applications
    canopy_eligibility): the fraction of households carrying a recent alert-status determination that hold ≥1 active assignment. Aim for ~100% before cutover; every uncovered household is an alert no caseworker’s panel will surface (supervisors still see it on /all).

    -- on canopy_eligibility: recent alert households
SELECT DISTINCT household_id FROM program_determinations
 WHERE status IN ('denied','sanctioned','time_limit_exceeded',
                  'disqualified','terminated','abawd_exceeded')
   AND signature_verified
   AND determined_at > now() - interval '90 days';
-- on canopy_applications: which of those carry an active assignment
SELECT household_id FROM household_assignments
 WHERE unassigned_at IS NULL AND household_id = ANY(:above);

  3. Deploy order is free. The old path is retired rather than aliased: an old eligibility replica 404s the new paths and a new replica 404s the old one, so ANY mixed window fails closed (panel error state) — never an unscoped leak. Single-service rollback likewise fails closed.

Index note (large live deployments)

Migration 20260819000000 supersedes the alert index with the VERSIONED idx_program_determinations_alert_status_v2 (adds the id DESC tiebreak + signature_verified), adds idx_program_determinations_alert_household, and drops the old idx_program_determinations_alert_status last (metadata-only). On a multi-million-row LIVE deployment, pre-create BOTH new indexes CONCURRENTLY out of band and verify pg_index.indisvalid first — the guarded CREATE IF NOT EXISTS statements then no-op (the versioned name is what makes this safe: a same-name rebuild would have dropped and rebuilt your pre-built index under a full write lock).

Monitoring

  • canopy_eligibility_assignment_lookups_total{outcome} — rising error / breaker_open is "caseworker panels are erroring while supervisor /all still serves" (the breaker opens after 5 consecutive failures for 30s; every failure arm is a coded 502 applications_unreachable, never unscoped data).

  • canopy_eligibility_assignment_cardinality — caseload growth radar for the 5,000 fail-closed cap (assignment_set_too_large).

  • canopy_eligibility_scoped_alerts_query_duration_seconds — the LATERAL top-N; degradation suggests the household partial index is missing or invalid.

  • Access breadcrumbs: eligibility.cross_program_alerts.accessed events (actor, effective worker, scope, assignment count, result count) land in canopy-security’s audit chain.

Revocation semantics

The per-request assignments lookup is the authorization linearization point: an unassignment committed mid-request does not retract that in-flight response, and the panel cache is unconditionally bypassed for this surface, so the NEXT refresh reflects the revocation.

Edit this page · default