Plan: October-COLA bulk re-determination — program decomposition (#1213, epic &73)

Epic: &73
Issues: #1213 (core) · #1467 #1468 #1469 #1470 #1471 #1472 (prerequisites) · relates #1133
Branch: feature/1213-cola-program-plan (this docs MR); implementation branches per issue

Scale-audit finding H6: a determination is only ever a synchronous, single-attempt, ~30-call HTTP fan-out per household inside one interactive request (services/canopy-eligibility/src/orchestrator.rs:1398-1412). The mandatory 7 CFR 273.12(e) October COLA (rulesets/federal/indexing.toml pins snap-cola to Oct 1 with grace_days = 0; GA ≈ 800K households) has no driver.

The first #1213 plan put everything in one MR. The external review found that unshippable, most fundamentally: the October 2026 policy cannot even be represented today — snap boot-loads hardcoded FY2026 parameter filenames (services/canopy-snap/src/params.rs:107-118), the rules corpus deliberately excludes parameter JSONs (services/canopy-rules/src/engine.rs:63-66), and benefit dates derive from wall clock (services/canopy-snap/src/determine.rs:377). A 2026-10-01 bulk run would have signed FY2026 amounts with October-2nd dates — the driver would be theater. The remaining blockers (replay non-convergence, consumer deadlock, TOCTOU admission, wrong supersession baseline, unsafe signed-schema rollout, un-gated downstream blast radius) are each owned by a named fix below.

Enrollment apply-semantics stay out of scope (#1133): an already-enrolled household’s re-determination PARKs at enrollment (services/canopy-enrollment/src/auto_enroll.rs:141-170, the blessed #1130 bridge). The unpark scanner re-offers 50 events/min, so a full-caseload production enact would accumulate ~800K parks taking days to drain — production COLA enactment at full scale is therefore operationally gated on #1133. #1213 ships the machinery and small-cohort operation.

In scope (program): the six prerequisite issues (#1467–#1472) and the #1213 core (cohort substrate, dispatcher/consumer, admin surface with real dry-run, fail-closed enact gate, operator controls, the failure-mode test suite).

Out of scope: #1133 enrollment adjust-vs-supersede + 273.13 adverse-action routing; snap params/corpus unification (P1 adds selection + attestation, not a merged pin); FTI actor-JWT minting for FTI-bearing cohort programs (follow-up filed at core time); snap-side same-context supersession replay (follow-up filed at core time).

Frozen by the approved program plan; the core’s plan round turns each into files/DDL/signatures.

Retry ownership. Case rows own ALL retries; MQ is a trigger only. The consumer handler never returns Err for business outcomes (only malformed payloads DLQ). No FOR UPDATE held across HTTP (the reviewed deadlock: handler-tx row lock + second-connection bookkeeping). Execution claim = CAS on a short separate connection (… WHERE state='dispatched' AND dispatch_generation=$n AND attempt_started_at IS NULL); CAS miss ⇒ ack no-op. Scheduling lives in next_attempt_at (exponential + jitter); the dispatcher republishes. Dissolves the MQ-attempts-vs-case-attempts contradiction and RabbitMQ’s immediate-requeue-no-backoff.

Dispatch generations fence everything. dispatch_generation on the case, stamped into every event; consumer CAS requires a match; re-arm bumps it (stale queued duplicates are no-ops); in-flight accounting counts the current generation. bulk_redispatch_ttl_secs boot-validated > max_inflight × self_call_timeout / consumer_concurrency.

Entry idempotency — generation-scoped natural key. At first dispatch the case freezes its canonical DetermineRequest (request_body JSONB + hash); every attempt resends identical bytes. eligibility_requests += bulk_case_id, bulk_attempt_generation; unique partial index on (bulk_case_id) WHERE status IN ('pending','in_progress'). Resume: a completed request from the current generation with a definitive result ⇒ succeeded (replay from persisted rows); non-definitive ⇒ bump generation, re-execute. The self-call sends no idempotency-key header — eligibility IS wrapped by the optional #1003 middleware (crates/canopy-api/src/lib.rs:191-195 + the #1463 fleet schema), and a keyed state-dependent 409 would be cached replayable for 24h (idempotency.rs:1052-1060). The D4 hash key lives at the per-program layer, keyed over frozen body + policy target + generation.

Outcome classification is body-parsed (P5’s typed failures): succeeded requires every requested program definitive (approved/denied bucket, determination_id: Some); pending/quarantined ⇒ transient; supersession-conflict ⇒ skipped_superseded; terminal-4xx ⇒ failed_terminal; 408/429/5xx/transport ⇒ transient. Bulk suppresses register_pending_verifications entirely (D6).

Interactive priority — serialized admission. create_eligibility_request takes pg_advisory_xact_lock(hashtext(household_id)) on both paths; the bulk arm probes for ANY live request for the household (sibling applications included — the (application_id, household_id) index doesn’t fence them) with the 5-minute staleness qualifier, under the lock. No TOCTOU. Recorded residual: an interactive request arriving while a bulk determination is already executing sees today’s standard in-progress 409 (bounded seconds); "interactive always wins" = wins every admission race, never queues behind bulk.

Supersession baseline. Cases persist baseline_determination_id from the P4 snapshot row. A typed supersession-conflict ⇒ skipped_superseded (own terminal state, counted separately) — an intervening successor post-flip already carries the new policy; pre-flip intervening successors surface on the failures list for re-run. No blind retries against snap’s one-successor constraint.

Signed-provenance validation. Bulk arm rejects as contract_violation (terminal) any envelope whose trigger, policy target, or previous_determination_id mismatch the dispatch context — an old snap that ignored the new fields cannot silently pass. DetermineRequest.bulk requires claims.service_id() == Some("canopy-eligibility") (exact identity); BulkDispatchContext carries cohort_case_id + dispatch_generation, validated against the durable case row.

Provenance persisted. cohort_case_attempts (case, generation, eligibility_request_id, outcome, error code/excerpt, started/finished) — attempt history, never overwritten. Program rows (P5) + snap persist trigger and the policy target; replay reconstruction reads persisted rows.

Side-effect blast radius — fail-closed with accountable override. bulk_runs_enabled default false. Enact additionally requires CANOPY_ELIGIBILITY__BULK_ENACT_ACCEPT_DOWNSTREAM=true; the refusal names what enact unleashes pre-#1133 (per-case determination.completed.snap → notices NOAs, ELE, Medicaid granting-source effects, enrollment parks at 50/min drain). Fail-closed default + explicit per-control accountable override, per the ADR-041 doctrine. A doc warning is not a gate; this is a gate.

Worker pass order + fencing. Per pass: (1) deadline/finalization, (2) enacting dispatch, (3) re-arm, (4) materialization — an upstream outage can’t starve finalization. Per-run error isolation; run-row FOR UPDATE around cursor/count writes with expected-value predicates. Run-level circuit breaker: failure ratio over a sliding window ⇒ run paused (operator resumes).

Startup/shutdown/readiness. Subscribers start only after the HTTP server is ready (bind-then-subscribe); disabled ⇒ never subscribed; consumer handles + worker registered for graceful shutdown; readiness exposes worker and subscriber liveness (non-gating).

Pool + concurrency honesty. The dedicated 2K+2 pool serves consumer bookkeeping/inbox only; the loopback handler intentionally uses the interactive pool — K bounding concurrent self-calls IS the contention control. Boot validation db_max_connections ≥ 2K + 4. K is per-replica; replicas share work via SKIP-LOCKED dispatch + competing consumers. CANOPY_MQ_PREFETCH_COUNT asserted ≤ 4 at boot when consumers are enabled (never a silent 32). Shared program circuit breakers: breaker-open ⇒ transient ⇒ the run breaker pauses the run rather than burning attempts.

Scale is measured, not claimed. Throughput = replicas × K ÷ mean per-case latency; the run deadline is a knob. Acceptance: a seeded soak (≥5K cohort on devstack) measuring throughput + interactive latency under drain; a backlog-slope metric + alert; a production sizing table in the runbook.

Operator surface. GET …/bulk-runs/{id}/failures (paginated); POST …/pause, /resume, /cancel, /retry-failures (bounded reset, bumps generation); status = per-state counts, throughput, ETA, breaker state. Retention reaper for terminal cases/attempts/entry keys.

Run/state invariants. Finalization requires succeeded + failed_terminal + skipped_superseded + canceled = materialized_count; failures ⇒ completed_with_failures, clean ⇒ completed. Completed-target re-run needs allow_rerun=true; enact during another enact / on a failed run ⇒ 409 naming it; Location on 202s; BulkRunAccepted.request_id named per D10’s wording. Clients must not send idempotency-key headers on bulk-runs POSTs (keyed state-dependent 4xxs cache 24h).

Create contract restricted (v1). trigger must be cola; programs must be [Program::Snap] (canopy_reference::Program, duplicates rejected); as_of required, validated against the snap-cola cutover window. A signed change-report trigger that actually ran a COLA scan would be cryptographically authoritative and semantically false.

Principals split (D3/D8). Run persists created_by, enacted_by, and the captured authz basis separately; per-case requested_by = system:cola-redetermination:<run> because the cohort is machine-selected — the D3 "real originating worker" clause is satisfied at the run level. FTI actor-JWT follow-up stands.

Config truth table. bulk_runs_enabled=false ⇒ no subscribe, no worker, 503 on POSTs (the only dormant state). =true with any URL/knob missing or out of range ⇒ boot error — never silently dormant. Cross-field checks: the TTL inequality, the prefetch assertion, the pool floor.

Step 0 is this MR (see Status). Steps 1–4 (#1468, #1469, #1471, #1470) are implementable directly from §Design — prerequisites; each lands as its own MR with its issue’s acceptance criteria as the test bar. Step 5 (#1467) and Step 7 (#1213 core) are design-bearing and get their own plan rounds; the core plans against §Design — core, and its verification carries the review’s failure-mode suite verbatim: crash-after-program-row, middleware cache expiry, same-key-different-body, intervening successor, churn between preview and enact vs the frozen snapshot, outage longer than the redispatch TTL, worker lock loss, startup with a preloaded queue, disable with queued work, all-cases-failed finalization, canonical replay, rolling-restart signature tolerance.

Status: Done (2026-08-16) — the core landed as one MR (!1141, merge 68bab6c5: contracts/common, snap delta, cohort substrate, bulk arm, consumers + worker + config, admin surface, the live AC6
failure-mode suite, these docs), after #1473 shipped as its own MR (!1140, merge 25f8614e). The as-built record — the epoch/ledger split, refinements 1–8, and every deviation from this plan’s §Design — core skeleton — is ADR-002 Amendment 3; operations live in the bulk COLA scaling runbook. Follow-ups filed at implementation start: #1480 (pending-verification dedup), #1481 (breaker half-open CAS), #1482 (re-baseline superseded cohort cases), #1483 (FTI actor-JWT).

All review findings are folded in; three deliberate deferrals are recorded here rather than silently dropped: