Plan: CMD change-report pipeline — facts → order → signed re-determination (#575, epic &77)

On this page
NOTE

Extends the ADR-002 Amendment 1 async determination contract with its single-case Order arm (the D1 field list taken at its word); governed by ADR-004 (event allowlist), ADR-028 (signing ritual), and ADR-043 (the exchanged-bearer worker action). Enrollment apply-semantics stay #1133; the medicaid NOA route and the 10-day escalation workflow are filed, linked gaps — not silently absorbed (§D6). Architecture rulings (2026-08-18): no shortcuts, no fig leafs, no pre-1.0 compat; fact writes go through the BFF worker action.

Status

Step Description Status

0

Epic &77 + child issues #1503–#1508; #575 attached; this plan committed + nav-linked.

Done (2026-08-18) — direct-to-main docs commit

1

#1503 (MR-1) Closed vocabularies: DisabilityStatus + CmdEventType in canopy-reference; contracts-persons write validation; rules-comparison sweep test; RequestOrigin type.

Done (2026-08-18) — MR !1163, merge 860c3112; #1509 (TANF != null exemption trap) filed by the sweep

2

#1504 (MR-2) The order substrate: DeterminationRequestedV1 two-arm reshape, determination_orders table, consumer generalization, origin on eligibility_requests + DeterminationCompletedV1, deterministic order key (KAT), order sweep retry, ACL/topology, rollout runbook.

Done (2026-08-18) — MR !1164, merge ef6d02a7; deviations recorded in D2 (standalone always-on sweep, ORDER_MAX_ATTEMPTS=12, claim-token settle fence)

3

#1505 (MR-3) Medicaid plumbing mirroring snap #1213 D-6: context trigger/previous_determination_id, gated as_of, signed effective_date, row columns
CHECK, medicaid.determined additive trigger.

Done (2026-08-18) — MR !1165, merge c905c973; hardened in review (approval-only effective_date, validated+gated pin, supersession-fork 409 backstop)

4

#1506 (MR-4) The CMD subsystem: lifecycle migration (state/deadline/legacy), ingest reshape (202, typed event type, application_id, receiver-contract guard
EffectiveUser, order staged in-tx), cmd-settle consumer, list endpoint, 10-day clock metrics + alarm.

Done (2026-08-18) — MR !1166, merge aebe9591; recorded window: the worker CMD form is dark until step 5 lands (deliberate; both deploy together)

5

#1507 (MR-5) The BFF two-step fact action: persons disability update
exchanged-bearer CMD ingest, closed-vocab form, cmd-queued banner, determination-tab status.

Done (2026-08-18) — MR !1167, merge ad74fadf; hardened in review (identity-first guard ordering, parse-before-write, ssi_recipient allowlist)

6

#1508 (MR-6) Flip-test e2e suite, ADR-002 Order amendment, CHANGELOG, Antora, compliance annotation, gap filings (medicaid NOA, escalation workflow, /relate #1133); #575 closes.

Done (2026-08-18) — this MR; flip test green live (baseline SSI approval → change-report DENIAL w/ supersession linkage, settled in ~2s); gaps filed #1510 (NOA) + #1511 (escalation)

Epic: &77
Anchor issue: #575 (type::bug, priority::medium, T1)
Children: #1503 (w2) · #1504 (w5) · #1505 (w3) · #1506 (w5) · #1507 (w3) · #1508 (w3)

Context

The bug. POST /v1/cmd/ingest records a medicaid_cmd_events row and nothing ever happens: no fact changes, no re-determination, processed_at NULL forever, a banner tells the worker to click Run Determination. External review established the deeper truth: even wiring a re-run would be inert — the SSI COA keys solely on persons.disability_status == "ssi_recipient" (rulesets/georgia/medicaid-non-magi.json:31), that column is free TEXT no worker surface can edit, SOLQ’s ssi_active is never consulted for eligibility, and a naive MQ-consumer-calls-determine loop mints duplicate signed determinations on retry.

The architecture.

  1. Facts change first, through their owner. The worker-portal CMD action becomes a real two-step: canopy-web updates the person’s disability_status via canopy-persons (worker-attributed, program-scope + household-membership IDOR guards — the income-editor precedent), THEN records the CMD event in medicaid. disability_status becomes a closed, validated vocabulary.

  2. The re-determination rides ADR-002’s async contract, taken at its word. determination.requested is reshaped to the Amendment 1 D1 field list — typed subject
    programs + pinned as_of + signed trigger + idempotency-derivation identity — as a tagged two-arm payload (Cohort = the #1213 shape; Order = single-case). Eligibility owns a durable order-execution substrate generalized from the bulk core (D4 deterministic key, D5 durable per-unit state, D6 bounded retry, D9 signed change-report trigger
    previous_determination_id). Medicaid publishes the order and consumes determination.completed (extended with an origin-correlation slot) to settle its CMD row. Pure event choreography — no bespoke HTTP retry loop in medicaid.

  3. Honest boundary. #575 completes at "facts changed + signed re-determination linked
    CMD row settled + 10-day clock visible/alarmed". The medicaid NOA route, enrollment supersession (#1133), and the escalation workflow are program-wide gaps existing for ALL medicaid determinations — filed and linked (§D6), with the compliance catalogue annotation updated to say exactly what is and is not covered.

Verified substrate (at HEAD): eligibility_requests one-pending-slot idx_unique_pending_request (application_id, household_id) WHERE pending/in_progress
5-min stale sweep; DeterminationCompletedV1 {request_id, application_id, household_id, programs_approved, programs_denied} staged atomically with completion (#1471), queue-bound by NO domain consumer today; bulk_dispatch_idempotency_key (crates/canopy-contracts-eligibility/src/bulk.rs:219) + the cohort CAS/ledger/settle machinery (services/canopy-eligibility/src/bulk/store.rs); medicaid ApplicationContext.as_of exists (TMA arithmetic; un-gated); the medicaid envelope signs effective_date: None (determine.rs:1386) and never sets previous_determination_id; snap’s trigger plumbing to mirror (context → envelope.trigger → row column w/ CHECK → completed-event additive field); the persons fetch honors ?as_of=; medicaid is already an ADR-043 receiver (#1426); BFF fact-editor guards deny_unless_in_scope
ensure_household_member (fact_editor.rs:25,98); the notices oldest-age gauge pattern (canopy-notices/src/metrics.rs:121); ACLs: eligibility may publish determination.(completed|requested), medicaid’s regex has neither (devstack/rabbitmq/definitions.json:335,356).

Design

D1. Closed vocabularies (canopy-reference + contracts-persons) — #1503

  • DisabilityStatus enum in canopy-reference (IncomeType/AssetType template: serde + strum snake_case). Variants = the UNION of every value the tree actually compares: none, disabled, disabled_veteran, ssi_recipient, plus the non-MAGI comparison set found by the implementation-time sweep of all rules JSONs + code comparisons (ssdi_recipient, state_determined, not_disabled, blind expected). Rule: no variant ships that nothing compares; no comparison survives that the enum lacks — pinned by a sweep test. Contracts-persons CreatePerson/UpdatePerson validate against it (the validate_ssn_digits custom-validator wiring); the Person read DTO keeps Option<String> on the wire; the write path refuses unknown values. Existing DB rows are inside the union.

  • CmdEventType enum in canopy-reference. v1 vocabulary: ssi_terminated. Adding a variant later = defining its fact-write semantics — deliberate, not free.

  • RequestOrigin { source: String, ref_id: Uuid } in canopy-contracts-eligibility — the requester-correlation identity for async orders (e.g. ("medicaid-cmd", cmd_event_id)).

D2. determination.requested reshaped; the order substrate (canopy-eligibility) — #1504

Contract (pre-1.0 reshape, no compat arm): DeterminationRequestedV1 becomes a tagged enum:

pub enum DeterminationRequestedV1 {
    Cohort { run_id, case_id, dispatch_generation: i32, phase: CohortPhase },   // = today
    Order  { origin: RequestOrigin, application_id, household_id,
             programs: Vec<String>, as_of: NaiveDate,
             trigger: DeterminationTrigger, requested_by: String },
}

The Order arm IS the ADR-002 D1 field list. IDs + scalars only (ADR-004). CHANGELOG Changed records the wire reshape; the bulk worker/consumer update in the same diff.

Durable order state — migration determination_orders:

CREATE TABLE determination_orders (
    id UUID PRIMARY KEY CHECK (uuid_extract_version(id) = 7),
    origin_source TEXT NOT NULL, origin_ref UUID NOT NULL,
    application_id UUID NOT NULL, household_id UUID NOT NULL,
    programs TEXT[] NOT NULL, as_of DATE NOT NULL,
    trigger TEXT NOT NULL, requested_by TEXT NOT NULL,
    state TEXT NOT NULL DEFAULT 'pending' CHECK (state IN
        ('pending','dispatched','succeeded','failed_terminal','failed_retryable')),
    program_epoch INT NOT NULL DEFAULT 1 CHECK (program_epoch >= 1),
    attempt_count INT NOT NULL DEFAULT 0, next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    attempt_started_at TIMESTAMPTZ,
    eligibility_request_id UUID, last_error_code TEXT, last_error_excerpt TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (origin_source, origin_ref)              -- the durable dedup identity
);

eligibility_requests gains origin_source TEXT, origin_ref UUID (paired-null CHECK, index) so completion can carry the correlation.

Consumer generalization: the existing canopy-eligibility.bulk-redetermination subscriber dispatches on the payload arm. Cohort → the untouched #1213 path. Order → handle_order: upsert the order row (ON CONFLICT (origin_source, origin_ref) DO NOTHING + read-back — redelivery converges), CAS-claim (attempt_started_at IS NULL AND state IN ('pending','failed_retryable') AND next_attempt_at ⇐ now()), then execute through the same in-service path the bulk arm proved: self-call POST /v1/eligibility/determine with a deterministic idempotency key hash("canopy-order-dispatch-v1\0" ‖ origin ‖ program ‖ epoch) (D4; KAT-pinned sibling of bulk_dispatch_idempotency_key), requested_by from the order, and the request row created with the origin columns. Retry semantics mirror the bulk classification matrix: transient/5xx/409-slot-busy → failed_retryable + backoff (60s doubling, 1h cap, jitter — deferred via next_attempt_at; the MQ delivery acks and a 1-minute order sweep in the existing worker re-stages due orders, so hot-retry exhaustion cannot occur); other 4xx → failed_terminal (loud, DLQ-equivalent state, metric); a non-definitive completed result → epoch bump + retry. Crash-after-determine recovery: the 24h idempotency middleware replays; the completed request row with matching origin is adopted on the next attempt (result replay, not re-execution).

Orchestrator threading: the determine path accepts pinned as_of + trigger
previous_determination_id resolution for order-driven runs and threads them into context assembly (fetch_household_context(?as_of=) already exists) and the medicaid context (D3). stage_determination_completed extends DeterminationCompletedV1 with origin: Option<RequestOrigin>.

MQ/ACL: medicaid’s topic-write regex gains determination.requested (definitions.json:356); the topology lint stays clean (consumer exists first); the production ACL step is recorded in the rollout runbook section.

D3. Medicaid determination plumbing (mirror snap’s #1213 D-6, point for point) — #1505

  • ApplicationContext (contracts-medicaid) gains trigger: Option<DeterminationTrigger> and previous_determination_id: Option<Uuid> (serde default + skip — wire-stable when absent); as_of gains snap’s #1467 posture: a non-fallback value requires the exact canopy-eligibility identity (403).

  • Envelope: set envelope.trigger and envelope.previous_determination_id (both already signature-bound in canopy-signing) BEFORE signing; effective_date = Some(as_of) replaces the hardcoded None at determine.rs:1386 — and the row write at :1612.

  • Row: migration adds trigger TEXT (kebab-case CHECK, snap’s 20261121000000 shape)
    previous_determination_id UUID to medicaid_determinations, plus the snap-mirrored one-successor-per-prior partial unique (409 supersession_conflict backstop against concurrent-run chain forks); per-member prior resolution = latest determination row for (household_id, person_id) (per-member scoping is native — person_id NOT NULL). The explicit context pin is orchestrator-gated and validated (exists + same household) before signing.

  • Event: the medicaid.determined payload gains trigger additively.

  • Full adjust-vs-supersede enforcement remains #1133 (linked); the signed linkage this MR adds is what #1133 consumes.

D4. The CMD subsystem (canopy-medicaid) — #1506

Migration — medicaid_cmd_events reshaped honestly (pre-1.0):

ALTER TABLE medicaid_cmd_events
    ADD COLUMN application_id UUID,          -- the worker-chosen operative application
    ADD COLUMN state TEXT NOT NULL DEFAULT 'received' CHECK (state IN
        ('received','requested','completed','failed','legacy')),
    ADD COLUMN deadline_at TIMESTAMPTZ,      -- ingested_at + 10 days (PAMMS 2750)
    ADD COLUMN determination_request_id UUID, ADD COLUMN failure_code TEXT;
UPDATE medicaid_cmd_events SET state = 'legacy' WHERE processed_at IS NULL;  -- pre-subsystem rows, explicit

processed_at semantics corrected: stamped at state='completed' only (contract doc updated; now() from the settling statement, not tx start).

Ingest reshaped (CmdIngestRequest: gains application_id: Uuid, cmd_event_type: CmdEventType; drops submitted_by — spoofable prose): the guard moves to the live receiver machinery — contract.require_service_or_exchanged(…​)
EffectiveUser attribution (medicaid is receiver #1426; the future automated DCH feed is the service arm, the worker action is the exchanged arm). Handler: one tx — insert CMD row (state received, deadline_at) → flip to requested → stage determination.requested::Order { origin: ("medicaid-cmd", cmd_event_id), application_id, household_id, programs: ["medicaid"], as_of: effective_date, trigger: ChangeReport, requested_by: EffectiveUser attribution } → commit. Response: 202 + the row (OpenAPI regenerated — the current annotation doesn’t declare a success body).

Completion consumer: new group canopy-medicaid.cmd-settle bound to determination.completed; skip events whose origin.source != "medicaid-cmd" (the first domain consumer of this key; audit consumers unaffected). Settle: match origin.ref_id → CMD row; medicaid ∈ approved∪denied ⇒ state='completed', processed_at=now(), determination_request_id=request_id; else leave requested (the order substrate owns retry). Since #1511 a TERMINAL order failure no longer stalls silently: the canopy-medicaid.cmd-fail consumer marks the row state='failed' + failure_code off the origin-echoing determination.order_failed.

Visibility: GET /v1/cmd/events?household_id= (service/exchanged read) listing rows with state + deadline; GET /v1/cmd/escalations (#1511) feeding the worker-dashboard panel (approaching-deadline + failed rows, [medicaid].cmd_escalation_warning_days window); gauges canopy_medicaid_cmd_oldest_unresolved_age_seconds + depth + the (live since #1511) _terminal_failed.

D5. The BFF action (canopy-web) — #1507

The CMD form (templates/cases/_det_action_form.html:259-269) becomes the two-step fact action:

  1. Fields: person select (household members), cmd_event_type select (closed vocab — the contact_method pattern), effective date, the NEW disability-status select (what the person’s status becomes — for ssi_terminated the worker picks the post-termination value, e.g. disabled/none), notes.

  2. Handler: deny_unless_in_scope + ensure_household_member (IDOR) → typed UpdatePerson { disability_status } via persons (worker-attributed fact edit — the income-editor chain) → CMD ingest to medicaid under the exchanged bearer (the #1426 write-action pattern) with the case’s application_id from the page context. Partial failure: persons-update success + ingest failure ⇒ the error fragment tells the worker the fact saved but the change-report didn’t record — resubmit is idempotent-safe (a second identical persons update is a no-op COALESCE write; the second CMD ingest mints a new event id, so operators may see one duplicate order — acceptable: both settle on the same facts, the second supersedes with linkage; noted in the runbook).

  3. The redirect banner: notice=cmd-queued — "SSI change recorded — an automatic re-determination is running; this tab updates when it lands" (the click-Run-Determination copy dies for this action). The medicaid determination tab renders CMD rows (state/deadline) from D4’s list endpoint.

D6. What #575 does NOT claim (filed + linked, not absorbed) — in #1508

Gap Disposition

medicaid.determined has no notices route (NOA) — ALL medicaid determinations

File feat(notices) issue; link to #575 + the notices manifest gap register

Medicaid enrollment/supersession enforcement

#1133 (exists) — D3’s signed linkage is its input; /relate

10-day escalation workflow (worker task on alarm)

SHIPPED as #1511: terminal order failures map onto state='failed' via the origin-echoing determination.order_failed (staged atomically with the ledger’s failed_terminal flip); GET /v1/cmd/escalations + the worker-dashboard CMD-escalations panel surface approaching-deadline and failed rows; the clock pair is [medicaid].cmd_clock_days / cmd_escalation_warning_days jurisdiction data (ADR-003)

Compliance catalogue /cmd/ingest binding

Annotate honestly: automated re-determination covered; notice-of-action NOT — cite the two filed issues

Tests

Test Asserts

e2e cmd_ssi_termination_changes_outcome (medicaid tests/cmd_reeval_e2e_test.rs)

Seed household with disability_status="ssi_recipient"; baseline determination approves the SSI COA → the two-step change (persons update to disabled + ingest with effective_date) → poll: CMD row completed, NEW determination row with trigger='change-report', previous_determination_id = the baseline member row, effective_date = the pinned date, and the SSI COA now DENIED — the outcome flips, proving the rerun is not inert.

e2e cmd_order_is_idempotent_across_redelivery

The same Order event published twice → exactly one new determination set; the order row converges; the CMD row settles once.

e2e cmd_without_prior_application_fails_terminal_loudly

Unknown application_id → order failed_terminal, CMD row flips failed
failure_code (#1511; superseded the original stays-requested acceptance), the _terminal_failed gauge increments, no false processed_at. (Never built as an e2e; the #1511 pins cover the flip at the substrate + consumer + feed layers.)

integration: order sweep retry (eligibility)

Transient program failure → failed_retryable + future next_attempt_at; the sweep re-stages; success on attempt 2; attempt accounting correct.

unit/property

DeterminationRequestedV1 both-arm roundtrips (proptest — parser rule); the order idempotency-key KAT; DisabilityStatus/CmdEventType roundtrips + unknown-value rejection at the persons write path; the per-member prior-resolution query.

existing

The bulk e2e/AC6 suite stays green (the Cohort arm is semantically untouched); cmd_ingest_persists_with_app_supplied_v7_id is REWRITTEN for the new contract (202, typed event type, application_id, no submitted_by) — a genuine recorded contract change, not a weakening.

Delivery

Each MR: fmt/clippy/targeted tests per commit, the full battery at push, merge on green per the standing authorization, close its issue with SHAs, update this plan’s Status table. Rollout ordering (runbook section in MR-2): broker ACLs → deploy eligibility (consumer) → deploy medicaid producer + web.

MR Branch Content

1

feat/575-vocabularies

D1 — #1503

2

feat/575-order-substrate

D2 — #1504 (bulk consumer adapted same-diff)

3

feat/575-medicaid-plumbing

D3 — #1505

4

fix/575-cmd-subsystem

D4 — #1506 (OpenAPI regen)

5

feat/575-bff-fact-action

D5 — #1507

6

test/575-e2e-docs

Tests + ADR-002 Amendment (Order arm) + CHANGELOG
Antora + compliance annotation + D6 filings — #1508; #575 closes

Verification

  • The flip test is the acceptance proof: a seeded SSI household’s COA outcome CHANGES through the pipeline with signed trigger + linkage + pinned date.

  • Battery per MR; mq-topology, compliance, quality-budgets, OpenAPI drift gates.

  • Manual devstack: submit the CMD form; watch the tab flip received→requested→completed without any Run Determination click; check the oldest-age gauge and a forced-failure alarm path.

Edit this page · default