Plan: Concurrency-safe, recoverable applicant-draft finalization

On this page

Status

Step Description Status

MR0

This plan .adoc + ADR-038 + nav/arch + consumer order-independence check

Done (2026-07-13) — !830

MR1a

Single-source the outbox schema (canonical in canopy-mq + generator + parity gate) + first-class event-hold + ADR-039

Done (2026-07-13) — !831

MR1

canopy-persons: transactional finalize receipt + generation gate + held-event staging

Done (2026-07-13) — !832

MR2

canopy-persons: finalize-operation register/release/cancel/get endpoints + shred compensation

Done (2026-07-13) — !833

MR3

canopy-persons-client: thread finalize step identity + shared header consts

Done (2026-07-13) — !834

MR4

canopy-applications: finalize_operations saga store + shared lease guard + app-builder

Done (2026-07-13) — !835

MR5

canopy-applications: rewrite finalize_draft as a recoverable saga (feature-flagged)

Done (2026-07-13) — !836

MR6

canopy-applications: lease/compensation-aware draft reaper

Done (2026-07-14) — !837

MR7

canopy-applications: finalize reconciler (compensate + release-retry + pruner)

Done (2026-07-14) — !838

MR8

Cross-service finalize acceptance suite + activate the flag

Done (2026-07-14) — !839

MR9

cargo xtask sweep-finalize-orphans one-shot existing-orphan remediation

Done (2026-07-14) — !840

Epic: &71
Issues: #1005 (umbrella) · #1046 (MR0) · #1057 (MR1a) · #1047 (MR1) · #1048 (MR2) · #1049 (MR3) · #1050 (MR4) · #1051 (MR5) · #1052 (MR6) · #1053 (MR7) · #1054 (MR8) · #1055 (MR9)
Branch: feature/{n}-… per child
ADRs: ADR-038 · ADR-039 (outbox single-source + hold)

Context

finalize_draft (services/canopy-applications/src/api/mod.rs) creates the applicant’s person → household → membership → income/asset/expense graph in canopy-persons through ~6+ separate HTTP calls before it opens the local transaction, locks the draft, inserts the applications row (the reserved draft id as PK), stages outbox events, deletes the draft, and commits. No idempotency ties the persons writes to the reserved application id.

ADR-026 §5 (materialize-at-finalize) guarantees the intra-applications applications-INSERT + application_drafts-DELETE are one transaction, and §6 (sliding reaper) serialises the reaper with finalize on the draft row. But ADR-026 §5 explicitly scopes the cross-service persons writes out ("that cross-service ordering is the pre-existing orchestration concern, not introduced by this ADR"). This plan closes exactly that gap.

Failure modes today — all leave orphaned PII in canopy-persons with no owning application (the ADR-025 failure mode, now under concurrency):

  • crash / 5xx mid-graph;

  • the reaper wins between the persons writes and the draft lock (late lock-miss 404s);

  • a losing concurrent racer (both build the graph; the loser 404s with its graph stranded);

  • a double-submit / retry re-creates the graph.

This is a data-integrity + PII-hygiene defect in a federal eligibility system (GitLab #1005, priority::high, type::bug).

Why the transactional-receipt design (not the generic middleware). An earlier draft rode the generic idempotency middleware (crates/canopy-api/src/idempotency.rs) for cross-service replay-safety. That is wrong on two counts the middleware documents about itself: it is "exactly-once happy path / at-least-once on crash" — the domain transaction commits before the response-cache write, and it re-executes after a 24h TTL — and it caches plaintext PII (raw Person bodies with names + DOB) outside crypto-shred. This plan instead makes each persons write idempotent at the persons layer via a transactional receipt, with an operation generation, a keyed request digest, a row-locking claim, a durable compensating state, crypto-shred (not delete) compensation, held→released events, and applications-only authz.

Acceptance criteria (from #1005)

Criterion

(a)

concurrent finalize → one application + one graph

(b)

retry after a lost response returns the original, no new writes

(c)

failure after each step is recoverable by retry or bounded compensation

(d)

reaping cannot race a live finalize

(e)

stuck ops are observable + reconcilable

(f)

remote writes carry stable operation identity + are idempotent

(g)

no lock/transaction held across a network call

(h)

integration tests: concurrent, timeout-after-commit, mid-step failure, restart, reaper

(i)

identify existing orphans without exposing PII

Scope

In scope:

  • A persons-side transactional idempotency receipt + operation-generation gate + held outbox events (MR1).

  • A persons finalize control surface (register / release / cancel / get) with crypto-shred compensation + shared-graph quarantine, gated to canopy-applications (MR2).

  • A typed persons-client for that surface + the shared StepKey and header constants (MR3).

  • An applications-side durable saga store with a linearizable draft-row-locking claim (MR4).

  • The finalize_draft rewrite behind a feature flag (MR5).

  • A lease/compensation-aware reaper (MR6) and a finalize reconciler + pruner (MR7).

  • A cross-service acceptance suite that flips the flag on (MR8).

  • A one-shot existing-orphan remediation tool (MR9).

Out of scope:

  • Two-phase commit / distributed transactions (barred by ADR-001; the saga + outbox is the sanctioned pattern).

  • Cross-service event ordering guarantees (see Cross-service event ordering; the hold guarantees only that downstream never sees a compensated finalize).

  • Any change to the applicant portal’s client-side crypto or the draft lifecycle before finalize (ADR-026 §§1–4 unchanged).

Design

Corrected foundation (thesis)

Each persons write is made idempotent, atomic, and provenance-tagged at the persons layer by a transactional receipt keyed on (operation_id = reserved app id, generation, step_key); concurrency and recovery are governed by a linearizable, draft-row-locking claim on a durable finalize_operations saga record; a partial graph is undone by crypto-shred + event-drop, never a hard delete; and downstream sees events only for a committed finalize. No reliance on the generic middleware; no PII leaves persons in any generic cache.

today is pinned per (op, generation) so re-runs build byte-identical bodies; received_at is pinned and threaded into create_application_with_id (today None); a keyed digest of the full typed request is pinned per (op, generation) and re-validated on resume.

Data model

canopy-applications — saga record + local step cache

-- state is TEXT + CHECK, not a Postgres enum: canopy-applications uses
-- TEXT+CHECK everywhere (a new state is a forward-only CHECK swap, ADR-016).
CREATE TABLE finalize_operations (
    application_id   UUID PRIMARY KEY,            -- reserved id (= application_drafts PK)
    generation       INT  NOT NULL DEFAULT 1,     -- ++ on each aborted re-submit (new filing)
    state            TEXT NOT NULL DEFAULT 'in_progress'
        CHECK (state IN ('in_progress','compensating','completed','aborted')),
    lease_holder     UUID,                        -- per-attempt claim_id fence
    lease_expires_at TIMESTAMPTZ,
    basis_date       DATE        NOT NULL,        -- pinned `today`
    received_at      TIMESTAMPTZ NOT NULL,        -- pinned; threaded into the app row
    request_digest   BYTEA       NOT NULL,        -- keyed HMAC of the canonical full FinalizeRequest
    household_id     UUID,                        -- set at COMPLETED (reconstructs FinalizeResponse)
    events_released  BOOLEAN NOT NULL DEFAULT false, -- persons release confirmed post-commit
    attempts         INT NOT NULL DEFAULT 1 CHECK (attempts > 0),
    created_at       TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    CHECK (state <> 'in_progress'  OR (lease_holder IS NOT NULL AND lease_expires_at IS NOT NULL)),
    CHECK (state <> 'completed'    OR household_id IS NOT NULL),
    CHECK (state NOT IN ('completed','aborted') OR lease_holder IS NULL)
);
CREATE INDEX finalize_operations_stuck_idx ON finalize_operations (state, lease_expires_at)
    WHERE state IN ('in_progress','compensating');
CREATE INDEX finalize_operations_unreleased_idx ON finalize_operations (state)
    WHERE state='completed' AND events_released=false;

-- Local skip-cache (perf; the persons receipt is the correctness source of truth).
CREATE TABLE finalize_steps (
    application_id UUID NOT NULL REFERENCES finalize_operations(application_id) ON DELETE CASCADE,
    generation     INT  NOT NULL,
    step_key       TEXT NOT NULL,
    remote_kind    TEXT NOT NULL,     -- RemoteEntityKind enum
    remote_id      UUID NOT NULL,     -- the stored stable id (person_id / household_id / fact_id)
    recorded_at    TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    PRIMARY KEY (application_id, generation, step_key)
);

canopy-persons — receipt + generation gate + event hold

-- Generation gate: every finalize-tagged write checks state='active' IN ITS TX (closes the late-write race).
CREATE TABLE finalize_operation_generations (
    operation_id UUID NOT NULL,
    generation   INT  NOT NULL,
    state        TEXT NOT NULL DEFAULT 'active' CHECK (state IN ('active','cancelled')),
    created_at   TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    PRIMARY KEY (operation_id, generation)
);
-- Idempotency receipt: written in the SAME tx as the entity + outbox event.
CREATE TABLE finalize_receipts (
    operation_id UUID NOT NULL,
    generation   INT  NOT NULL,
    step_key     TEXT NOT NULL,
    entity_kind  TEXT NOT NULL,
    stable_id    UUID NOT NULL,       -- person_id / household_id / fact_id (survives corrections)
    created_at   TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    PRIMARY KEY (operation_id, generation, step_key),
    FOREIGN KEY (operation_id, generation)
        REFERENCES finalize_operation_generations(operation_id, generation)
);
-- Event hold: a first-class outbox capability, single-sourced in canopy-mq and
-- delivered by MR1a (#1057, ADR-039) — NOT a persons-local column. The canonical
-- outbox migration adds the hold columns to every service's event_outbox; the
-- shared drainer skips held rows; canopy-mq exposes publish_tx_held/release_held/
-- drop_held. MR1 consumes this to stage persons finalize events held.
ALTER TABLE event_outbox ADD COLUMN hold_operation_id UUID, ADD COLUMN hold_generation INT;
-- drainer claim CTE gains: AND hold_operation_id IS NULL   (additive: NULL for all non-finalize rows)
CREATE INDEX event_outbox_held_idx ON event_outbox (hold_operation_id, hold_generation)
    WHERE hold_operation_id IS NOT NULL AND published_at IS NULL;

Step / ordinal enumeration (StepKey newtype)

StepKey (person(0) / household() / member(i) / income(j) / asset(k) / expense(m), with Display / FromStr) is both the step_key value and the receipt key. Order (per finalize_draft): person:0 = applicant → household → member:0 = applicant self → per member i (0-based in req.household_members): person:{i+1}, member:{i+1} → income:{j} / asset:{k} / expense:{m} = position in the flat request vec (each entry carries its own person_index, resolved from the local step cache / receipt).

Saga

State machine

State Entry → completed → recover → compensate

(none)

claim locks draft + op, pins basis/received/digest, gen=1, lease

in_progress

run steps (each: check local cache → else call persons with (op,gen,step) → persons upserts receipt in-tx → record stable id locally → heartbeat); then the network-free final tx

final tx (lease-fenced) inserts app/programs/outbox, deletes draft, mark_completed; then persons release(op,gen) un-holds events + sets events_released

lease expires → a retry re-claims (steal), skips receipted steps, finishes

reconciler: in_progress + lease-expired > grace → compensating

compensating

fenced; not client-reclaimable

reconciler: persons cancel(op,gen) (mark gen cancelled → drop held events → per entity: shred+deactivate if exclusive under lock, else terminal quarantine, left intact) → mark_aborted on cancel success even with quarantined entities

completed

terminal; household_id stored

claim returns the stored FinalizeResponse; a completed && !events_released op is retried for release by the reconciler

aborted

terminal; graph shredded, gen cancelled

a re-submit (same app id) re-claims → generation++, fresh basis/received/digest, clears local steps → in_progress

No lock/tx spans a network call (criterion g): the lease is a heartbeat’d row value; the only transaction is the network-free final commit.

Linearizable claim (one short local tx — a row lock, not a CTE)

claim_or_resume(pool, app_id, today, received_at, digest, cfg) → ClaimOutcome:

BEGIN;
  SELECT 1 FROM application_drafts
    WHERE application_id=$app AND expires_at>clock_timestamp() FOR UPDATE;
     -- absent ⇒ ROLLBACK, Err(DraftGone)   (serializes with the reaper's FOR UPDATE SKIP LOCKED)
  SELECT * FROM finalize_operations WHERE application_id=$app FOR UPDATE;   -- may be absent
  branch on the locked row:
    absent                        → INSERT gen=1, lease; COMMIT → Won{gen:1, basis:today, received}
    completed                     → COMMIT → Completed{household_id}
    compensating                  → COMMIT → InProgressElsewhere{retry_after}   (never reclaim)
    in_progress & lease live      → COMMIT → InProgressElsewhere{retry_after from lease_expires_at}
    in_progress & lease expired   → UPDATE lease (keep gen/basis/received/digest); COMMIT → Won{steal}
    aborted                       → UPDATE gen=gen+1, state=in_progress, basis=today, received, digest,
                                    lease; DELETE finalize_steps WHERE application_id=$app; COMMIT
                                    → Won{gen++, fresh basis}
COMMIT;

Row-locking (not a snapshot CTE) makes it linearizable — concurrent claimers serialise on the op row; retry_after is read from the locked lease_expires_at. On a Won{steal} the saga re-validates the incoming request against the pinned request_digest (mismatch ⇒ 409 — an edited request mid-saga); on Won{gen++} the digest is freshly pinned.

enum ClaimOutcome { Won { generation, basis_date, received_at, claim_id }, Completed { household_id }, InProgressElsewhere { retry_after } }.

Completion + release + 23505

The final tx re-takes lock_draft_for_update (FOR UPDATE) before insert/delete (preserving the ADR-026 §6 reaper serialisation). mark_completed(household_id) is in that same tx as app/program/outbox/draft-delete, lease-fenced (WHERE lease_holder=$claim_id; 0 rows ⇒ ROLLBACK — lease lost). Because it is one tx, another tx cannot observe the applications PK before completed commits.

On a genuine 23505 (is_unique_violation, promote to pub(crate)) PG has aborted the tx ⇒ ROLL BACK, then read the authoritative applications.household_id (not finalize_steps) and return FinalizeResponse.

Draft-gone-on-resume: a lock_draft_for_update miss in the final tx consults finalize_operations — completed ⇒ return the stored response, else 404.

After commit, call persons release(op,gen); on failure leave events_released=false for the reconciler to retry (idempotent).

Cross-service event ordering

application.submitted (applications outbox, drains on commit) and the released persons graph events have no guaranteed order — as for all cross-service events on the bus. The hold guarantees only that downstream never sees a compensated finalize’s events; it does not order across services. Because the persons rows are committed synchronously (via HTTP) before either event drains, a consumer can always resolve a referenced entity by a synchronous GET even if it has not yet seen that entity’s event. ADR-038 records this contract; MR0 verifies the finalize-graph consumers (renewals / medicaid / security) are order-independent (upsert / tolerate app-submitted-before-graph); a consumer that isn’t is escalated, not silently relied upon.

Persons-side contracts (canopy-persons, applications-only authz)

Every finalize endpoint is gated by require_service_caller()? then claims.service_id() == Some("canopy-applications") — the role-derived check after the coarse one, because service_id() falls back to azp, so gating on it alone would admit an azp-only OIDC client (no service: role) that require_service_caller rejects — a weaker trust class on a shred-capable endpoint. This is the first specific-caller allow-list. MR9’s steward path uses require_data_steward().

  • Create/claim (MR1): each accepts an optional FinalizeStep { operation_id, generation, step_key }. When present, the handler, in its existing tx and in this lock order — gen-gate FOR SHARE before lock_fact (consistent order, no deadlock):

    1. SELECT state FROM finalize_operation_generations WHERE (op,gen) FOR SHARE — absent / cancelled ⇒ 409/410 (refuse a write to a dead generation; this FOR SHARE conflicts with cancel’s mark-cancelled `UPDATE, so a stale writer and a cancel serialise — the stale write either commits its receipt before the cancel’s post-mark re-inventory sees it, or is refused);

    2. INSERT finalize_receipts (op,gen,step,kind,stable_id) ON CONFLICT (op,gen,step) DO NOTHING — if it conflicted, SELECT stable_id and return the existing entity (idempotent replay); else insert the entity + every outbox event the path stages held (hold_operation_id/hold_generation — e.g. the income path stages both income.claimed and persons.income_changed; miss none) + the receipt, and commit.

      Header/body coherence (op/gen/step ↔ entity-kind) is validated; malformed ⇒ 400.

  • POST /v1/internal/finalize-operations/{op}/{gen}/register (MR2): upsert the generation active (the claim calls this before writes).

  • …/release: un-hold all (op,gen) events (hold_operation_id=NULL) in one tx (idempotent).

  • …/cancel (READ COMMITTED, idempotent + resumable): (1) UPDATE finalize_operation_generations SET state='cancelled' WHERE (op,gen); (2) re-inventory finalize_receipts (op,gen) after the mark, so a stale in-flight write that just committed its receipt is caught; (3) DELETE held-undrained (op,gen) outbox events — every finalize event type (incl. persons.income_changed, the second event the income path stages); (4) per receipted entity, in one tx holding lock_fact(fact_id) (facts) / a person_id advisory lock (persons), check exclusivity under the lock — a fact with any later non-finalize version, or a person in another active household, is quarantined (recorded for the steward, left intact, not shred); an exclusively-owned entity is shred by the inventoried dek_id (shred_with_dek_id, never (subject_kind, subject_id)) + deactivated. Quarantine is terminal — it never blocks completion; cancel returns the quarantined-id list and the reconciler still reaches aborted.

  • GET …/{op}: PII-free counts / ids for the reconciler.

(/v1/internal/* mounts under the service /v1 router — persons' helper accepts only /v1/….)

Request digest

request_digest = keyed HMAC (a server secret from settings) over a domain-separated canonical serialisation of the full typed FinalizeRequest (applicant, members, income/assets/expenses, programs_requested, contact, consent, screening inputs). Keyed ⇒ not offline-guessable from PII. Pinned per (op, generation); re-validated on every resume within a generation (mismatch ⇒ 409); re-pinned on gen++. Canonicalisation is deterministic — amount strings serialised as-sent, explicit omitted-vs-null, stable field order — so a legitimate portal resend recomputes the identical digest (a non-deterministic canonicalisation would false-409 a valid resume). Closes the "retry with an edited/reordered request skips steps from attempt A, runs the rest from attempt B" hole.

Reaper + reconciler

  • Reaper (MR6): add AND NOT EXISTS (SELECT 1 FROM finalize_operations f WHERE f.application_id=application_drafts.application_id AND f.state IN ('in_progress','compensating')) to reap_expired_drafts. Draft-row FOR UPDATE SKIP LOCKED already serialises with the claim’s draft FOR UPDATE. Lifecycle: a non-terminal op is never reaped; an aborted / absent op’s draft is reaped only at its own 30-day expiry. Also align draft_exists to check expires_at like get_draft (or have the claim use the expiry-checking lock).

  • Reconciler (MR7): a leader-elected canopy-applications.finalize-reconciler tick (mirror the reaper / recovery-pruner + assert_lock_election_behavior). Per list_stuck op: (1) in_progress + lease-expired > grace → claim_for_compensation (atomic in_progress→compensating, fenced — clients can’t reclaim); (2) persons cancel(op,gen); (3) mark_aborted (reached even with quarantined entities). A failed step leaves compensating for the next tick (idempotent resume). Separately: completed && !events_released → retry persons release until confirmed. Terminal-row pruner: WHERE (state='aborted' OR (state='completed' AND events_released=true)) AND updated_at < retention — it must never prune a completed && !events_released op (that would strand persons-held events → the drainer never publishes them → downstream never learns the finalized household exists); alarm on completed && !events_released older than a threshold. PII-free metrics / logs (ids / counts only).

Existing-orphan reconciliation (MR9)

A high-confidence orphan = a canopy-persons household whose id is not in SELECT household_id FROM applications (anti-join) AND whose self-membership carries origin='finalize' (household_member_versions.origin — a persisted, readable provenance signal finalize always writes). A person/household created but with no membership (crash before self-membership) is ambiguous → quarantine for a data steward, never auto-shred.

Live-op exclusion (must): skip any household covered by a finalize_receipt at all — an in_progress op has no applications row yet and an origin='finalize' self-membership, so the anti-join + provenance alone cannot distinguish it from an orphan. A truly pre-fix orphan has no receipt (a live op always writes one), so a no-receipt candidate cannot be in-flight; MR9 thus keys off saga state, not graph inference alone. (Implemented as the conservative superset of the non-terminal/unreleased condition: receipt-covered ⇒ saga-era ⇒ the reconciler owns the lifecycle, terminal or not — the sweep never touches it. Enforced twice: the discovery anti-join skips receipt-covered households, and the endpoint re-checks zero-receipts across the whole inventoried graph in the compensation transaction.)

The one-shot cargo xtask sweep-finalize-orphans (a narrower finalize-specific sibling of ADR-025’s unbuilt seed sweep-orphans FU) builds an immutable reviewed manifest (digest-sealed candidate list) during a quiescence/maintenance window, re-validates each candidate’s non-reference immediately before acting, is resumable (per-candidate results sidecar), dry-run by default, PII-free output. --apply compensates via a NEW data_steward-gated persons endpoint (POST /v1/households/{id}/compensate-finalize-orphan) rather than the MR2 cancel surface — cancel keys off finalize_receipts, which pre-saga orphans by definition lack. The endpoint re-checks the provenance + zero-receipt guards in its own transaction under the household advisory lock and reuses `cancel’s shred-or-quarantine machinery per entity (shred, not delete; a person sharing another finalize household quarantines). Closes the destructive TOCTOU.

Key decisions

# Decision Rationale

Foundation

Persons-side transactional receipt. finalize_receipts(operation_id, generation, step_key) UNIQUE, written in the SAME tx as the entity + its outbox event; a repeat returns the stored stable id. NOT the generic middleware.

True exactly-once at the owning layer; no PII in any generic cache; atomic with entity+event (no crash-gap); provenance + generation built in.

Filing date

New filing on an aborted re-submit: fresh received_at/valid_from under a new generation.

The aborted attempt created no application (graph fully compensated), so the successful re-submit is the filing.

Idempotency key

Receipt keys on caller-supplied (operation_id, generation, step_key), returns the stored stable id on conflict.

Entity ids are minted mid-handler, so the server-minted id can’t be the retry key; the receipt carries the caller correlation + returns the persisted id. Stable id = fact_id (survives corrections), not version_id.

Events

Hold → release/drop: finalize persons events are staged held; released only when the application commits; dropped on compensation.

Downstream never sees a partial/compensated finalize → no reversal-event blast radius across every consumer.

Compensation

Crypto-shred (ADR-036) + deactivate, NOT hard delete — under a per-fact lock, scoped to the dek_id captured at inventory (a new shred_with_dek_id variant), never (subject_kind, subject_id).

The redaction_keys one-way trigger rejects DELETE/TRUNCATE, version rows FK-block deletes, and shred_with(subject_kind, fact_id) is lock-free + matches every live DEK for the subject → would silently destroy a later legit correction.

Shared-graph safety

Per entity, under lock_fact(fact_id) / a person_id advisory lock, check exclusivity; exclusive ⇒ shred+deactivate; shared/contaminated ⇒ terminal quarantine (left intact, recorded for a steward — never blocks the op).

The per-fact DEK is shared across all versions + minted lock-free, and person↔household is many-to-many; the guard must hold the append lock (TOCTOU), and quarantine must terminalise so a legitimately-shared entity can’t wedge the op.

Tunables (FinalizeSagaConfig, validated builder — single source)

Knob Default Constraint

op lease

30s

> heartbeat

heartbeat

10s

< lease

reconciler grace

~1h

persons-request-timeout (30s) < grace < 24h

completed-op retention

≥ a defined retry SLA (e.g. 30d)

pruned rows reconstructable from applications

Steps

Step MR0: Plan + ADR-038 + nav/arch + consumer check

Files: docs/modules/ROOT/pages/plans/concurrency-safe-applicant-finalization.adoc, docs/modules/ROOT/pages/adrs/adr-038-concurrency-safe-applicant-finalization.adoc, docs/modules/ROOT/nav.adoc, docs/modules/ROOT/pages/architecture.adoc, CHANGELOG.adoc

Commit this plan + ADR-038 (amends ADR-026 §5/§6; builds on ADR-025; uses ADR-036 shred; reaffirms ADR-001/018/019). Add the nav + arch-index entries. Verify — read-only — that the finalize-graph consumers (renewals / medicaid / security subscribers) are order-independent per Cross-service event ordering; escalate (new issue) any order-dependent consumer rather than relying on ordering.

Step MR1a: single-source the outbox schema + first-class event-hold

Files: crates/canopy-mq/outbox-migrations/, crates/canopy-mq/src/{publisher.rs,outbox_drainer.rs,lib.rs}, xtask/src/cmd/outbox_migrations.rs, services//migrations/event_outbox, docs/modules/ROOT/pages/adrs/adr-039-*.adoc

The 18 per-service event_outbox migrations were byte-identical hand-copies with no source + no drift gate. Make crates/canopy-mq/outbox-migrations/ the canonical source; add cargo xtask outbox-migrations --check|--write (parity gate + generator, wired into validate); fold in the hold migration (hold_operation_id/hold_generation + partial index); add the drainer predicate + Publisher::publish_tx_held + release_held + drop_held + EventHold. Introduces ADR-039 (amends ADR-018). Because there are no deployments, the schema is restructured freely. MR1 consumes the hold API.

Step MR1: persons receipt + generation gate + held-event staging

Files: services/canopy-persons/migrations/, services/canopy-persons/src/{store,api}/

Two migrations (finalize_operation_generations, finalize_receipts). The 6 create/claim paths accept an optional finalize step via headers (claim DTOs are deny_unknown_fields); gen-gate FOR SHARE (absent/cancelled ⇒ 409) → receipt upsert (return-stored-id on conflict) → entity + events staged held (MR1a’s publish_tx_held) + receipt, all in the existing tx. Property test the receipt-key determinism; the income path stages both income.claimed and persons.income_changed held.

Step MR2: persons finalize control surface + compensation

Files: services/canopy-persons/src/api/, services/canopy-persons/src/store/redaction.rs, services/canopy-persons/src/store/, contracts crate

register / release / cancel / GET internal endpoints (applications-only authz). cancel = mark-cancelled → re-inventory → drop held events → per-entity shred-if-exclusive-under-lock else quarantine. New shred_with_dek_id scoped to a captured dek_id. DTOs + roundtrip.rs.

Step MR3: persons-client + shared consts

Files: crates/canopy-persons-client/*, crates/canopy-auth/src/client_ext.rs

Thread FinalizeStep through every create/claim post(); add register/release/cancel/get; StepKey newtype + round-trip; move the finalize header/id consts into canopy-auth::client_ext (shared, no duplicated literals).

Step MR4: saga store + lease guard + app-builder

Files: services/canopy-applications/migrations/, services/canopy-applications/src/store/finalize_ops.rs, services/canopy-applications/src/config.rs, services/canopy-applications/src/lib.rs, crates/canopy-db/

Two migrations (finalize_operations, finalize_steps). store/finalize_ops.rs: claim_or_resume (the row-locking tx above) + heartbeat/record_step/load_progress/mark_completed/claim_for_compensation/mark_aborted/list_stuck, all lease-fenced. FinalizeSagaConfig validated builder. Extract the shared lease-guard helper to canopy-db. Add a lib.rs app-builder (unblocks MR8).

Step MR5: rewrite finalize_draft as the saga (feature-flagged)

Files: services/canopy-applications/src/api/mod.rs, services/canopy-applications/src/store/drafts.rs

Behind a feature flag: claim → digest pin/validate → register(op,gen) → pinned steps (skip via the local cache) → final tx (lease-fenced, mark_completed) → release → response. Helpers ≤ ~40 lines; drop finalize_draft below the 100-LOC budget (offset B2 in-MR). Handle draft-gone-on-resume + 23505-from-authoritative-app; InProgressElsewhere → 503 + Retry-After. Drop the stale ADR-026 §5 comment.

Step MR6: lease/compensation-aware reaper

Files: services/canopy-applications/src/store/drafts.rs

Add the non-terminal-op guard to reap_expired_drafts; align draft_exists/expiry with get_draft.

Step MR7: finalize reconciler + pruner

Files: services/canopy-applications/src/* (reconciler task + app-builder registration)

Leader-elected tick: compensate lease-expired in_progress ops; retry release for completed && !events_released; prune terminal rows on retention (never a completed && !events_released op); PII-free observability.

Step MR8: cross-service acceptance suite + flip the flag

Files: services/canopy-applications/tests/*

The failure matrix + two-connection concurrency/reaper/compensation-vs-stale-writer barriers via the MR4 app-builder + a hand-built fault handler over mock::spawn_router; DB-time backdating for lease/grace. Flip the feature flag on.

Step MR9: existing-orphan remediation

Files: xtask/src/{cmd/sweep_finalize_orphans,psql}.rs, services/canopy-persons/src/{api,store}/*, crates/canopy-contracts-persons/src/{finalize,paths}.rs

cargo xtask sweep-finalize-orphans: digest-sealed manifest (quiescence) + per-candidate revalidation + live-op exclusion + resumable sidecar + dry-run default + PII-free. --apply drives the new data_steward-gated persons endpoint POST /v1/households/{id}/compensate-finalize-orphan (pre-saga orphans have no receipts, so the MR2 cancel surface cannot address them); the endpoint re-validates provenance + zero-receipts in-tx and reuses the MR2 shred-or-quarantine machinery. Shared docker exec psql helpers extracted to xtask/src/psql.rs (from seed-verify).

Files Touched

File Change

services/canopy-persons/migrations/*

finalize_operation_generations + finalize_receipts (MR1)

services/canopy-persons/src/{store,api}/*

receipt upsert, gen-gate, held events, control surface, shred/quarantine (MR1/MR2); steward-gated orphan compensation endpoint (MR9)

crates/canopy-mq/{outbox-migrations,src}/*, xtask/…​/outbox_migrations.rs

canonical outbox schema + generator/parity gate + hold columns + drainer WHERE + publish_tx_held/release_held/drop_held (MR1a)

services//migrations/*event_outbox

regenerated from canonical (hold columns, all 18 services) (MR1a)

crates/canopy-persons-client/*

FinalizeStep threading + register/release/cancel/get + StepKey (MR3)

crates/canopy-auth/src/client_ext.rs

shared finalize header/id consts (MR3)

services/canopy-applications/migrations/*

finalize_operations + finalize_steps (MR4)

services/canopy-applications/src/{store,config,lib,api}/*

saga store, config, app-builder, finalize_draft rewrite, reaper, reconciler (MR4–MR7)

crates/canopy-db/*

shared lease-guard (MR4)

xtask/src/*

sweep-finalize-orphans + shared psql helpers extracted from seed-verify (MR9)

docs/modules/ROOT/**, CHANGELOG.adoc

ADR-038, plan, per-service data-model/api pages, services catalog, runbooks (all MRs)

Verification

Per MR: cargo fmt --all; cargo clippy -p <crate> --all-targets --profile test — -D warnings; cargo xtask quality-budgets --fail-on-regression; cargo xtask plan-lint; cargo xtask check-docs; cargo deny check. Migrations → cargo xtask dev refresh before integration; set -a; source .ports.env; set +a; cargo nextest run -p <svc> --profile integration; regenerate + verify OpenAPI snapshots (cargo xtask api-docs --update). The pre-push battery is the merge gate.

Testing focus:

  • Property (proptest): the request digest + per-(op,gen,step) receipt key are deterministic across attempt orderings; StepKey Display/FromStr round-trip; the MR2 DTO round-trip.

  • Failure matrix (per step): pre-commit 5xx, timeout-before-commit, commit-then-lost-response, response-then-crash-before-record, record-then-crash, lease-steal-mid-call, final-commit-then-lost-response.

  • Concurrency (two real connections, deterministic barriers): first-claim race; reaper-vs-claim; compensation-vs-stale-writer (assert the gen-cancel refuses the stale write); DELETE/mark-aborted failure resume; 23505. Criterion (g): a blocked remote call while another connection acquires the draft row (FOR UPDATE NOWAIT).

  • Restart = fresh service state against the same DBs. Lease/grace expiry = backdated DB timestamps (paused Tokio can’t move clock_timestamp()).

  • Mock persons models the handler-commit/receipt separation (a real failpoint at the entity-commit boundary), not replay-the-first-response; cross-service dedup also covered on devstack.

Documentation Updates

  • adr-038-concurrency-safe-applicant-finalization.adoc + nav + architecture.adoc ADR index (MR0).

  • adr-039-single-source-outbox-schema-and-event-hold.adoc + nav + arch index (MR1a).

  • data-models/canopy-persons.adoc, data-models/canopy-applications.adoc (tables) — per implementing MR.

  • api/canopy-persons.adoc (endpoints) + cargo xtask api-docs --update (MR2).

  • services.adoc — routes / tables / schedulers (reconciler, reaper) / xtask command.

  • shared-crates.adoc — canopy-mq outbox hold, canopy-db lease-guard, canopy-persons-client surface.

  • configuration-reference.adoc + default YAML — FinalizeSagaConfig, the feature flag.

  • Operator runbooks — reconciler + sweep-finalize-orphans.

  • CHANGELOG.adoc == Unreleased — per MR.

  • roadmap.adoc — on plan completion (final MR).

Edit this page · default