Finalize Reconciler: stuck applicant-finalize operations

On this page

Table of Contents

Overview
Log lines that matter
- ALARM: unreleased events past grace
- Quarantined entities
Tunables
Relationship to the orphan sweep

Overview

The canopy-applications finalize reconciler (ADR-038, epic &71 MR7) is a leader-elected background tick (every 5 minutes, advisory lock canopy-applications.finalize-reconciler) that drives every stuck finalize_operations row to a terminal state. It is the recovery half of the finalize saga: the saga handler makes each attempt safe; the reconciler guarantees an attempt that died is eventually undone or finished.

Per tick it works one bounded batch (100 ops, oldest first — overflow rolls to the next tick and is logged) in three lanes:

Lane Trigger Action

Lane	Trigger	Action
Compensate	`in_progress` with the lease lapsed longer than `finalize_reconciler_grace_secs` (default 1 h)	Atomically move to `compensating` (a client can never reclaim that state), call canopy-persons `cancel(op, gen)` — mark the generation cancelled, drop the still-held outbox events, crypto-shred exclusively-finalize entities, quarantine shared ones — then terminalize `aborted`. The applicant’s next submit starts a fresh generation (a new filing).
Release-retry	`completed` with `events_released = false`	Retry canopy-persons `release(op, gen)` until confirmed, then set `events_released`. Downstream (renewals / medicaid / security) cannot see the finalize’s persons events until this lands.
Prune + alarm	terminal rows older than `finalize_completed_retention_days` (default 30 d)	Delete `aborted` and `completed && events_released` rows. A `completed && !events_released` row is never pruned — pruning it would strand the held persons events forever — and instead raises the alarm below once older than grace.

Compensate

in_progress with the lease lapsed longer than finalize_reconciler_grace_secs (default 1 h)

Atomically move to compensating (a client can never reclaim that state), call canopy-persons cancel(op, gen) — mark the generation cancelled, drop the still-held outbox events, crypto-shred exclusively-finalize entities, quarantine shared ones — then terminalize aborted. The applicant’s next submit starts a fresh generation (a new filing).

Release-retry

completed with events_released = false

Retry canopy-persons release(op, gen) until confirmed, then set events_released. Downstream (renewals / medicaid / security) cannot see the finalize’s persons events until this lands.

Prune + alarm

terminal rows older than finalize_completed_retention_days (default 30 d)

Delete aborted and completed && events_released rows. A completed && !events_released row is never pruned — pruning it would strand the held persons events forever — and instead raises the alarm below once older than grace.

Every sub-step is idempotent: a failed persons call or crash leaves the op in its current durable state and the next tick resumes it. A quarantined entity never blocks the op reaching aborted.

Log lines that matter

Every field the reconciler itself emits is PII-free (operation ids, generations, counts). The error fields forward the persons client’s error text — the status plus a bounded 256-byte body excerpt (#1063); the full non-2xx body surfaces at debug level inside canopy-persons-client (do not enable that level for the client in production). Two warn! lines are operator signals:

ALARM: unreleased events past grace

completed finalize operations with UNRELEASED persons events past grace — the release
retry is persistently failing (persons down / release erroring); downstream cannot see
these finalizes until release succeeds    count=N

The release retry has been failing for longer than the grace window. The applications row is committed (the applicant got their 201 and credential), but the persons-side graph events are still held — downstream services do not yet know the household exists.

Response:

Check canopy-persons health (cargo xtask dev logs persons on devstack; the service health endpoint in deployment) — the usual cause is persons being down or the internal release endpoint erroring.
Confirm the reconciler is actually running and winning the leader lock: look for finalize reconciler tick complete (leader) on some replica. A replica logs finalize reconciler not started (no canopy-persons client) at boot when it has no OIDC service credentials — if every replica logs that, no one can release, and the credential config is the real problem.
Once persons recovers, the next tick confirms the release and the alarm count returns to zero on its own. No manual action against the database is needed — or safe.

Quarantined entities

cancel quarantined shared entities for a data steward (op still aborts)    operation_id=… quarantined=N

Compensation found an entity a later non-finalize write shares (for example a worker correction on a fact the aborted finalize created). Crypto-shredding it would destroy legitimate data, so it is left intact and recorded on the persons side. The operation still aborts.

Response: route to a data steward — the persons-side quarantine records (see the MR2 cancel surface in the persons API page) identify the entity kind + stable id. The steward decides whether the shared entity’s finalize-authored version should be superseded or kept. There is no automated follow-up by design.

Tunables

All validated at boot by FinalizeSagaConfig (bad combinations fail startup with a typed error); see the configuration reference.

Key (CANOPY_APPLICATIONS__…) Default Constraint / effect

Key (`CANOPY_APPLICATIONS__…`)	Default	Constraint / effect
`FINALIZE_RECONCILER_GRACE_SECS`	3600	> the 30 s persons request timeout, < 24 h. How long past lease expiry an attempt is presumed dead. Lower = faster PII cleanup after crashes, higher = more tolerance for stalled-but-alive attempts.
`FINALIZE_COMPLETED_RETENTION_DAYS`	30	≥ 1 day. Terminal saga rows are kept this long (a pruned `completed` op is still reconstructable from the authoritative `applications` row).
`FINALIZE_LEASE_SECS` / `FINALIZE_HEARTBEAT_SECS`	30 / 10	Saga-attempt liveness; the reconciler only acts once lease + grace have both lapsed.

FINALIZE_RECONCILER_GRACE_SECS

3600

> the 30 s persons request timeout, < 24 h. How long past lease expiry an attempt is presumed dead. Lower = faster PII cleanup after crashes, higher = more tolerance for stalled-but-alive attempts.

FINALIZE_COMPLETED_RETENTION_DAYS

≥ 1 day. Terminal saga rows are kept this long (a pruned completed op is still reconstructable from the authoritative applications row).

FINALIZE_LEASE_SECS / FINALIZE_HEARTBEAT_SECS

30 / 10

Saga-attempt liveness; the reconciler only acts once lease + grace have both lapsed.

The 5-minute cadence and the 100-op batch bound are compile-time constants (reconciler.rs) — they bound a stuck op’s extra dwell past grace to one tick without hammering canopy-persons.

Relationship to the orphan sweep

The reconciler handles operations the saga knows about (a finalize_operations row exists). Pre-ADR-038 orphans — persons graphs created by the old non-idempotent finalize with no saga record at all — are the job of the one-shot cargo xtask sweep-finalize-orphans (epic &71 MR9), which explicitly excludes anything with a live saga op so the two can never race.

Edit this page · default

Copy

View