Finalize-Orphan Sweep: pre-saga orphaned PII graphs

On this page

Tool: cargo xtask sweep-finalize-orphans (#1055, ADR-038 MR9)
Audience: operators holding (or working with a holder of) the data_steward role.

What it fixes

Before epic &71, a crash or lost response inside finalize_draft could leave a fully-formed canopy-persons graph — household, members, income/asset/expense facts, all stamped origin='finalize' — with no applications row: orphaned applicant PII no case ever references. The ADR-038 saga (receipts
reconciler) prevents new orphans; this sweep finds and compensates the pre-existing stock, one-shot.

An orphan candidate must satisfy both:

the household id appears in no canopy_applications.applications row (anti-join), and
the household has a finalize-authored self membership (origin = 'finalize' AND relationship = 'self' — provenance finalize always wrote, saga and pre-saga alike).

Two exclusions:

Saga-era graphs — any household covered by a finalize_receipts row — are skipped permanently: their lifecycle (retry or compensation) belongs to the finalize reconciler, never this sweep. A true pre-saga orphan predates the receipt table and cannot have one.
Ambiguous graphs — finalize-origin households with no self membership (a crash before the self-membership step, or unattributable data) — are listed in the manifest for steward review and never auto-compensated.

Phase 1 — discovery (dry-run, the default)

cargo xtask sweep-finalize-orphans

Reads the persons + applications databases (via docker exec psql on the devstack; run it wherever those DBs are reachable the same way) and writes a digest-sealed, PII-free manifest — household ids only — to test-results/finalize-orphans/manifest.json (override with --manifest). Nothing is mutated. Review the three lists:

candidates — will be compensated by --apply.
ambiguous — steward-manual; resolve via the redaction endpoints or leave.
skipped_saga_era — reconciler-owned; if one lingers, check the reconciler runbook instead.

The digest field seals the candidate list: --apply refuses a manifest whose candidates were hand-edited after discovery (re-run discovery instead).

Phase 2 — apply

Prefer a quiescence window (finalize traffic paused) — not for correctness (the guards below hold regardless) but so the operator reviews a stable picture.

export CANOPY_SWEEP_TOKEN="$(…a data_steward bearer token…)"  (1)
cargo xtask sweep-finalize-orphans --apply \
  --manifest test-results/finalize-orphans/manifest.json

1	A user token for an account holding the dedicated `data_steward` role (ADR-036 Decision M — admins do not inherit it). One mint is all it takes: on the devstack, password-grant `data.steward` / `password`:

KEYCLOAK=http://localhost:8180  # the devstack issuer; adjust per environment
export CANOPY_SWEEP_TOKEN=$(curl -s \
  "$KEYCLOAK/realms/canopy/protocol/openid-connect/token" \
  -d grant_type=password -d client_id=canopy-api \
  -d username=data.steward -d password=password | jq -r .access_token)

The compensate route is user-only under the #1428 receiver contract (ADR-043 §C) — where CANOPY_PERSONS__ENFORCE_USER_ONLY_ROUTES is on (the devstack; production at cutover), a raw password-grant token is refused (403 aud_not_exact). Since #1501 the tool handles that itself: --apply RFC 8693-exchanges the minted token for an exact aud=canopy-persons user-context token before the first POST. Knobs (defaults suit the devstack): CANOPY_SWEEP_EXCHANGER_CLIENT_ID / CANOPY_SWEEP_EXCHANGER_SECRET (default: the public-by-design canopy-web-exchanger pair — production sets both to its provisioned exchanger client), --keycloak-url / CANOPY_SWEEP_KEYCLOAK_URL (default: the devstack host port from .ports.env), and CANOPY_SWEEP_KEYCLOAK_REALM (default canopy). Tokens ride only in the Authorization header of the persons calls; passing them via env — never a CLI flag — keeps them out of shell history and process lists.

Per candidate, in order:

Re-validate the anti-join immediately before acting — a household now referenced by an application is recorded now_referenced and left intact.
POST /v1/households/{id}/compensate-finalize-orphan on canopy-persons (data_steward-gated). The server re-checks, in one transaction under the household advisory lock: the finalize self-membership provenance, and that zero finalize receipts touch any entity in the graph — then compensates through the same shred-or-quarantine machinery as the saga’s cancel: exclusively-finalize entities are crypto-shredded (ADR-036 tombstone, by inventoried DEK) and deactivated/superseded; anything shared with non-finalize data is quarantined — left intact and reported, never destroyed.
The outcome is appended to the results sidecar (<manifest>.results.json) after every candidate — a stopped run resumes with the same command and skips completed work. error: outcomes (network, 5xx) are retried on re-run; compensated, now_referenced, and refused_409 are terminal.

Outcomes

Sidecar status Meaning / operator response

Sidecar status	Meaning / operator response
`compensated (N entities, M quarantined)`	Done. `M > 0` means shared entities were left intact — route the quarantine list (entity kinds + ids in the persons response) to the data steward.
`now_referenced (left intact)`	An application appeared since discovery — not an orphan; no action.
`refused_409: …`	A server guard tripped (provenance or receipts) — state changed since discovery, or the manifest came from another environment. Investigate before re-running discovery.
`error: …`	Transient (persons unreachable, 5xx). Re-run `--apply`; only these retry.

compensated (N entities, M quarantined)

Done. M > 0 means shared entities were left intact — route the quarantine list (entity kinds + ids in the persons response) to the data steward.

now_referenced (left intact)

An application appeared since discovery — not an orphan; no action.

refused_409: …

A server guard tripped (provenance or receipts) — state changed since discovery, or the manifest came from another environment. Investigate before re-running discovery.

error: …

Transient (persons unreachable, 5xx). Re-run --apply; only these retry.

The command exits non-zero only when error: outcomes remain.

Verification

Spot-check a compensated household id in the persons DB: households.active and each member’s persons.active are false; the graph’s household_member_versions / income_versions / … rows have superseded_at set (append-only audit trail retained); redaction_keys rows for the graph’s sealed subjects carry shredded_at (tombstoned, ADR-036 — values are unrecoverable, rows remain as proof). Re-running discovery no longer lists the household (inactive households leave the candidate pool).

Idempotency & safety properties

The endpoint is idempotent — a replayed apply (lost response, resumed run) re-walks the graph and no-ops per already-compensated entity.
Compensation is crypto-shred + deactivate/supersede, never row deletion (redaction_keys is trigger-protected against DELETE; version rows are the audit trail).
Every artifact (manifest, sidecar, stdout, the persons request/response) is PII-free: ids, entity kinds, counts, and reason strings only.

Edit this page · default

Copy

View