Runbook: Appeals Reconciliation (receipts vs links, parked elections)

On this page

The nightly scanner (services/canopy-appeals/src/reconcile.rs, advisory lock canopy-appeals.reconciliation) is the durable backstop the DLQ cannot be: it compares appeals' PERSISTED stay receipts against enrollment’s per-appeal links (GET /v1/adverse-actions/{id}/stays/{appeal_id}) and flags CB elections parked in pending_stay past the operational SLA ([appeals].pending_stay_alert_hours, default 24). It is report-only — structured ERROR logs plus the typed report from POST /v1/internal/appeals/reconcile. Remediation is yours.

Running it on demand

# service-class token (any canopy service principal with appeals access)
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{}' \
  "$APPEALS_URL/v1/internal/appeals/reconcile" | jq .

The report: receipts_checked, lingering_checked, findings[] (appeal_id, adverse_action_id, kind, detail), and skipped_unreachable (rows the sweep could not compare because enrollment was down — a non-zero value means re-run later, not all-clear).

Alert: lingering_stay

A withdrawn/decided appeal still holds a LIVE stay in enrollment — the #1099 fence-first paths commit the appeals-side transition first and command enrollment after (household-safe ordering), so a release/veto failing post-commit leaves exactly this state. Both sides agree the link is stayed, which is why this is its own lifecycle-vs-link sweep rather than a receipts-vs-links disagreement. The failing request was timeline-noted at the time (action_command_failed). The stay only DELAYS the action (household-safe), but it blocks enactment until cleared.

Remediation: re-issue the command the failed call would have made (idempotent, #1096) — release for a finalized withdrawal or an agency-favorable decision, veto for a reversal — using the curl below, then re-run the sweep.

Alert: receipt_link_disagreement / link_missing

The appeal’s cb_stay_link_status contradicts enrollment’s link (or no link exists). Known producers, most-likely first:

  1. The command landed but the receipt write-back failed — enrollment’s link moved (released/vetoed) and the appeals row never heard.

  2. An out-of-band actor drove the link directly on enrollment (another appeal’s veto moots ALL links; operator surgery).

Remediation:

  • Read the appeal’s timeline (GET /v1/appeals/{id}) and the action (GET /v1/adverse-actions/{id}) — the pair tells you which side moved.

  • When the appeals row is stale (link legitimately moved on — producer 1): no data fix is required for a TERMINAL divergence (released/vetoed on the enrollment side); the enrollment link is the ground truth for enactment gating. Record what happened on the appeal’s timeline if the case narrative needs it.

  • link_missing with a recorded receipt means the link row is GONE on enrollment — that should be impossible outside operator surgery; treat as an incident and reconstruct from the two services' timelines/signals.

  • Re-run the sweep and confirm the finding clears.

The re-issue curl (used by the lingering_stay remediation above):

curl -s -X PUT -H "Authorization: Bearer $APPEALS_SVC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"command":"release","actor":"runbook:appeals-reconciliation"}' \
  "$ENROLLMENT_URL/v1/adverse-actions/$ACTION_ID/stays/$APPEAL_ID"

Alert: pending_stay_past_sla

A household ELECTED continuation and the grant is still pending because the stay never landed — the retry worker (60s cadence) has been failing for at least the SLA window. The household’s legal position is elected-but-ungranted: fix this before the action’s enact_not_before arrives.

  1. Check enrollment health (/livez) and the appeals logs for the retry worker’s per-row error (pending-stay retry failed / enrollment unavailable for stay).

  2. If enrollment recovered, the next worker tick completes the grant — re-run the sweep to confirm.

  3. If the row references a dead action (cancelled/vetoed underneath the filing), the worker re-resolves it on its next successful contact; a row that STAYS parked after enrollment is healthy means the stay is being refused against a still-scheduled action — that is outside enrollment’s #1096 contract and warrants a bug report with both rows attached.

Boundaries

  • The scanner bounds each sweep at 500 rows per pass (newest receipts first, newest terminal-stayed first, oldest parked first) — a backlog larger than that surfaces over consecutive nights; a checked-count of 500 is the tell.

  • No mutation, no events: activation of appeal-side pipeline events is Phase 3 (epic &72 plan, MR 3.2).

Edit this page · default