Report Runs — Operations Runbook

AUTH (#1438, ADR-043 §C): the generate POSTs are USER-ONLY — the caller needs the worker’s exchanged aud=canopy-reporting token (supervisor-or-above); service-class tokens and, under enforcement, direct broad-audience worker bearers are 403. The runs poll/list surfaces below stay dual (service or supervisor).

The five generate POSTs (/v1/reporting/snap/fns-388, …/snap/qc-universe, …/tanf/acf-199, …/medicaid/tmsis, …/medicaid/cms-416) become the enqueue surface in MR5/MR6: 202 + ReportRunAccepted{run_id, generation_id, poll_url}
Location; 409 with the in-flight run’s handle when the (kind, period) already has an active run; 503 at the queue cap or when runs_enabled=false.

One active run per (kind, period) — enforced by a partial unique index. The 409 body carries the active run’s handle, so "conflict" is always pollable (runs are org-visible for exactly this reason).

Period canonicalization is enforced, not documented: monthly kinds pin the month start, CMS-416 pins Jan 1 of the report year, QC keys the snapshot date. The enqueue path normalizes before insert; the schema CHECKs would refuse anything else.

GET /v1/reporting/runs/{id} → ReportRunStatus. Counters come from run COLUMNS and detail_counters from the generation row — progress is never decoded on the read path, so a malformed progress blob cannot 500 a poll. Queued/running responses carry Retry-After (one worker tick). A done run carries result_url pointing at the kind’s read endpoint.

GET /v1/reporting/runs?kind=&period=&limit= — newest-requested first, limit clamped to the house bounds, unknown kind → 422.

RBAC: supervisor-or-above OR any service-class caller.

A 404 on a previously-valid run id usually means the run was REAPED (see below). Its provenance is not lost — the generation row is permanent and carries the copied run summary.

/readyz renders a non-gating worker check (label report-runs): pending before the first pass (the delayed first tick is normal), ok, degraded on a stalled loop or an accumulating error streak, and disabled under the operator override. Worker degradation NEVER 503s readiness — pulling a replica cannot revive its own in-process worker.

Metrics (OTel meter canopy_reporting): canopy_reporting_run_claims_total, canopy_reporting_runs_serviced_total, canopy_reporting_run_abandons_total, canopy_reporting_run_finalizes_total{state,error_code}, canopy_reporting_run_queue_depth, canopy_reporting_run_oldest_queued_age_seconds, canopy_reporting_run_worker_last_success_age_seconds (-1 = never).

Every worker decision logs with run_id; look for report-run pass failed (engine errors, capped backoff) and claim fenced mid-pass (lost lease — informational, the run continues elsewhere).

the worker parks at boot (logs report-run worker DORMANT; /readyz worker check reads disabled), AND

enqueue answers 503 — a 202 for work that will never run is a lie, so the queue can never accumulate dead-letter runs while disabled.

Run reap — terminal (done/error) report_runs rows older than run_reap_days (default 7; a 7-day floor is enforced in SQL) are deleted. Polling a reaped run returns 404; provenance lives on the generation row, which is never deleted.

Generation-row janitor — superseded and abandoned generations' OUTPUT rows (the five report tables) and report_run_universe rows are deleted once past run_generation_row_retention_days (default 30). Retention clocks: a superseded generation is measured from its SUCCESSOR’s published_at (exact — promotion supersedes and publishes in one transaction); an abandoned one from its created_at (conservative; it was never visible). report_generations rows are PERMANENT — the janitor only removes bulk rows. Batch-bounded (50 generations per pass) and idempotent.