Bulk COLA Runs — Scaling & Operations Runbook (#1213)

Residual during a rolling snap restart: an old replica lacks the guard; its unattested write is caught by the orchestrator’s post-verify binding (provenance_rejected, quarantine-marked) and recovered by adopt-or-skip once the fleet settles — verified by the failure-mode suite.

renewals_url + rules_url set; snap registered; CANOPY_MQ_PREFETCH_COUNT=1 (parsed by the subscriber’s own semantics — 0/absent/garbage mean 32 and REFUSE boot); db_max_connections ≥ 2K+4. Per-replica DB total = db_max_connections + the 2K+2 bookkeeping pool.

Before the first enabled deploy: grant the eligibility broker user topic-write ^(determination\.(completed|requested))$ on canopy.events (devstack definitions.json is the template); pre-declare canopy-eligibility.bulk-redetermination durable + its .dlq on canopy.dlq; alarm on canopy.unrouted; verify a routing preflight (publish + consume one synthetic determination.requested on a prefixed queue); rollback = disable the flag (consumers/worker stop at next deploy; pause/cancel/GETs stay live, H18).

Preview complete + failures page reviewed (overriding an unclean preview is audited); the H17 ladder re-resolves corpus + provenance and requires exact pin equality, attestation_enabled=true, and snap legal-today ≥ the run’s as_of — an enact on September 30 is a typed 409 enact_too_early.

K=2 consumers/replica (1..=4), bulk_max_inflight=16, self-call timeout 90s (cap 240 < STALE_REQUEST_TTL 300), redispatch TTL 1800s > inflight×timeout/K, v1 cohort ceiling 1000 (raising it is a deliberate config change; full-caseload is post-#1133). Throughput ≈ replicas × K × (1/determine-latency); the canary holds the first case solo per phase.

≥5K seeded cohort, ≥2 replicas, K=2, with next-FY parameter fixtures STAGED so the preview/enact amount DELTA is asserted (devstack stages one set, so the AC6 suite pins provenance, not deltas): sustained settles/min; interactive acceptance = success/409 rate + p95 during the drain; monotone backlog; breaker/canary/deadline drills; the alert rule on canopy.unrouted + consumer-attach degradation live.