ADR-039: Single-Source Outbox Schema + First-Class Event-Hold

Accepted (2026-07-13)

Realized by epic &71 (#1057); consumed by the finalize saga (#1047+). See the plan.

Per ADR-001 every service owns its own PostgreSQL database, so the event_outbox table must be created by a migration in each service’s own migrations/ dir (sqlx runs one Migrator per service, from a compile-time path). The two outbox migrations (create + lease columns) were therefore copy-pasted byte-identically across all 18 outbox-bearing services with no shared source and no drift gate — identity was maintained purely by hand (the create migration’s own header even under-counted the copies). The shared drainer (crates/canopy-mq/src/outbox_drainer.rs) is a single hardcoded SQL string executed against every service DB via that service’s pool, so it structurally assumes a uniform outbox shape.

Separately, the epic &71 finalize saga needs to publish canopy-persons graph events transactionally with the domain write (so they are emitted iff the write commits) yet not have them delivered until the owning cross-service operation commits — and to discard them if the operation is compensated. The generic idempotency middleware cannot provide this (it is at-least-once-on-crash and caches plaintext PII), and staging events in a side table then copying them into the outbox on release would duplicate the outbox’s serialization + drain machinery. The natural home is the outbox itself.

Adding a hold column to the outbox by hand would have grown the 18× duplication. Pre-1.0 there are no deployments, so the schema can be freely restructured — fix the duplication first, then add the hold once.

The original decision left one lifecycle question open: what happens to a held row whose owning operation never calls release_held/drop_held? The drainer’s claim predicate skips held rows and the ADR-018 janitor deletes only published rows, so an abandoned hold accumulates forever, silently. The failure mode is real, not hypothetical: a coordinator crash before the applications-side finalize_operations record exists leaves held rows that the ADR-038 reconciler can never see (it walks finalize_operations, not event_outbox), and the steward orphan sweep deliberately refuses receipt-covered graphs. The devstack accumulated 36 such rows in canopy_persons.event_outbox during pre-reconciler crash testing.

Decision. The mq layer exposes the signal and never reaps. A third drainer-owned task (the held-row watch) polls the held set every 5 minutes, exports canopy_mq_outbox_held_rows and canopy_mq_outbox_held_oldest_age_seconds gauges (behind the otel feature; the OTLP resource carries service.name), and emits a tracing warn — count, oldest age, oldest hold_operation_id; ids and counts only, no payload — once the oldest hold outlives CANOPY_MQ_HELD_AGE_WARN_SECS (default 7200, twice the reconciler’s default stuck-op grace).

Why no mq-level TTL reaper. The hold key is deliberately opaque to canopy-mq (an (operation_id, generation) pair with no semantics attached), so the mq layer cannot distinguish a live-but-slow saga’s hold from an abandoned one. Auto-dropping on age would delete a slow-but-live finalize’s events (violating the ADR-038 §6 atomic-visibility guarantee from the Consequences above); auto-releasing would publish a partial graph — strictly worse. Semantic release/drop therefore stays with the hold’s creator side: the finalize saga post-commit, its reconciler’s retry/compensate paths, and — for holds with no surviving operation record — a steward-driven remediation informed by exactly this signal (the warn names the orphaned operation id). If a recurring no-op-record crash class emerges, the fix belongs in the coordinator’s ordering (persist the op record before staging holds), not in an mq guess.