Event-Delivery Protocol

On this page

Canopy’s benefit pipelines (adverse actions, continued benefits, assessments — epic &72) are event-driven, and a lost event is a lost legal obligation. This page is the protocol every load-bearing producer and consumer follows. Substrate changes shipped in #1088 (durable topology) and #1089 (parked-state inbox + versioning + the binding-first gate); the crate mechanics live in canopy-mq.

Durable topology (#1088)

  • Subscriber queues are durable by default; the outbox drainer publishes delivery_mode=2 — the persistence chain is end-to-end.

  • The devstack broker keeps its state in a compose volume with a pinned hostname; unroutable publishes land in the durable canopy.unrouted capture queue via an alternate-exchange policy instead of being discarded. Production brokers must be provisioned equivalently (production-gap register, epic &72).

The parked-state inbox (#1089)

A load-bearing handler faces three distinct non-success cases, and each has exactly one correct disposition:

Case Disposition Mechanism

Malformed payload on a load-bearing key

DLQ — never warn-and-ack (that silently discards a legal event)

return Err (nack → requeue ×N → DLQ)

Delivery this binary cannot INTERPRET (unknown event type during a rolling deploy; schema_version newer than supported)

Park — durable, self-recovering

return Err(ParkEvent::unrecognized(..)/needs_schema(..).into())

Legitimately not-for-us (e.g. another program’s event)

Ack (a considered skip, usually with a debug!/warn!)

return Ok(())

Parking rolls back the handler transaction (never committing partial domain writes), upserts the inbox row with parked_at/park_reason/ queue_name, and ACKs the broker — the inbox row is the durable copy. Every subscriber runs an unpark scanner (CANOPY_MQ_UNPARK_INTERVAL_SECS, default 60s) that re-offers parked rows to the CURRENT binary, so a later deploy that understands the event processes it with no operator action. canopy_mq::run_unpark_pass is the on-demand surface for tests and operator tooling. Real handler failures during an unpark attempt keep the row parked with attempts/last_error bumped — there is no broker delivery left to DLQ, so the row itself is the operator signal:

-- Operator triage: parked backlog per queue
SELECT queue_name, count(*), min(enqueued_at)
FROM event_inbox
WHERE parked_at IS NOT NULL AND processed_at IS NULL
GROUP BY queue_name;

The inbox also serializes concurrent deliveries of one event: the classify step takes a row lock (FOR UPDATE), so two replicas can never double-run a committed-but-unprocessed row (#1089; previously possible via the InFlightRetry path). The inbox schema is single-sourced with the outbox (ADR-039): canonical files in crates/canopy-mq/outbox-migrations/, distributed by cargo xtask outbox-migrations --write, parity-gated in the battery.

Envelope schema versioning (#1089)

EventEnvelope.schema_version (default 1 for pre-#1089 envelopes). The compatibility rule is additive within a version: consumers tolerate unknown payload fields; a change that cannot be expressed additively bumps CURRENT_SCHEMA_VERSION, and older consumers park (ParkEvent::needs_schema) instead of misinterpreting. Removing or re-meaning a field without a bump is a correctness defect.

Binding-first deployment (#1089)

An MR that ADDS routing keys to a queue (the consumer/topology half) lands and deploys BEFORE the MR that emits them (producer activation). Every epic-&72 phase splits its MRs this way. The tree-level half is mechanically gated: cargo xtask mq-topology (battery stage 9e) fails when a published key has no in-tree non-wildcard consumer and no entry in xtask/mq-topology-allow.toml — the allowlist is the honest register of today’s unbound keys, each with a reason, and goes stale-checked in the same gate (an entry whose key gains a consumer must be removed). The canopy-security # audit binding is deliberately excluded from "consumer" — it would make the gate vacuous.

The same gate cross-checks the PRODUCER half (#1122): every key published from a service’s tree must match that service’s topic-permission write regex in devstack/rabbitmq/definitions.json (failure — the broker would refuse the publish with ACCESS_REFUSED, the #1102 incident class), a key published only from shared-crate code must be writable by at least one principal (also a failure), and a decomposable ACL branch whose key is neither published, subscribed, nor present as a source literal is flagged stale (advisory).

The publish scanner resolves three key-argument shapes: string literals, contracts-crate pub const paths, and — since #1272 — a local variable assigned from a match over consts/literals (the medicaid ELE grant dispatch shape, whose granted/extended keys previously hid as an uncounted "dynamic" site and shipped without ACL coverage). A genuinely opaque key expression (format!, a computed value) counts as a skipped dynamic site, and a dispatch with any opaque ARM records what it can resolve but stays in the dynamic tally too — the lint prints the count so a rising number is visible in review, and a partially-resolvable match can never hide a key more silently than a fully-dynamic site does.

Queue migration procedure

Renaming a queue or changing its properties on a live broker:

  1. Declare the NEW queue (versioned name, e.g. <queue>.v2) + bindings — consumer MR, deployed first.

  2. Let producers continue publishing (topic routing delivers to both).

  3. Drain the OLD queue to zero depth with no in-flight deliveries (rabbitmqctl list_queues name messages messages_unacknowledged).

  4. Unbind, then delete the old queue. Its DLQ is retained until empty and triaged — never deleted with content.

  5. Rollback = re-bind the old queue (never deleted before the new one is proven under load).

Property changes on the SAME name (e.g. #1088’s transient→durable) ride the subscriber’s one-time self-heal: PRECONDITION_FAILED → if-empty delete → redeclare. A mismatched queue still holding messages fails loudly for operator action.

Provenance

source_service on the envelope is labeling; the broker principal is the authentication (#1093). Every service connects with its own RabbitMQ user (canopy-<service>, devstack creds public by design — Kerckhoffs; production injects real secrets at deploy) carrying:

  • resource permissions scoped to canopy.events, canopy.dlq, and the service’s own canopy-<service>.* queues (write on canopy.dlq is required by the broker’s declare-time check on queues carrying x-dead-letter-exchange; since the DLX is a direct exchange, topic ACLs don’t apply there — the residual: a service could inject into a neighbor’s DLQ, and while it cannot read a neighbor’s EXISTING queue, it could bind its own queue to the shared DLX with a neighbor’s routing key and siphon FUTURE dead letters; per-service DLX exchanges are the hardening if that residual ever matters), and

  • a topic-permission write regex on canopy.events enumerating exactly the routing keys the service’s tree publishes (^$ for publish-only-nothing services) — the broker refuses a forged foreign key at publish time, regardless of what source_service claims. Adding a service’s first/new event key means widening its ACL in devstack/rabbitmq/definitions.json in the same MR. The matrix was derived from the cargo xtask mq-topology --verbose publish map and is hand-maintained; drift is now statically gated by the same mq-topology battery stage (#1122 — a published key its principal cannot write fails the lint before it can fail at the broker), with acl_test.rs proving the broker-side enforcement live.

Consequently source_service stays diagnostic — consumers must still never branch authorization on it (the principal, not the label, is the guarantee). One recorded carve-out (#1519): the envelope’s programs field is the AUTHORITATIVE program-scope assertion for the audit-row programs column — the publisher is the authority on which programs its own event concerns (the same trust model as programs_requested on application rows), and the audit store validates the vocabulary and derives from the routing key when the field is absent. Payload-embedded program labels remain diagnostic; the store never reads them. Replay caveat: the #433 admin replay re-publishes a consumed envelope under the service’s OWN principal, so replaying an inbox row whose routing key belongs to another service is refused by the broker and lands in the replay report’s failed bucket. The admin/tooling principal (canopy) retains full publish for operator surgery; a broker-friendly targeted-replay design is tracked in the epic &72 follow-ups.

Edit this page · default