ADR-025: Cross-Service Referential Integrity at the HTTP Boundary

On this page

Context

Per ADR-001 each canopy program and domain service owns its own PostgreSQL database. Cross-service references (e.g. household_id originating in canopy-persons, application_id in canopy-applications, determination_id in canopy-eligibility) flow through the wire as bare UUIDs in request bodies, and the receiving service persists them with no Postgres foreign-key enforcement available — the referenced rows live in a different database.

A workspace-wide audit on 2026-05-26 (post-MR for canopy-verification’s first domain DB, !372) confirmed that every domain service accepts cross-service IDs at face value and inserts. Concrete observed failure modes:

POST /internal/v1/ievs/match on canopy-verification persists ievs_hits rows scoped to any household_id the caller supplies; a faulty caller (or an integration test using uuid::Uuid::now_v7() as a stand-in) leaves orphan rows that surface in the worker-dashboard "IEVS alerts" panel as anchors to non-existent case-detail pages.
POST /v1/verifications on canopy-verification has the same shape and the same failure mode for the "Pending verifications" panel.
Analogous gaps exist on POST /v1/applications (canopy-applications), all five program services' POST /v1/determine handlers, POST /v1/renewals/snap/certifications, POST /v1/enrollments, POST /v1/appeals, and POST /v1/notices.

canopy-eligibility is a near-exception: its orchestrator validates household_id against canopy-persons via fetch_household_context, but the eligibility_requests INSERT happens before that validation, so an orphan request row briefly exists if validation fails.

The integrity story has to live at the HTTP boundary, and it doesn’t today. Tests passing with fake UUIDs is the bug — the API should reject the input.

Decision

Every cross-service write validates the referenced IDs against their owning service before persisting any row that carries them. Failures return 422 Unprocessable Entity with a structured UnprocessableEntity ApiError whose body cites the missing entity by kind + id.

The validator lives in a new canopy-validators crate (kept distinct from canopy-common to avoid a circular dependency on canopy-auth’s `ServiceTokenSource) and exposes:

#[async_trait]
pub trait CrossServiceValidator: Send + Sync {
    async fn validate_household(&self, id: HouseholdId)
        -> Result<(), ValidationError>;
    async fn validate_person(&self, id: PersonId)
        -> Result<(), ValidationError>;
    async fn validate_application(&self, id: ApplicationId)
        -> Result<(), ValidationError>;
    async fn validate_determination(&self, id: DeterminationId)
        -> Result<(), ValidationError>;
}

pub struct HttpCrossServiceValidator { /* … */ }

HttpCrossServiceValidator is the production impl: a reqwest::Client over a ServiceTokenSource (ADR-019) pointing at the relevant owning service. Each validate_* method issues a single GET /v1/{owning-entity}/{id} with bearer_auth(token); 200 is pass, 404 is ValidationError::NotFound, anything else surfaces as ValidationError::Upstream and 502s the caller (preserving the "fail loudly" posture for upstream outages rather than silently letting orphans land).

Each service wires the validator once in main.rs as an axum::Extension<Arc<dyn CrossServiceValidator>> and handler bodies call it before any write that carries a cross-service ID. canopy-eligibility’s fetch_household_context call sequence is amended so the validation precedes the eligibility_requests INSERT rather than following it.

Rollout is per-service in subsequent MRs (canopy-verification ships as the first adopter in the MR that introduces this ADR; one tracker issue + one issue per remaining service is filed at ADR-acceptance time).

Options considered

Option A: HTTP validation at the API boundary (selected)

Receiving service calls owning service via the existing service-class-JWT HTTP stack before insert. Failures return 422.

Pros: Aligns with ADR-001 (each service authoritative for its own data); single network hop per validation; no schema or DB-level coupling between services; works for any cross-service entity without per-pair plumbing; the wire shape is already a GET /v1/{entity}/{id} on every owning service; observable in the existing audit-log + tracing pipelines.
Cons: Adds one network call per write (per ID). Latency impact: per the 2026-05-13 SOLQ benchmark canopy-persons GET is <50 ms p99 inside the devstack pod network; production deployments where canopy-persons sits on a different rack would pay correspondingly more. Mitigated by batched fan-out via futures::stream::buffer_unordered(32) when a single handler validates multiple IDs (see canopy-eligibility’s pattern at orchestrator.rs:465-479).

Option B: Event-driven referential integrity with denormalized caches (rejected for v1)

canopy-persons emits household.created / household.deleted events; other services subscribe and maintain a local denormalized cache of valid IDs. Write-side handlers check the cache.

Pros: No per-write network call; cache lookups are sub-millisecond; tolerant of canopy-persons outages.
Cons: Cache consistency is non-trivial (event ordering, replay, cold-start hydration); a cache miss for a newly-created household creates a write-vs-event race window where legitimate writes fail; failure-mode space explodes (cache divergence, partial deliveries, replay drift); rabbitmq becomes a load-bearing dependency for every write where it’s currently best-effort observability. Defer until v1 latency proves untenable.

Option C: Defensive read-side rendering (rejected — doesn’t fix root cause)

Accept any ID on write; handle missing-reference cases gracefully on the read path (e.g., case-detail returns 404 / empty state instead of crashing).

Pros: Zero write-path changes; backward-compatible.
Cons: Doesn’t actually prevent orphan rows from accumulating in the producer’s DB; the dashboard still surfaces dead-link anchors; integrity invariants stay broken at the persistence layer; merely papers over symptoms. Rejected as inconsistent with the "no half-implementations" rule.

Consequences

New canopy-validators crate landing alongside this ADR. Cargo workspace gains one new member.
Every domain service grows an outbound HTTP edge to the entity-owning services (canopy-persons primarily; canopy-applications and canopy-eligibility secondarily). Health monitoring should add per-validator circuit breakers similar to canopy-eligibility’s ProgramServiceRegistry.
docker-compose env wiring adds CANOPY_{SERVICE}__PERSONS_URL (and analogous) where absent.
Integration tests previously using uuid::Uuid::now_v7() as a stand-in household_id must instead insert via canopy-persons first; the test harness already exposes this via canopy_test_lib::TestClient.
Pre-existing orphan rows (from past test runs) need a one-shot cleanup. A cargo xtask seed sweep-orphans subcommand (separate FU) removes rows whose cross-service references no longer resolve.

Rollout

MR landing this ADR: canopy-validators crate + canopy-verification adopting it (POST /internal/v1/ievs/match + POST /v1/verifications). Tests updated. Polluted rows removed.
Subsequent MRs (tracker issue filed at ADR acceptance): canopy-applications, the five program services (snap/tanf/medicaid/caps/wic), canopy-renewals, canopy-enrollment, canopy-appeals, canopy-notices. canopy-eligibility’s fetch_household_context reordered to precede the eligibility_requests INSERT.
Follow-up: cargo xtask audit cross-service-refs lint that greps every handler accepting a Json<T> with cross-service-ID fields and ensures a validator.validate_* call precedes any persistence call. Catches regressions in CI.

Edit this page · default

Copy

View