ADR-002: Black-Box Determination Contract

On this page

Status

Accepted

NOTE
Amended by ADR-028 (determination input snapshot).
NOTE
Amended by ADR-034 (per-program determination context-mapping) — specifies how the application context this ADR "submits" to each program is built: per-program, typed, and complete-or-provisional.
NOTE
Amended by Amendment 1 (async/bulk determination variant — mass-change machinery, scale audit epic &73, #1237) — which adds an async/system-initiated determination path (a queued determination.requested trigger) and designates the existing signed previous_determination_id (ADR-028 §57), plus a typed trigger enum, as the initial-vs-re-determination signal for that path. The Decision text is unchanged; see Amendment 1 §D2 for the async signing invariants and §D9 for the signal.

Context

Once program services are isolated (see ADR-001), a question arises about what the contract between program services and the eligibility orchestrator (canopy-eligibility) looks like.

Two broad approaches are possible:

  1. The orchestrator is given access to the program service’s data and makes the determination itself.

  2. The program service makes the determination internally and returns only the result.

The first approach — orchestrator-as-brain — is how most integrated eligibility systems are designed. It is also how the data isolation requirements are violated: once the orchestrator can query FTI, the logical isolation of canopy-tanf is defeated.

The second approach — program-as-black-box — follows directly from ADR-001 and has independent technical and legal advantages worth documenting explicitly.

Decision

Program services operate as black boxes with respect to the eligibility orchestrator.

The contract between a program service and canopy-eligibility is a signed determination object:

{
  "program": "snap",
  "household_id": "...",
  "application_id": "...",
  "determination": "approved",
  "benefit_amount": 847.00,
  "benefit_unit": "monthly_usd",
  "effective_date": "2026-04-01",
  "expiration_date": "2027-03-31",
  "renewal_date": "2027-02-01",
  "basis": "magi_gross_income_under_130pct_fpl",
  "program_service_version": "1.4.2",
  "determined_at": "2026-03-25T14:32:00Z",
  "signature": "<detached JWS>"
}

The signature is a detached JWS (JSON Web Signature) produced by the program service’s private key. canopy-eligibility verifies the signature before accepting the determination. No program service may accept a determination that fails signature verification.

The orchestrator:

  • Submits an application context (household composition, self-reported income, program applied for) to each relevant program service

  • Receives signed determinations

  • Applies the federal eligibility hierarchy (EE15 — most advantageous group assignment) across determinations

  • Assembles the combined result for the applicant

The orchestrator does not:

  • Query program databases

  • Access verification data (FTI, SSA, IEVS)

  • Reconstruct the reasoning behind a determination

  • Override a determination without routing a new evaluation through the program service

Rationale

Legal

The black-box contract is the logical enforcement mechanism for ADR-001’s data isolation requirement. An orchestrator that can query program data defeats isolation in practice even if the isolation exists in principle. By making the contract a determination object rather than a data query, the boundary is architecturally enforced rather than policy-enforced.

An IRS Pub 1075 auditor can observe that canopy-eligibility receives a determination from canopy-tanf without receiving the FTI that produced it. The audit boundary is clear and demonstrable.

Cryptographic integrity

The JWS signature on each determination object serves two purposes:

  1. Tamper-evidence: No downstream service can modify a determination in transit or at rest without invalidating the signature. A SNAP determination that says "approved" cannot be changed to "denied" — or vice versa — without `canopy-snap’s private key.

  2. Non-repudiation: The determination is attributable to a specific program service at a specific version. This matters for audit trails, appeals, and federal reporting — all of which require knowing which system made a determination and when.

This pattern is derived from the CRAIG project’s JWS intake signing (CRAIG ADR-010, external to this repository), which uses ECDSA P-256 detached JWS for partner-submitted intake reports. The same cryptographic approach applies here with the program service as the signer and canopy-eligibility as the verifier.

Operational resilience

A program service can be deployed, updated, or restarted independently without affecting determinations already in flight. Signed determinations are immutable records — a canopy-snap deployment does not invalidate existing approved determinations.

Alternatives considered

Alternative 1: Orchestrator queries program databases directly Rejected. Defeats the data isolation guarantee of ADR-001. Also creates tight coupling between the orchestrator’s data model and each program’s schema, making independent program deployment impossible.

Alternative 2: Shared determination database, programs write, orchestrator reads A compromise — programs write determinations to a shared store, orchestrator reads from it. Rejected because a shared determination store becomes a target for cross-program data leakage: if the store contains TANF determinations alongside SNAP determinations, and the TANF determination includes any FTI-derived fields, the shared store is subject to Pub 1075 controls. The black-box contract keeps the determination object minimal by design.

Alternative 3: Unsigned determination objects Simpler to implement but rejected because it provides no tamper-evidence guarantee and no non-repudiation for appeals and audit purposes.

Consequences

  • Each program service generates and manages a signing key pair.

  • canopy-eligibility maintains a verification key registry — one public key per program service.

  • Key rotation must be coordinated; key rotation plan documented separately.

  • Determination objects are append-only records. Once signed and accepted, they are not modified — only superseded by a new determination from a new evaluation.

  • The black-box contract means canopy-eligibility cannot explain why a program approved or denied without querying the program service for a human-readable explanation separately — which the program service may provide via a /determination/{id}/explanation endpoint that returns narrative, not data.

Amendment 1 — Async/bulk determination variant (mass-change machinery) (scale audit epic &73, #1237, 2026-07-27)

Status unchanged (still Accepted; amendments extend, they do not supersede). The 2026-07-25 scale-readiness audit (epic &73) found (H6) that a determination is only ever triggered by a synchronous, single-attempt, ~30-call HTTP fan-out per household inside one interactive request lifetime (services/canopy-eligibility/src/orchestrator.rs:1398-1412) — there is no queued/bulk path and no async trigger. So a mandatory mass change — the canonical case is the annual 7 CFR 273.12(e) COLA rebudgeting that re-determines every ongoing SNAP case (canopy-policy pins snap-cola to Oct 1 with grace_days=0, so the ruleset flips on time but nothing applies it to the caseload; GA ≈ 800K households ≈ 6.7h of saturated synchronous fan-out best-case) — has no contract-level home, and a naive bulk driver would collide with the one-pending-slot invariant and starve live interactive determinations. This amendment adds the async/bulk determination variant: a queued determination.requested trigger, the async-path signing invariants, stable idempotency, durable checkpoint/resume, bounded retry, one-pending-slot deferral, and — reusing the existing signed previous_determination_id (ADR-028 §57) as the supersession linkage plus a typed trigger enum as the classifier — an initial-vs-re-determination signal. The synchronous signed-determination Decision (§Decision) is left byte-immutable — the per-program signed determination object and its detached-JWS trust boundary are unchanged; what this amendment adds is when/how a determination is triggered and the lifecycle/dedup/retry semantics around it.

This amendment pins the contract (D1–D10), the invariants, and the acceptance criteria; the implementation children own the byte-level. #1213 (the mass-change / October-COLA driver) builds the determination.requested consumer, the bulk-enqueue admin surface (cohort selection, dry-run then enact), the durable checkpoint/resume tables, bounded concurrency, and the per-dispatch idempotency keys; #1133 owns enrollment apply-semantics for an already-enrolled re-determination. Everything beyond the shipped synchronous path is not built yet — the clauses are normative (MUST/SHALL), not as-built. This amendment governs the determination contract only; how enrollment applies a re-determination (adjust-in-place vs supersede under the #1130 one-live-enrollment fence, and the 7 CFR 273.13 reduction-type adverse-action routing) is #1133 and is explicitly out of scope (§D9, §Consequences).

Settled decisions

  1. The async path reuses the exact synchronous signer — no "system" key. Every determination on the async path is signed with the same boot-acquired per-program signing key, the same ADR-036 key-derived kid, and the same signing_key_history registration as the interactive path, and runs the full ADR-028 seal-hash-then-sign ritual. A separate "system" signing key would make the orchestrator’s per-Program verification registry treat every async determination as unknown and quarantine it (§D2).

  2. The signal reuses the existing signed previous_determination_id (ADR-028 §57) — this amendment introduces no new field. The determination envelope already carries a signed previous_determination_id: Option<DeterminationId> (ADR-028 §57, part of canonical_signing_payload); this amendment designates it the async supersession linkage #1133 uses to pick which live enrollment to adjust/supersede. Because its None is tri-valued (a first determination, a legacy row, or a program not yet implementing supersession), None alone is NOT a sound initial-vs-re-determination signal — the authoritative classifier is a typed trigger/reason enum (§D9), with previous_determination_id carrying the linkage. Both are inside the signed JWS payload, so neither can be stripped or spoofed by envelope/routing metadata (§D9).

  3. An already-enrolled household’s re-determination is a first-class, expected contract outcome — not a defect, not a DLQ candidate. The current enrollment PARK (services/canopy-enrollment/src/auto_enroll.rs:143-171) and the create-API 409 are documented as the bridge until #1133 implements adjust-vs-supersede (§D9).

  4. The Decision text is left byte-immutable — this amendment extends, it does not supersede.

The async/bulk determination contract (children own the byte-level)

D1 — Async trigger: a determination.requested command event. The async path is triggered by a new determination.requested routing key — a request/COMMAND event, explicitly distinct from the existing determination.completed* fact events. No such key exists today. It MUST carry a typed payload defined in a contracts crate (never inline JSON): cohort_run_id, the subject (application_id, household_id), the programs, the pinned as_of + corpus/ruleset hash (§D8), the typed trigger classification (§D9), and the stable idempotency key or its derivation inputs (§D4). It MUST obey the ADR-004 no-PII/no-FTI event allowlist (ids
status + scalars only). canopy-eligibility owns the producer AND the bulk consumer; the consumer is the enqueue trigger only and calls the existing POST /v1/eligibility/determine (canopy-eligibility’s synchronous orchestration entry — which internally fans out to each program’s signed /v1/determine, applies the EE15 hierarchy, holds the one-pending-slot 409, and assembles the CombinedResult) — so the signed-determination path is reused unchanged. mq obligations: the consumer lands in-tree before the producer activates (binding-first, #1089) or an xtask/mq-topology-allow.toml entry with a reason; canopy-eligibility’s topic-write ACL in `devstack/rabbitmq/definitions.json MUST be extended to the key (#1122) or the broker ACCESS_REFUSE`s the publish; the consumer MUST be idempotent (ADR-018 at-least-once; `event_inbox PK dedupe).

D2 — JWS signing/verification is byte-identical on the system-initiated path. The determination JWS binds only determination content + the ADR-028 snapshot_hash and carries NO per-request auth identity; verification is a pure function of (Program, kid, canonical bytes), so trust semantics are identical whether a determination is interactive or system-initiated. The async path MUST reuse the same boot-acquired program signing key + key-derived kid (ADR-036) and the same signing_key_history registration, and MUST run the full seal-hash-then-sign ritual (assemble the ADR-028 snapshot; AEAD-seal leaves under a per-determination DEK from the KEK; set snapshot_hash before signing; preserve the byte-stability invariants — micros truncation, money rescale-2, JCS canonicalization). It MUST NOT introduce a separate "system" signing key. Verify-before-accept/persist/count is preserved: an unverified determination is signature_quarantined and excluded from totals.

D3 — accessed_by for the FTI audit chain (not a signed field). The accessed_by actor feeds the ADR-014 FTI hash-chain (NOT the signature) and is MANDATORY for FTI-bearing programs (no ADR-028 snapshot without a chain row). A system-initiated run has no inbound worker, so the contract defines a well-defined principal: a real originating-worker actor when the batch was human-queued, else a reserved system principal encoding the cohort_run/job id (e.g. system:cola-redetermination:<cohort_run_id>) — never an empty or misleading "the service did it". This reaffirms the ADR-019 separation: the actor JWT (aud=canopy-internal-actor) and the determination JWS (typ=canopy-determination+jwt) are distinct token systems, keys, and registries; omitting the actor JWT has zero effect on determination verifiability.

D4 — Stable, deterministic idempotency key. The per-dispatch Idempotency-Key MUST be a deterministic hash(cohort_run_id, application/household_id, program, as_of/corpus) — stable across resend and checkpoint-resume (a replay hits the #1003 middleware), yet distinct across cohort-runs and across the COLA ruleset flip (so a post-COLA determination is not a false replay of the pre-COLA one). This replaces the orchestrator’s freshly-generated (non-stable) per-dispatch UUID-v7. Two layers, and the durable one lives in the driver: the HTTP idempotency middleware (#1003) is transport resend-safety only (24h TTL; at-least-once on crash-after-side-effect) and MUST NOT be the durable dedup layer; determination persistence MUST be idempotent on a stable natural key so a replayed unit UPSERTs/supersedes rather than duplicating (consistent with the append-only/supersede consequence, §Consequences).

D5 — Durable, driver-owned checkpoint/resume. The driver owns durable checkpoint state — a cohort-run row
per-case item rows — mirroring the renewals NOT EXISTS + ON CONFLICT domain-state idiom, surviving multi-hour / 800K-case runs and restarts; cohort selection via a persisted keyset cursor (the ADR-001 Amendment 1 B2 shape); a per-unit state machine (pending/dispatched/succeeded/failed-terminal/failed-retryable); and a single-active-run guard via advisory-lock leader election. This is contract-visible only insofar as each emitted determination is independently signed + idempotent — the exact table/schema is #1213’s byte-level.

D6 — Bounded retry + failure classification. Off the request path the deliberate single-attempt-no-retry rationale (don’t amplify a synchronous ~30-call fan-out) no longer applies. The driver MUST apply bounded retry-with-backoff + a per-job deadline + dead-letter/park, with a terminal-vs-transient split mirroring DryRunError: 4xx = terminal (do not retry); 5xx/transport = transient (bounded retry then DLQ). A re-determination-not-yet-done MUST NOT be converted into a terminal synthetic pending_verification worker-queue item; and pending_verification items MUST be deduped on (application, household, verification_type) (the verifications table has no dedup key today).

D7 — One-pending-slot interaction + interactive priority. idx_unique_pending_request (at most one live eligibility run per (application_id, household_id)) stays the race-safe backstop that maps to a 409. A bulk member that finds the slot held by a live interactive determination MUST skip-and-requeue (defer with bounded backoff) — never clobber, never surface a 409 to a worker; interactive > bulk. The driver enforces this priority as a fast-path yield before it races the INSERT, with the DB index as the backstop for a lost race. Bulk runs MUST use bounded concurrency (a semaphore / a separate low-priority lane) so a caseload-wide drain cannot starve the interactive fan-out (audit H6). A queued/deferred representation MUST exist so a slot-busy member is "waiting" rather than forced into the run-or-fail closed set; prefer an idempotent enqueue keyed on (application_id, household_id) that collapses duplicate submits into the existing handle instead of a raw 409.

D8 — as_of / corpus pinned once per cohort-run. The as_of date + the target corpus/ruleset MUST be resolved once per cohort-run and stamped on every dispatch, so all cases score the same COLA-effective day and ruleset (ADR-028/ADR-034), and folded into the idempotency key (§D4) so pre-/post-COLA determinations are correctly distinct. requested_by + the inbound authz decision MUST be captured at enqueue (the caller’s session/clock is absent at worker time); the ADR-019 service-token boundary already survives async (the orchestrator dispatches with a service identity, not the caller bearer).

D9 — Initial-vs-re-determination signal + supersession linkage (reuses ADR-028 §57). The determination envelope already carries a signed previous_determination_id: Option<DeterminationId> (ADR-028 §57, part of canonical_signing_payload, skip_serializing_if so an absent value keeps the canonical bytes identical). This amendment introduces no new field — it designates that existing signed field the async supersession linkage #1133 uses to pick which live enrollment to adjust/supersede, and pins how the async path uses it: on a re-determination the producer MUST set previous_determination_id = Some(prior) before signing, so the signature binds the chain link (tamper-evident + non-repudiable; an envelope/routing-only field could be stripped or spoofed). Its None is tri-valued (a first determination, a legacy row, or a program not yet implementing supersession), so None alone MUST NOT be read as "initial"; the authoritative initial-vs-re-determination classifier is a typed trigger/reason enum (cola|fpl|ruleset-migration|change-report|renewal|worker-initiated) carried as signed provenance (also driving downstream 273.13-vs-benign routing). An already-enrolled re-determination is BLESSED as a first-class expected outcome (today it PARKs at enrollment / 409s — the bridge until #1133); effective_date is carried faithfully but the contract does NOT force immediate application (the notice-timing shift is #1133, with the 7 CFR 273.13 reduction path as #1002’s remainder). The completion events (determination.completed.snap + the orchestrator determination.completed) extend additively with the trigger classification so notices' status-based routing still resolves.

D10 — Completion / notification edge. Async submit becomes 202 + a request_id handle (vs today’s 200-carries-result); determination.completed is designated the authoritative push edge and MUST be made atomic with the CombinedResult write (closing the #477 non-atomic best-effort gap). The existing GET requests/{id}, /determinations, /results/{application_id} poll endpoints are retained — a hybrid poll+event model.

Consequences

  • A mass change (COLA / FPL / ruleset-migration) becomes N new immutable signed determinations that supersede, never mutate, their priors at caseload scale — preserving the append-only consequence (§Consequences) unchanged.

  • The black-box signed-determination trust boundary is identical on the async path; verification stays a pure function of (Program, kid, canonical bytes), and no separate "system" key is introduced.

  • Bulk runs never starve or clobber live interactive determinations (one-pending-slot deferral + bounded concurrency).

  • The doc flip surfaces the already-returned-but-undocumented 409 on POST /v1/eligibility/determine and the async collision-deferral behavior in docs/modules/ROOT/pages/api/canopy-eligibility.adoc.

  • Enrollment apply-semantics (#1133) and the driver implementation (#1213) ride separate MRs; this amendment is the contract keystone both build against. The eligibility one-pending-determination-slot (idx_unique_pending_request, existing; interaction owned by #1213) and the enrollment one-live-enrollment-per-household #1130 fence (#1133) are distinct layers and MUST NOT be conflated.

Amendment 2 — The D4/D8 policy identity is the composite target (#1467, 2026-08-14)

Status unchanged (still Accepted). In-document amendment to Amendment 1’s D4 + D8 wording: both said "corpus" where the policy identity is really the composite {corpus_hash, params_digest, effective_period} (canopy_common::policy_target::PolicyTarget — ADR-028 Amendment 6). The corpus alone cannot witness an annual COLA: the rules loader deliberately skips parameter JSONs, so a parameters-only change under an unchanged corpus would neither perturb the D4 idempotency key nor be visible in the D8 pin.

  • D4: the deterministic key’s policy component is the composite target, not as_of/corpus — a post-COLA determination differs from its pre-COLA twin by params_digest even when the corpus is unchanged.

  • D8: what is resolved once per cohort-run and stamped on every dispatch is the composite target. The corpus half comes from canopy-rules GET /v1/corpus (#1469); the parameters half from the program service’s GET /v1/params/provenance (#1467). Dispatches carry it as expected_policy_target — the program service answers 409 policy_target_mismatch BEFORE any evaluation or write, and (when emit_policy_attestation is enabled) binds the same target into the signed envelope, which the #1213 core’s dispatcher MUST validate on return (its signed-provenance check — a later plan step; #1479 tracks persisting the returned target eligibility-side).

Related: #1467, ADR-028 Amendment 6, the #1213 COLA program plan.

Amendment 3 — The bulk core as built: epochs, the results ledger, and recorded refinements (#1213, 2026-08-16)

Status unchanged (still Accepted). Amendment 1’s D1–D10 shipped in the #1213 core MR with two structural refinements and eight recorded deviations, all reviewed in the core plan round (rev 4).

Structural. (1) Delivery generations vs program epochs. dispatch_generation fences DELIVERY only (event, x-canopy-bulk-generation self-call header, consumer CAS); the program_epoch is the logical execution epoch — it alone enters the per-program idempotency key and the results ledger, and it bumps only on a non-definitive completion or an operator re-arm (enact / retry-failures). A delivery re-arm therefore replays the SAME key: crash recovery converges through the idempotency cache or the ledger instead of re-executing into a consumed supersession slot. (2) cohort_case_results is the durable D4 layer, inserted in the SAME transaction as the program-determination row on the bulk arm — recovery reads THIS ledger at any generation, never eligibility-request status.

Recorded refinements (as built). 1. No idempotency-key header on the bulk self-call (H19: replays must surface as guarded-state 409s). 2. The durable dedup layer is the deterministic per-program key + the ledger, not HTTP middleware. 3. D7 is serialized admission (one per-household advisory lock on BOTH paths) plus consumer yields — bulk try-locks and never blocks interactive. 4. The D1 payload is ids-only; the durable case row is the source and the CAS validates run/case/generation/phase. 5. Superseded pre-flip re-runs narrowed to adopt-or-skip: a successor matching baseline + trigger + the pinned parameter set (digest + window) is adopted (adopted=true on the ledger; the read view exposes neither corpus nor evaluated_as_of, so the matcher is deliberately narrower than the live H2 binding — re-baselining is #1482). 6. Terminal request rows leave the partial indexes automatically and are retained as audit anchors. 7. Resume matches definitive results by (case, program, epoch) via the ledger. 8. BulkDispatchContext carries NO generation — the frozen body stays byte-stable across re-arms.

Sequencing. #1473 (snap stages its post-determination events in the determination-persist transaction) landed FIRST as its own MR — a bulk success whose NOA/enrollment/ELE events could silently vanish would have made every bulk enact unsafe.

Core-MR deviations from the frozen plan text (all reviewed in-MR): BulkRunAccepted names the field run_id (the plan draft said request_id); cohort_cases gained successor_determination_id (B9 requires the foreign successor queryable on the failures surface); provenance-rejected envelopes persist quarantine-marked (signature_verified=false + a labelled basis) so a signed-but-wrong-pins verdict can never read as ordinary; enact and retry-failures each grant a FRESH deadline window from the configured default (the create-time deadline bounds the preview phase); run creation refuses an empty cohort (422 cohort_empty — a zero-case run would wedge in previewing).

Related: #1213, #1473, #1480–#1483, the #1213 COLA program plan, and the bulk COLA scaling runbook.

Amendment 4 — The single-case Order arm (#1504/#575, epic &77, 2026-08-18)

Amendment 1’s D1 field list is now the literal wire shape: determination.requested is an internally-tagged two-arm enum — Cohort (the #1213 cohort-case projection, unchanged semantics) and Order, carrying the full D1 list verbatim: a RequestOrigin {source, ref_id} requester identity, the subject (application_id/household_id), programs, the pinned as_of (D8), the signed D9 trigger, and requested_by captured at enqueue.

  • Producers extend beyond eligibility: a program service reporting a change (canopy-medicaid’s CMD subsystem first) publishes the Order; the broker topic-write ACL extends per producer. Eligibility remains the sole consumer.

  • The durable order substrate realizes D4/D5/D6 for one-case work: determination_orders materializes idempotently on UNIQUE (origin_source, origin_ref); settles fence on the claim’s own timestamp token; retry is backoff-scheduled (60s doubling, 1h cap, jitter) through an always-on 1-minute sweep — never hot MQ redelivery — with a deliberately-larger attempt cap (12) so slot-conflict chains cannot terminalize a legal re-determination quickly; the D4 deterministic dispatch key is hash("canopy-order-dispatch-v1\0" ‖ origin ‖ program ‖ epoch) (KAT-pinned) and crash recovery ADOPTS the latest DEFINITIVE completed run by origin instead of re-executing.

  • The D10 push edge carries the correlation: DeterminationCompletedV1 additively gains an origin echo, so the requester settles its own row — pure event choreography, no requester-side HTTP retry loop.

  • Boot consequence: the determination.requested consumers attach on every canopy-eligibility boot (no longer gated on bulk_runs_enabled); CANOPY_MQ_PREFETCH_COUNT=1 is a fleet-wide boot requirement.

  • The Decision text and Amendments 1–3 are left byte-immutable — this amendment extends, it does not supersede.

Edit this page · default