Plan: Eligibility Request Idempotency + Composition Graceful Degradation

On this page
NOTE

This plan was authored from a code-grounded investigation, not from the issue text. Both issues describe the symptom correctly but prescribe the wrong cause/fix; the corrected reality is recorded in #588 — request idempotency. Read it before implementing — the issue bodies will mislead you.

  • #588 frames the bug as a "second-click race". It is not — the orchestrator’s determine() is fully synchronous, so two HTTP calls cannot interleave the way the issue imagines. The real defect is an unconditional stale-row sweep plus a missing failed transition on the orchestrator’s error paths.

  • #658 asks for "drop unknown plugins". It must be layer-aware: drop unknown slugs introduced by the role/user override layers, but keep the existing hard-fail for the jurisdiction baseline + jurisdiction_live layers (an existing test asserts the latter).

Status

Step Description Status

#588 — eligibility request idempotency

1

canopy-eligibility store: TTL-gate the stale-row sweep; map the idx_unique_pending_request unique violation to ApiError::Conflict (409).

Done (2026-06-04) — TTL-gated sweep + STALE_REQUEST_TTL; unique-violation→Conflict via map_create_error.

2

canopy-eligibility orchestrator: extract determine() body into determine_inner; wrapper marks the request failed (best-effort) on any error path.

Done (2026-06-04) — determine_inner extracted; wrapper records failed best-effort on error.

3

DB-backed integration tests: live-request conflict returns 409; retry-after-error succeeds (no stuck in_progress); stale row past TTL is swept.

Done (2026-06-04) — request_idempotency_test.rs, 4 tests (collision / 409 / stale-sweep / failed-on-error).

#658 — composition graceful degradation

4

canopy-composition loader: snapshot trusted (baseline + jurisdiction_live) slug-set, then drop unknown role/user-layer slugs in Steps 9/11 instead of hard-failing.

Done (2026-06-04) — Step 6 split + collect_item_slugs snapshot; Step 9 layer-aware drop + tracing::warn!.

5

Tests: unknown role/user slug → dropped (surface still loads); unknown baseline/jurisdiction_live slug → still UnknownPlugin; dropped slugs are logged/metered.

Done (2026-06-04) — 2 loader tests + existing guardrail; warn! (no metrics crate in canopy-composition).

shared

6

Docs + CHANGELOG + GitLab issue updates (point #588/#658 at this plan).

Done (2026-06-04) — CHANGELOG + plan status; issues updated on merge.

Issues: #588, #658
Branches: fix/588-eligibility-request-idempotency, fix/658-composition-graceful-degradation (ship as two MRs — they touch disjoint crates and have independent verification)

Context

These are two unrelated correctness defects grouped into one plan because each is a small, well-scoped backend fix and both block clean operation of the worker portal under real (non-happy-path) conditions. They share no code.

#588 — Every POST /v1/eligibility/determine first runs an unconditional UPDATE … SET status='failed' WHERE status IN ('pending','in_progress') over the (application_id, household_id) pair before inserting the new request row (services/canopy-eligibility/src/store/mod.rs). The idx_unique_pending_request partial unique index permits at most one active request per pair, and the sweep exists to free that slot when a prior request was orphaned. But the sweep is indiscriminate: it also clobbers a genuinely live determination if a second request arrives while the first is still running, marking it failed even though it will go on to complete. The root cause of the orphans the sweep papers over is separate: the orchestrator marks a request in_progress and (on success) completed, but has no failed transition on any of its error/early-return paths, so any determination that errors leaves a stuck in_progress row forever.

#658 — The composition loader (crates/canopy-composition/src/loader.rs) resolves a 5-layer override stack (system defaults → jurisdiction baseline TOML → jurisdiction_live → role → user) into a dashboard or case-detail layout. Step 9 and Step 11 hard-fail with CompositionLoadError::UnknownPlugin (→ HTTP 500) the moment any item references a plugin slug that isn’t registered. That is correct for the trusted layers (a typo in the jurisdiction baseline should be loud), but for the user and role override layers it means one stale customization — e.g. a worker hid/kept a panel that was later renamed or removed from the build — takes the worker’s entire dashboard down with a 500. The surface should drop the unknown override item and render the rest.

This plan supports the SNAP UAT worker-portal reliability bar; neither fix changes an API contract.

Scope

In scope:

  • #588: TTL-gated sweep + Conflict (409) on concurrent active request + failed-on-error in the orchestrator + DB integration tests.

  • #658: layer-aware graceful degradation in the composition loader read path (drop unknown role/user slugs; keep hard-fail for baseline + jurisdiction_live) + observability for dropped slugs + tests.

Out of scope:

  • Asynchronous/queued determinations (#588’s "race" framing assumes them; the orchestrator is and stays synchronous). No queue, no background worker.

  • Changing the idx_unique_pending_request index definition or the request state machine beyond adding the failed-on-error transition.

  • Composition write-path validation changes — PUT /v1/composition/{surface}/user/me already rejects out-of-baseline slugs with 422 per ADR-024 (services/canopy-web/src/api/composition.rs). #658 is a read-path resilience fix for rows that became stale after a valid write (plugin later renamed/removed). The write path is unchanged.

  • Any change to canopy-rules, program services, or the JWS determination contract.

Design

#588 — request idempotency

Current code (verified against main):

  • create_eligibility_request — services/canopy-eligibility/src/store/mod.rs:10-59. Opens a txn, runs the unconditional UPDATE … status='failed' WHERE status IN ('pending','in_progress') sweep (lines 28-40), then INSERT … status='pending' (lines 42-55), then commits. The idx_unique_pending_request partial index is the safety net the sweep is feeding.

  • determine — services/canopy-eligibility/src/orchestrator.rs:568-1124. Calls create_eligibility_request (574-582, map_err → ApiError::internal), marks in_progress (584-586), then runs the whole determination (588-1108), then marks completed (1111-1113), then returns Ok(DetermineResponse) (1115-1123). Every ?/return Err(…​) between line 588 and line 1113 leaves the row stuck in_progress — there is no failed transition anywhere on an error path. Those stuck rows are exactly what the unconditional sweep was added to clear.

  • determine() is part of the library contract — services/canopy-eligibility/src/lib.rs documents that integration tests call orchestrator::determine directly, so its signature must not change.

  • ApiError::Conflict(String) already exists and maps to HTTP 409 (crates/canopy-common/src/error.rs:28,95).

The fix has two halves and both are required. TTL-gating the sweep without adding the failed-on-error transition would regress: a determination that errored would leave a stuck in_progress row that the now-TTL-gated sweep won’t clear until the TTL elapses, turning an immediate legitimate retry into a misleading 409.

Half 1 — store::create_eligibility_request

Change the sweep from unconditional to TTL-gated, and surface the unique-index collision as a typed conflict instead of pre-clearing live rows:

  1. Add a stale-request TTL constant, generously larger than the maximum determination wall-clock (the orchestrator dispatches to program services in parallel under per-call timeouts/circuit-breakers; a determination completes in seconds). Use 5 minutes:

    /// A `pending`/`in_progress` request older than this is presumed orphaned
/// (process crash / panic mid-dispatch) and may be swept. Must exceed the
/// maximum determination wall-clock so a genuinely live request is never
/// reclassified as stale. See plan eligibility-and-composition-correctness.
const STALE_REQUEST_TTL: chrono::Duration = chrono::Duration::minutes(5);

  2. Gate the sweep UPDATE with AND requested_at < $3 where $3 = Utc::now() - STALE_REQUEST_TTL. A live concurrent request (younger than the TTL) is therefore left untouched.

  3. Keep the INSERT. When a live request still holds the partial-index slot, the INSERT now fails with a Postgres unique violation on idx_unique_pending_request. Return a typed error so the orchestrator can map it to 409. Two acceptable shapes:

    • return sqlx::Error as today and let determine() inspect it (see Half 2), or

    • change the return type to a small store-level error enum. Prefer the first (least churn, and determine() already owns the map_err).

Keep the existing txn wrapper (sweep + insert stay atomic).

Half 2 — orchestrator::determine

  1. Map the conflict. At the create_eligibility_request call site (orchestrator.rs:574-582), replace the blanket map_err(|e| ApiError::internal(…​)) with one that inspects the error: a unique violation on idx_unique_pending_request becomes ApiError::Conflict("a determination for this household is already in progress".into()); anything else stays ApiError::internal. Detect the constraint via sqlx::Error::Database(db_err) → db_err.constraint() == Some("idx_unique_pending_request") (sqlx DatabaseError::constraint).

  2. Add the failed-on-error transition by extracting the determination body into a private helper so the wrapper can observe the Result:

    pub async fn determine(
    db: &PgPool,
    cfg: &DetermineConfig<'_>,
    request: DetermineRequest,
) -> Result<DetermineResponse, ApiError> {
    let elig_request = store::create_eligibility_request(
        db, request.application_id, request.household_id,
        &request.programs, &request.requested_by,
    )
    .await
    .map_err(map_create_error)?;          // <- unique-violation → Conflict

    store::update_request_status(db, elig_request.id, "in_progress")
        .await
        .map_err(|e| ApiError::internal("update status", e))?;

    let outcome = determine_inner(db, cfg, request, &elig_request).await;
    if outcome.is_err() {
        // Best-effort: a stuck `in_progress` row is exactly what the TTL
        // sweep was papering over. Failure to record `failed` must not
        // mask the real error, so log and swallow.
        if let Err(e) = store::update_request_status(db, elig_request.id, "failed").await {
            tracing::warn!(request_id = %elig_request.id, error = %e,
                "failed to mark eligibility request failed after determination error");
        }
    }
    outcome
}

async fn determine_inner(
    db: &PgPool,
    cfg: &DetermineConfig<'_>,
    request: DetermineRequest,
    elig_request: &EligibilityRequest,
) -> Result<DetermineResponse, ApiError> {
    // verbatim body of the old determine(), lines 588-1123:
    // fetch_household_context … dispatch … create_combined_result …
    // update_request_status(..,"completed") … Ok(DetermineResponse { … })
}

    The extracted body is unchanged — same indentation level (it was already at fn-body indent), uses request by value and elig_request.id/&elig_request exactly as before. The "completed" transition stays inside determine_inner (it only runs on the success path, which is correct). Do not reindent; this keeps the diff reviewable and the git blame legible.

NOTE
the in_progress update stays in the wrapper (before determine_inner) so that if it ever fails there is no half-created row to reconcile — create already succeeded, so a failure there returns ApiError::internal with the row left pending, which the TTL sweep reclaims. Putting in_progress inside determine_inner would also work but muddies the "wrapper owns lifecycle, inner owns work" split.

#658 — composition graceful degradation

Current code (verified against main):

  • load_composition — crates/canopy-composition/src/loader.rs:112+. Step 5 merges the jurisdiction baseline TOML (162-178). Step 6 (185-206) applies the DB layers in one loop in jurisdiction_live → role → user order (the SQL returns them ordered); for dashboard surfaces the user layer is deferred to Step 10.5, but jurisdiction_live and role (and, for case_detail, user) are all applied here into the working JSON. Step 7 (209) deserializes working into RawComposition; for case_detail, Step 7’s tail (217-219) moves raw.sections → raw.items.

  • Step 9 (230-253) iterates raw.items and returns UnknownPlugin if find_panel/find_case_section misses. Step 11 (282-312) re-looks-up each item (ok_or_else → UnknownPlugin) to read allowed_spans. Both are unconditional hard-fails.

  • Step 10 (256) is the role permission filter (silent-drop of items the role can’t see) — already graceful, unrelated.

  • ComposedItem/raw.items carry no layer provenance — once layers are merged you cannot tell which layer introduced a given slug.

  • Existing contract test loader_post_merge_unknown_plugin_rejects (crates/canopy-composition/tests/loader_test.rs:385-419) inserts an unknown slug into the jurisdiction_live layer and asserts UnknownPlugin. This must keep passing — it is the guardrail that the trusted layers stay loud.

Design — snapshot the trusted slug-set, then drop unknown override slugs:

Because items lose layer provenance at merge time, classify by snapshotting the trusted slug-set before the untrusted layers are applied. The trusted layers are: system defaults + jurisdiction baseline TOML + jurisdiction_live. The untrusted layers are: role + user.

  1. Split the Step 6 loop by layer. Apply only the jurisdiction_live (and lower — defaults/baseline already in working from Steps 4-5) patches, then take a snapshot, then apply role (and, for case_detail, user) patches. Concretely: iterate db_layers; for layer.layer == CompositionLayer::JurisdictionLive apply immediately; collect role/user layers into a deferred vec and apply them after the snapshot. (The dashboard user layer is already deferred to Step 10.5 — leave that exactly as-is.)

  2. Snapshot helper. After the jurisdiction_live patches are applied to working, extract the set of item slugs into trusted_slugs: HashSet<String>. Add a small helper that reads the slug list per surface from the JSON value — dashboards/sign-in read working["items"], case_detail reads working["sections"] (because the sections→items move hasn’t happened yet at this point). Each entry’s slug is its "item" field.

    fn collect_item_slugs(working: &serde_json::Value, surface: ComposableSurface) -> HashSet<String> {
    let key = match surface {
        ComposableSurface::CaseDetail => "sections",
        _ => "items",
    };
    working.get(key).and_then(|v| v.as_array()).into_iter().flatten()
        .filter_map(|item| item.get("item").and_then(|s| s.as_str()).map(str::to_owned))
        .collect()
}

  3. Step 9 — drop, don’t fail, for untrusted slugs. When find_panel/find_case_section misses:

    • if the slug ∈ trusted_slugs → return Err(UnknownPlugin { slug }) (unchanged — keeps the baseline/jurisdiction_live guardrail and the existing test green);

    • else (introduced by role/user) → mark the item for removal and record it in a dropped: Vec<String>.

      Collect-then-retain (don’t mutate raw.items while iterating): build the keep/drop decision, then raw.items.retain(…​). After the loop, if !dropped.is_empty(), tracing::warn!(surface = ?surface, role = %role.0, ?dropped, "dropped unknown override-layer composition items") and bump a counter metric (e.g. composition_items_dropped_total, label surface) so silent degradation is observable per the project’s "no silent caps" rule.

  4. Step 11 — operate on survivors only. Because Step 9 already removed unknown untrusted items, Step 11’s find_panel/find_case_section lookups can only miss on a trusted slug, so its existing ok_or_else(UnknownPlugin) is now correct as-is for the trusted case. Keep it. (Belt-and-suspenders: a trusted unknown slug that slipped past Step 9 still hard-fails here, which is the desired loud behavior.)

  5. Ordering note — case_detail user layer. For case_detail the user layer is applied in the (now-split) Step 6 after the snapshot, so a user add of an unknown slug is correctly classified untrusted and dropped. For dashboards the user layer is user_delta_v1 (Step 10.5), which can only hide/reorder/respan existing baseline panels — it cannot introduce a new slug — so no additional handling is needed there.

Observability: a dropped override item is a real (if benign) signal that a worker/role customization has gone stale. The warn! + counter is the surface for an operator to notice and re-run Studio cleanup. Do not drop silently.

Steps

Step 1: TTL-gated sweep + typed conflict (#588 store)

Files: services/canopy-eligibility/src/store/mod.rs

Add STALE_REQUEST_TTL const. Gate the sweep UPDATE with AND requested_at < (now - TTL) (bind Utc::now() - STALE_REQUEST_TTL). Keep the txn + INSERT. Leave the return type as Result<EligibilityRequest, sqlx::Error> so the unique violation propagates to the orchestrator. Update the doc-comment to describe TTL-gating + the 409 contract.

Step 2: determine_inner extraction + failed-on-error (#588 orchestrator)

Files: services/canopy-eligibility/src/orchestrator.rs

Add fn map_create_error(e: sqlx::Error) → ApiError mapping the idx_unique_pending_request constraint violation → Conflict, else internal. Extract lines 588-1123 into async fn determine_inner(db, cfg, request, elig_request); rewrite determine() per the snippet in #588 — request idempotency. Do not change determine()’s public signature. Best-effort `failed transition logs-and-swallows on its own error.

Step 3: #588 DB integration tests

Files: services/canopy-eligibility/tests/ (add or extend the orchestrator/store integration test module; follow the existing #[sqlx::test]/devstack-pool pattern in that directory)

Cover:

  • Concurrent active request → 409: create a request, leave it in_progress, call create_eligibility_request again for the same pair within the TTL → expect the unique violation; via determine (or map_create_error) → ApiError::Conflict.

  • Retry after error → success: simulate a determination that left a row in_progress, advance past the TTL (insert with a back-dated requested_at), then a new create_eligibility_request sweeps it and succeeds.

  • failed-on-error: drive determine() down an error path (e.g. unreachable persons URL via DetermineConfig) and assert the request row ends failed, not in_progress.

Step 4: layer-aware graceful degradation (#658 loader)

Files: crates/canopy-composition/src/loader.rs

Split the Step 6 DB-layer loop (apply jurisdiction_live, snapshot trusted_slugs, apply role/case_detail-user). Add collect_item_slugs. Rewrite Step 9 to drop untrusted unknown slugs (retain + dropped vec) and keep the trusted hard-fail. Add the warn! + composition_items_dropped_total counter. Leave Step 10/10.5/11 logic intact.

Step 5: #658 tests

Files: crates/canopy-composition/tests/loader_test.rs

  • New: unknown slug in a role-layer patch → surface loads, the unknown item is absent, other items present.

  • New: unknown slug in a user-layer patch (use case_detail so the user layer is RFC-6902 and can add) → dropped, surface loads.

  • Unchanged: loader_post_merge_unknown_plugin_rejects (jurisdiction_live unknown) still asserts UnknownPlugin — confirm it passes untouched.

  • Assert the dropped-item counter increments (if the test harness exposes the metrics registry; otherwise assert via the warn!-path side effect / the survivor set).

Step 6: docs + issue updates

  • CHANGELOG.adoc — one entry under == Unreleased per fix.

  • Antora: note the 409 contract on the eligibility determine endpoint page (docs/modules/ROOT/pages/api/canopy-eligibility.adoc) and the graceful-degradation behavior on the composition page.

  • Update GitLab #588 and #658 to reference xref:plans/archive/eligibility-and-composition-correctness.adoc and correct their wrong framing (link the Design NOTE).

Files Touched

File Change

services/canopy-eligibility/src/store/mod.rs

TTL-gate the sweep; STALE_REQUEST_TTL const; doc-comment.

services/canopy-eligibility/src/orchestrator.rs

map_create_error; extract determine_inner; failed-on-error wrapper.

services/canopy-eligibility/tests/…

Conflict / retry-after-error / failed-on-error DB tests.

crates/canopy-composition/src/loader.rs

Trusted-slug snapshot; layer-aware drop in Step 9; dropped-item observability.

crates/canopy-composition/tests/loader_test.rs

Role/user unknown-slug drop tests; keep jurisdiction_live reject test.

CHANGELOG.adoc, Antora api pages

Changelog + 409/graceful-degradation docs.

Verification

  1. cargo nextest run -p canopy-eligibility -p canopy-composition --lib — unit tests pass.

  2. cargo xtask dev refresh then the canopy-eligibility + canopy-composition integration tests against the devstack pool (CANOPY_PORT_POSTGRES_5432=<docker ps port> cargo test -p canopy-eligibility, likewise composition) — DB-backed conflict/retry/drop tests pass.

  3. cargo nextest run --workspace — no regressions.

  4. cargo xtask validate — fmt + clippy + docker build clean.

  5. Manual: two rapid POST /v1/eligibility/determine for one household → first 200, second 409 (not a clobbered first). A dashboard with a stale user-layer panel slug renders (minus the stale panel) instead of 500.

Documentation Updates

  • CHANGELOG.adoc — entries under == Unreleased.

  • Antora api/canopy-eligibility.adoc (409 contract) + composition page (graceful degradation).

  • Service Catalog — only if the eligibility route table’s error-code column is maintained there.

  • GitLab #588 / #658 — link this plan, correct framing.

Edit this page · default