Plan: SSR aggregate request deadline + honest degraded states (#1306, epic &73)

On this page
NOTE

Design selected via a comparative multi-design review, then hardened by an external review whose blocking findings (absolute-cutoff model, permit-through-decode, guard race, write exclusion, honesty coverage, config bounds) are folded in as rev 3. Scope is narrowed on-issue (#1306) to the six covered read routes + machinery; everything else is filed as #1319–#1326.

Status

Step Description Status

0

Pre-implementation: follow-ups #1319–#1326 filed + related; #1306 AC narrowed on-issue; this plan committed + nav-linked.

Done (2026-08-04) — this MR

1

MR1 feat(web): request deadline + typed service-error outcomes — deadline.rs, ServiceError overhaul + central classifier + safe Display, additive retry.rs Exhausted-class field, read-verb enforcement (guard/gate/cutoff), bounded token acquisition + auth_unavailable, config knobs, telemetry module. No handler stamps ⇒ no render-path change.

Done (2026-08-04) — !1071 (impl f6d19147, merge b61e72cd)

2

MR2 fix(web): honest dashboard + cases states under the deadline — stamp dashboard/cases/search/panel-fragment; my_queue SourceOutcomes (per-leg renewals)
shared queue_state(); panel audit (my_queue + recent_determinations); hero "—"; /cases/search typed row outcomes; harness build-out.

Done (2026-08-04) — !1072 (impl 5ace7fb9, merge 1b213755)

3

MR3 fix(web): case-detail deadline, manifest activation, section honesty, telemetry wiring — stamp case-detail + tabs; manifest timeout_ms on both dispatch paths; hero concurrent + honest; ?program=all per-row; the 15 COLLAPSE-section fixes (inventory below); telemetry wiring; docs; closes #1306.

Done (2026-08-04) — this MR

Epic: &73
Issue: #1306 (high)
Branches: feature/1306-ssr-deadline-mr1 → mr2 → mr3

Context

canopy-web SSR handlers block the HTML response on un-budgeted upstream fan-out. Only the dashboard panel dispatcher sets any budget — per call, not per chain. Case-detail sections ignore their manifest timeout_ms=5000. Serial phases accumulate unbounded (my_queue’s 3 sources + ≤20-call name resolution; case-detail’s serial pre-phase, section fan-out, then serial hero; the fully serial determination assembler). Token acquisition (10s/15s internals) sits outside every budget. One hung upstream stalls the page into the 15s e2e navigation ceiling.

Compounding it, upstream Err frequently renders as false success: my_queue drops every source error and shows "All caught up"; the hero shows .unwrap_or(0); ?program=all collapses errors into "No determinations". The inventory below found 15 of 23 case-detail sections with at least one silently-collapsing leg. In a benefits system that is dangerous, not merely wrong.

Requirements (issue #1306, reconciled spec): aggregate absolute deadline — not per-call budgets; preserve completed partials (no abort-that-discards); distinct timeout/partial/error/genuine-empty states; token acquisition inside the bound; activate the existing manifest timeout_ms contract; shared fan-out ceiling; categorical redaction-safe telemetry; handler-level fault tests; no pool change; never raise the 15s navigation budget.

Design

D1. Deadline type and the one-cutoff model

New services/canopy-web/src/deadline.rs:

// SPDX-License-Identifier: AGPL-3.0-or-later
#[derive(Clone, Copy, Debug)]
pub struct RequestDeadline { deadline: std::time::Instant }

  • from_now(budget) saturates at a hard 600s ceiling (no checked_add panic path even under the config override).

  • All math in pure now-parameterized functions (remaining_at(now), clamp_at(now, cap)) — proptested without wall clock; wall-clock methods are thin wrappers. std::time::Instant throughout (same clock domain as canopy_api::retry).

  • MIN_CALL_FLOOR = 50ms: below the floor a call cannot succeed — skip it, typed.

The cutoff model: each logical call computes one absolute call_deadline = min(page_deadline, verb_entry_now + component_cap) where component_cap = call_timeout | CLIENT_DEFAULT_TIMEOUT (5s). The winning min arm is recorded for DeadlineExceeded-vs-Timeout classification (page arm wins ties). Every stage derives its residual from that same absolute point, recomputed at the boundary: gate acquire, each retry attempt’s reqwest timeout, the retry policy’s overall bound, and the body/decode reads. Time can never "reset" across queueing, retries, or the headers→body seam.

Stamping: top-level read handlers only — get_dashboard, get_case_search, search_cases, get_case_detail, get_tab, get_panel_fragment. Nested helpers (composed-tab fallback, render_cross_program_summary, section renderers) inherit via the clients and never re-stamp (a second stamp would mint a second gate — forbidden, doc-commented). Stamp before identity: clients.with_deadline(d).with_service_identity(&svc).await (order tested). /cases and /cases/search gain the Extension<Arc<WebConfig>> they lack today.

D2. Threading — fields on the existing clone chain

  • InternalClient += deadline: Option<RequestDeadline>, page_gate: Option<Arc<Semaphore>>, auth_unavailable: bool; ServiceClients += mirror deadline field.

  • ServiceClients::with_deadline(d) clones all 15 clients (the with_timeout pattern) and creates one shared Arc<Semaphore>(UPSTREAM_FANOUT_LIMIT = 8).

  • with_timeout/with_token are field-preserving builders — the deadline survives the manifest with_timeout clone, rides SectionContext.clients, reaches every fetcher with no signature changes. test_service_clients + other ServiceClients literals gain the new fields (mechanical).

D3. Enforcement — read verbs only, one absolute cutoff, permit held through decode

Reads only. Guard/gate/clamp apply to get, get_terminal_status, and get_raw_streaming (header phase). Write verbs ignore the deadline entirely — a timed-out write that committed upstream is exactly the ambiguity the client docs warn about; no stamped handler writes today; bounded writes are #1320.

Un-stamped = untouched. With no deadline, the send paths run today’s exact match self.call_timeout code (incl. `put_idempotent’s retry headroom). Opt-in adoption per handler — not a compatibility layer.

With a deadline, per logical read call:

Stage Rule

Entry guard

residual < MIN_CALL_FLOOR ⇒ typed DeadlineExceeded, zero I/O. Unwinds serial chains in microseconds; completed work kept.

Gate

Acquire one owned permit for the whole logical call via timeout(residual, acquire_owned()) ⇒ typed DeadlineExceeded on lapse. Held through retries, status check, body read and decode; released at end. Holding through decode is what makes "hard 8" true; bounded by call_deadline ≤ ~5s.

Post-acquire recheck

Residual recomputed after the permit wait and before each attempt; below floor ⇒ typed fast-fail, zero server hits (closes the pass-then-queue race).

Per-attempt clamp

Attempt’s reqwest timeout = residual at that attempt’s start (covers connect→body for the attempt).

Retry policy

overall = residual at retry-loop start; per_attempt = min(residual, CLIENT_DEFAULT_TIMEOUT) — explicit per-attempt wins as-is in the engine, so attempt 1 gets the full budget (kills the ÷3 false-cancel); fast transients keep their #1270 second chance; the hard overall wrap guarantees the span.

Body/decode

json()/text()/bytes() wrapped in timeout(residual_at(now)) — the headers→body seam cannot mint fresh time.

  • No component-level tokio::timeout anywhere: page machinery never drops a future; completed partials render by construction.

  • Leaf-only gating is deadlock-free (a permit is never held while awaiting another permit) and collapses today’s 8×8=64 nested-fan-out hole to a true 8 per request. `bounded_join’s private per-join semaphores stay (width shaping only).

  • Streaming: get_raw_streaming takes guard + gate through the header phase, then releases the permit and is exempt from the body cutoff (a download may outlive a page render; separate policy, doc-commented). No stamped surface streams today.

D4. Token acquisition — bounded, and honest downstream

with_service_identity bounds the acquire at min(TOKEN_ACQUIRE_CAP = 5s, residual). On timeout/failure with a deadline present, it sets auth_unavailable = true on the cloned clients and every read leaf fast-fails with typed kind = AuthUnavailable, zero I/O ⇒ honest error panels + one token_timeout telemetry event — never an N×401 storm. Un-stamped path keeps today’s proceed-unauthenticated behavior (MR1 inertness); unification is #1324.

Ordering consequence (found in implementation, correct behavior): the D3 entry guard runs before the auth check, so a token wait that consumes the whole page budget makes subsequent reads classify DeadlineExceeded (time truly is exhausted), while a token failure with budget remaining classifies AuthUnavailable. Both arms are pinned by separate tests (hung-IdP ⇒ bounded return + deadline-class fast-fails; refused-IdP ⇒ AuthUnavailable with budget remaining and zero hits).

D5. ServiceError overhaul — typed, safe by default, centrally classified

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ServiceErrorKind {
    DeadlineExceeded, Timeout, Transport, Http(u16),
    Exhausted { last_status: Option<u16>, timeout_class: bool },
    Decode, AuthUnavailable,
}
pub struct ServiceError { service: &'static str, kind: ServiceErrorKind, diagnostic: String }

  • Private fields; constructors (::http, ::timeout, ::deadline, ::transport, ::exhausted, ::decode, ::auth_unavailable) + kind()/service()/diagnostic() accessors; is_time() = DeadlineExceeded | Timeout.

  • Display is safe: "{service} service error: {kind category}" — the upstream body moves to diagnostic, logged once server-side at the construction site. Fixes the two live sites rendering ServiceError::to_string() to users (actions_snap_issuance.rs:37, actions_intake.rs:92); aligns with ADR-041.

  • upstream_status() matches Http(s) (string-prefix parse deleted); Exhausted returns None — preserving the #594 exhaustion⇒502 contract (pinned by the existing 3-attempts-on-503 test) while telemetry sees exhausted distinctly.

  • Central classifier classify_reqwest(&reqwest::Error) → ServiceErrorKind used by every error site (retried + single-shot verbs, status_error, json_or_status_error, raw/terminal-status/body reads) — a body-read timeout is never misfiled as Decode.

  • Precedence: non-2xx headers ⇒ Http(status) even if the error-body read then times out; 2xx + body timeout ⇒ Timeout-class; page-vs-component tie ⇒ DeadlineExceeded.

  • One additive retry.rs change (canopy-api): RetryError::Exhausted records the last failure’s transport class (timeout/connect/other) instead of stringifying it. Additive field, own test.

D6. Honest states — inventoried across every stamped surface

Bounding calls without state work would increase false-empty renders, so every stamped surface gets an explicit decision. Render vocabulary stays the four-state manifest contract — timeout is state="error" with distinct static copy (is_time() ⇒ "Took too long to load — try again" / else "Couldn’t load right now") through the existing panel_error/error_block machinery; the 5-way distinction (incl. partial) lives in telemetry.

my_queue (MR2)

fetch_items → (Vec<WorkQueueItem>, SourceOutcomes); sources = applications, renewals (per-leg: Partial { failed, of } when some program legs fail), appeals: SourceOutcome { Ok, Partial{..}, Failed(kind) }. One shared queue_state() consumed by both the dashboard panel and GET /cases:

Sources Items Renders

all failed

—

error — never "All caught up"

any failed/partial

≥1

populated + degraded banner ("Some queue sources didn’t load — this list may be incomplete." + retry)

any failed/partial

0

error — unknowable ≠ empty

none failed

0

empty — earned

none failed

≥1

populated

Name-resolution misses stay cosmetic. Third caller command_palette.rs:117 destructures-and-ignores (per-keystroke, no state UI; doc-commented).

Other MR2 surfaces

  • Panel audit: exactly two panels COLLAPSE (inventory below) — my_queue and recent_determinations (collapse at recent_determinations.rs:57/:80, masked as "empty" at :107; gains the three-way split + per-program partial "—" rows). The 12 three-way panels + 4 visible-degrade panels are verified, not rewritten.

  • Dashboard hero: fetch_hero → Result; Err ⇒ "—" (never a fabricated 0).

  • /cases/search rows: PersonResult gains typed ancillary outcomes (status lookup
    household-program lookup: Ok(value) | Failed(kind)) — a failed status lookup renders "Unavailable — retry", distinct from the genuine "Pending"; a failed program lookup renders distinct from "—". Primary person rows and links always render; degradation adds a banner above results, never replaces them.

  • templates/cases/search.html gains state arms (today only {% if !queue_items.is_empty() %}).

Case-detail (MR3)

  • Hero: (a) runs concurrently with the section fan-out (tokio::join!, the dashboard’s shape) so a slow section wave cannot starve it — enabled by dropping the unused _sections param; (b) honesty inside: member count "—" on persons failure (today fabricates 0 at case_detail.rs:1664), determination/certification/ELE failures render degraded lines instead of silent suppression; SNAP-cert fetch joins the concurrent block.

  • ?program=all: per-row matrix honesty — ProgramDetOutcome { Determined(..), NoneFound, NotConfigured, Failed { is_time } }. Ok-empty ⇒ "No determinations" (a SNAP error must not silence TANF’s genuine empty); NotConfigured driven by the existing service_configured signal (case_detail.rs:2022-2027); Err ⇒ error/timeout row. tr[data-program=…] attrs kept.

  • Sections: RenderedSection/RenderedPanel gain outcome: ComponentOutcome set explicitly by each fetcher; every COLLAPSE row in the inventory below is fixed — each silently-collapsing leg either maps to an explicit outcome/degraded render or carries // SILENT-OK: <reason>. Audit oracle: the .ok()/unwrap_or_default()/if let Ok grep sweep + per-fixed-section mixed-feed tests.

D7. Manifest timeout_ms activation — both dispatch paths

  • sections::dispatch_fetch: resolve the manifest via ctx.plugins.find_case_section (the injected source — not the panel dispatcher’s hardcoded CompileTimePluginSource, which would bypass test plugin sources) and scope ctx.clients.with_timeout(manifest_ms).

  • The 13 explicit tab arms in get_tab call renderers directly and never pass dispatch_fetch — a shared helper section_scoped_clients(plugins, slug) applies the same manifest cap before each arm’s renderer.

  • DEFAULT_PANEL_TIMEOUT_MS hoists to one shared DEFAULT_COMPONENT_TIMEOUT_MS.

  • Semantics unchanged from #527 (a per-call cap); it becomes component_cap in the D1 cutoff.

D8. Config — two knobs, bounded, overridable

#[serde(default = "default_page_deadline_ms")]     // = 12_000
pub page_deadline_ms: u64,
#[serde(default = "default_fragment_deadline_ms")] // = 8_000
pub fragment_deadline_ms: u64,

  • Two budgets because the ceilings differ: full pages render under the 15s navigation budget (12s + ~3s margin); htmx fragments (get_tab, panel retry, /cases/search) render under the 10s htmx response wait (tests/e2e/lib/helpers.ts:7) ⇒ 8s.

  • Boot validation: page ∈ [1_000, 14_000], fragment ∈ [1_000, 9_000]; out-of-range rejects at boot — unless CANOPY_WEB__DEADLINE_OVERRIDE=true (per-control accountable override; deployment owns the risk; loud warn every boot; from_now still hard-caps at 600s). Raising budgets is never the fix for a slow upstream.

  • Production default in config/canopy-web/default.yaml; WebConfig literal in composition_api_test.rs:144 updated; deadline.rs wired into the module tree. Tests may construct WebConfig literals directly with sub-minimum values (validation runs at env load).

  • configuration-reference.adoc lands in MR1 with the knobs.

D9. Telemetry — services/canopy-web/src/telemetry.rs, exporter-tested

Prereq: opentelemetry workspace dep added to canopy-web (none today). Singleton instruments via OnceLock; meter global::meter("canopy_web").

Instrument Recording point Labels

canopy_web.upstream.call_outcome (Counter)

Once per logical call, in the verb tail after status+decode (recording in send_retrying would mislabel returned 4xx as ok)

service; kind ∈ {ok, deadline_exceeded, timeout, transport, http_4xx, http_5xx, exhausted, decode, auth_unavailable, token_timeout}

canopy_web.component.outcome (Counter)

Fetchers/finalize helpers via record_component(surface, slug, outcome); surfaces without RenderedPanel/Section (heroes, search, program-all rows, explicit tabs) call it directly

surface {dashboard, cases, case_search, case_detail, tab, program_all}; component = slug normalized against the compile-time plugin registry (unknown ⇒ "unknown"); outcome {populated, empty, partial, error, timeout}

canopy_web.page.outcome (Histogram: duration_ms + remaining_ms)

Every handler exit path (scopeguard at handler top — early errors included)

surface; outcome {ok, degraded, error}

Redaction boundary is structural: only kind names + registry slugs cross into metrics; diagnostic never does. Tests use a manual exporter/reader asserting exact counts and the full label vocabulary.

Erratum (2026-08-04, MR3): the registry-only normalization above would collapse the non-plugin components (hero, search rows, program-all rows) to "unknown", defeating their D9 purpose. As built, normalization additionally accepts the closed compile-time STATIC_COMPONENTS allowlist (hero, search_rows, program_all) — still a closed label set with zero cardinality growth; the redaction boundary is unchanged.

Inventory (verified 2026-08-04 against main @ bf6bbf79)

Citation base: services/canopy-web/src/; cd.rs = api/case_detail.rs, dv.rs = determination_view.rs. Verdicts: OK = explicit error state / visible degrade on every leg; COLLAPSE = at least one leg’s failure renders as empty/absent data; NO-UPSTREAM = no network calls.

Case-detail sections (23): 15 COLLAPSE / 6 OK / 2 NO-UPSTREAM

slug fetcher path upstream calls current Err handling verdict

household

sections/household.rs:22 → render_household_tab cd.rs:2520

persons households/{id}; per-member SERIAL get_person_info; persons addresses (cd.rs:2454); (SNAP) renewals certifications — 3+N serial

household Err → warn + members = Vec::new() (cd.rs:2583-2599), members-empty state (no error branch); address leg → explicit address.error visible card (cd.rs:2510-2517); per-member Err → "Person xxxxxxxx" row (cd.rs:2553-2560); SNAP cert .ok() cd.rs:2612 → "No active certification" (cd.rs:2629-2635)

COLLAPSE — households failure false-empties the member table; renewals outage reads as "No active certification"

income

sections/income.rs:21 → render_income_tab cd.rs:2951

persons households/{id}/full; (SNAP) verification discrepancies (cd.rs:3480) — 2 serial

/full Err → explicit fetch_error + retry_href (cd.rs:2966, 3201-3202) — OK; IEVS leg unwrap_or_default() cd.rs:3493-3499 — variance column silently blanks

COLLAPSE (partial) — the IEVS discrepancy leg’s failure disappears

determination

sections/determination.rs:20 → render_determination_tab cd.rs:3561

assemble_determination_view (dv.rs:463): persons households/{id} (dv.rs:521); SERIAL per program: eligibility dets (dv.rs:549), tanf dets (dv.rs:570), medicaid dets (dv.rs:614) + serial resolve_name (dv.rs:627), caps dets (dv.rs:671) + resolve_name
caps_payee 2-chain (dv.rs:778-795), wic dets (dv.rs:728) + resolve_name; then hearing-view (cd.rs:3595), appeal requestors (cd.rs:2850), enrollment ×2 (cd.rs:928), adverse actions (cd.rs:864), tsnap (cd.rs:983) — ~12+ legs, nearly all serial

Every leg silent: let Ok .. else None per program group (dv.rs:556/575/619/676/733) — a program-service outage renders as NO determination; household else → empty map (dv.rs:526-530); caps_payee .ok()? (dv.rs:786/792); hearing .ok() (cd.rs:3601); enrollment .ok()? (cd.rs:936/946); open actions Err(_) ⇒ Vec::new() (cd.rs:879); tsnap .ok()? (cd.rs:991); requestors Err → empty vec w/ visible disabled state (cd.rs:2860-2864)

COLLAPSE — all five program legs + enrollment/tsnap/hearing/open-actions vanish on Err

notices

sections/notices.rs:21 → render_notices_tab cd.rs:3632

notices ?household_id&program&limit=50 — 1 call

unwrap_or_else → warn + Vec::new() (cd.rs:3671-3675); no fetch_error field

COLLAPSE — outage renders as "no notices"

renewals

sections/renewals.rs:25 → render_renewals_tab cd.rs:3702

renewals snap/nudges?household_id — 1 call

unwrap_or_else → warn + Vec::new() (cd.rs:3726-3730)

COLLAPSE — outage renders as "no pending nudges"

appeals

sections/appeals.rs:20 → render_appeals_tab cd.rs:3742

appeals ?household_id&limit=50 — 1 call

unwrap_or_else → warn + Vec::new() (cd.rs:3783-3787)

COLLAPSE — outage renders as "no appeals"

activity

sections/activity.rs:21 → render_activity_tab cd.rs:3799

security events?household_id (limit 50) — 1 call; #1309 BFF scope filter post-fetch

explicit fetch_error flag (cd.rs:3814) + error_block/Retry

OK

abawd

sections/abawd.rs:23 → render_program_tab cd.rs:3861

snap abawd/tracking — 1 call

Ok-empty → "No data" card (cd.rs:3922-3931); Err → visible service-error block (cd.rs:3950-3963); unconfigured → visible error (cd.rs:3871-3877)

OK

work_req

sections/work_req.rs:21 → render_tanf_work_req cd.rs:3970

tanf work-requirements/{hh}; + /activities — 2 serial

requirement .ok().map(..) cd.rs:3981 → no-data state; activities unwrap_or_default() cd.rs:4004

COLLAPSE — both legs render as "no work requirement / no activities"

time_limits

sections/time_limits.rs:20 → render_tanf_time_limits cd.rs:4048

tanf time-limits/{hh} — 1 call

.ok().and_then(..) cd.rs:4053-4056 → no-data state

COLLAPSE — outage renders as "no time-limit data"

categories

sections/categories.rs:20 → render_medicaid_categories cd.rs:4065

medicaid dets?limit=200; + applications/{app}/categories — 2 serial

both unwrap_or_default() (cd.rs:4074-4077, 4089-4091)

COLLAPSE — both legs render as "no categories"

authorization

sections/authorization.rs:20 → render_caps_authorization cd.rs:4127

caps dets?household_id; + dets/{det}/authorizations — 2 serial

dets Err() ⇒ None cd.rs:4159; authorizations Err() ⇒ Vec::new() cd.rs:4189

COLLAPSE — both legs render as "no authorizations"

nutrition

sections/nutrition.rs:20 → render_wic_nutrition cd.rs:4207

persons households/{id}; per-member SERIAL wic risk assessments — 1+N serial

household Err → warn + empty members (cd.rs:4223-4226); per-member Err → warn
continue (cd.rs:4236-4240)

COLLAPSE — household and per-member failures render as "no assessments"

guidance

sections/guidance.rs:20 → render_guidance_tab cd.rs:4278

none (WorkflowTemplates from Extension)

template render fallback only

NO-UPSTREAM

assets

sections/assets.rs:23 → render_assets_tab cd.rs:3282

persons /full — 1 call

explicit fetch_error + retry_href (cd.rs:3293, 3325-3326)

OK

expenses

sections/expenses.rs:23 → render_expenses_tab cd.rs:3334

persons /full — 1 call

explicit fetch_error + retry_href (cd.rs:3345, 3377-3378)

OK

address

sections/address.rs:32 → render_address_tab address.rs:184

persons /full — 1 call; #1310 confidentiality threaded in

explicit fetch_error + retry_href (address.rs:193, 224-225); Unknown fail-closed → withheld + error block (address.rs:211, 226-228)

OK

persons

sections/persons.rs:24 → render_persons_tab cd.rs:3389

persons /full; + persons households/{id} (member-id map) — 2 serial

/full: explicit fetch_error + retry_href (cd.rs:3400, 3438) — OK; second read if let Ok drop (cd.rs:3413-3421) → Remove forms silently hidden (tab_persons.html:77)

COLLAPSE (partial) — the member-id leg’s failure silently removes the Remove affordance

verifications

sections/verifications.rs:82

verification ?application_id&status=pending&limit=50; per-verification SERIAL /responses — 1+N serial

list: explicit fetch_error + error_block/Retry (verifications.rs:114, 156) — OK; responses legs unwrap_or_else → warn + Vec::new() (verifications.rs:135-144) → group reads "Awaiting response" (false state)

COLLAPSE (partial) — a failed responses leg reads as "applicant hasn’t responded"

audit

sections/audit.rs:116

tokio::join! 2-wide (audit.rs:129-134): security events + chain status (get_terminal_status)

events Err → explicit fetch_error + error_block/Retry (audit.rs:136, 198); chain Err → visible "Unable to verify chain" pill (audit/stream.rs:174-186)

OK

fact_history

sections/fact_history.rs:46 → api/fact_history.rs:155

persons /full (fact_history.rs:217); per-member CONCURRENT bounded_join security fact-change-history (fact_history.rs:254-277)

household leg → explicit fetch_error + error_block (fact_history.rs:168-174, 242-250) — OK; per-member leg Err → warn + skip (fact_history.rs:271-275)

COLLAPSE (partial) — per-member failures vanish (partial history reads as complete)

cross_program

sections/cross_program.rs:20

none (coming-soon stub, #562)

render_coming_soon only

NO-UPSTREAM

documents

sections/documents.rs:71

applications /{id}/documents — 1 call (no-application short-circuit at :79)

unwrap_or_else → warn + Vec::new() (documents.rs:99-110); no fetch_error field (documents.rs:56-68)

COLLAPSE — outage renders as "no documents uploaded"

Dashboard panels (22): 2 COLLAPSE / 16 OK / 4 NO-UPSTREAM

slug current Err handling verdict

analyst_audit_export

Stub — always state="empty" (analyst_audit_export.rs:33-41)

NO-UPSTREAM

analyst_case_search

Stub — search form only (analyst_case_search.rs:29-36)

NO-UPSTREAM

analyst_pipeline_funnel

Per-stage Err → None (:174-180); all-None → state="error" (:124-129); single failed stage → visible "—" (:141-146)

OK

at_a_glance

4-wide join; per-feed Err → "—" tile (at_a_glance.rs:54-65, 73-84)

OK

audit_events

Three-way split (audit_events.rs:66-75)

OK

cross_program_alerts

Three-way split (cross_program_alerts.rs:70-79)

OK

ievs_alerts

Three-way split (ievs_alerts.rs:60-69)

OK

ievs_rollup

Stub — always state="empty" (ievs_rollup.rs:33-41)

NO-UPSTREAM

my_queue

All three legs drop on Err (if let Ok my_queue.rs:118/182/211); names swallow (:289-309); total failure → state="empty" (:333)

COLLAPSE

overdue_cases

Three-way split (overdue_cases.rs:58-67)

OK

overpayment_rollup

Three-way split (overpayment_rollup.rs:83-101)

OK

pending_hearings

Three-way split (pending_hearings.rs:70-86, :114-127)

OK

pending_verifications

Three-way split (pending_verifications.rs:60-69)

OK

recent_applications

Main leg three-way split (:57-66); per-row outcome .ok() (:113) degrades a label, not a false-empty

OK

recent_determinations

Per-program legs collapse: .ok() :57 + unwrap_or_default() :80; masked as state="empty" at :107 — no error state exists

COLLAPSE

recent_notices

Three-way split (recent_notices.rs:55-64)

OK

sanctions_rollup

Three-way split (sanctions_rollup.rs:60-77)

OK

supervisor_caseload_trend

Err → None (:163-169) → state="error" + Retry (:95-116)

OK

supervisor_kpis

Spine Err → state="error" (:148-161, :116-119); side legs → "—" (:130-131, :163-192)

OK

system_messages

Stub — always state="empty" (system_messages.rs:31-39)

NO-UPSTREAM

team_queue

Three-way split (team_queue.rs:67-82)

OK

upcoming_appointments

Three-way split (upcoming_appointments.rs:57-66)

OK

Review resolutions (external review, rev 3)

  • Token cancel-safety: the mint_lock reasoning covers the self-validation path — which is production (bootstrap.rs:180); tests construct with_self_validation mode and cancel at each stage (lock-wait, token HTTP, JWKS refresh, revalidation backoff). Dropping the future releases the tokio mutex (no poisoning) and skips enter_cooldown (observed-failure only). A cancelled mint the IdP already processed is harmless: token discarded, next acquire re-mints.

  • No e2e weakening: dashboard.spec.ts:96-102 keeps its table-or-earned-empty assertion unchanged — adding an error arm would let a broken devstack pass a healthy-stack test. A genuine my-queue source failure during e2e now fails the spec honestly; controlled-failure e2e is #1325.

  • Claim discipline: this plan bounds upstream I/O within the budget; session extraction, local composition SQL, and template render are outside the envelope (local-PG, Err→500 class — documented residual). The acceptance claim is the aggregate upstream-I/O cutoff + honest states, not "pages always respond in 12s".

Delivery

Three MRs, each independently green (branches chain off the previous until merged). MR1/MR2 use Relates to #1306; MR3 Closes #1306 (AC narrowed on-issue 2026-08-04).

Follow-ups filed + related: #1319 (remaining handlers), #1320 (bounded writes), #1321 (queue-name batch lookup), #1322 (determination assembler parallelism), #1323 (timeout retuning), #1324 (middleware stamping + token-semantics unification), #1325 (devstack fault injection + degraded-page e2e), #1326 (cross-request capacity).

Test plan

  • Cutoff integrity: gate-wait + retry/backoff + headers-just-before-cutoff + slow body ⇒ total bounded by call_deadline (mock records request timestamps; last hit ≤ cutoff; zero hits post-expiry); saturated gate then single-shot call (no double budget); ≤8 active non-streaming bodies; expiry immediately after permit acquisition ⇒ zero server hits.

  • Classification: JSON/raw/terminal-status/non-success-body timeout mapping; repeated attempt-timeout vs repeated-503 exhaustion; 2xx+body-timeout ⇒ Timeout not Decode; binding-arm ties ⇒ DeadlineExceeded.

  • Threading: identical deadline + same Arc<Semaphore> through with_timeout/with_token in both orders.

  • Token: canopy-web side — hung-IdP ⇒ bounded return + deadline-class fast-fails; refused-IdP ⇒ AuthUnavailable on all read verbs with budget remaining + zero hits + token_timeout metric on lapse. canopy-auth side — with_self_validation cancellation regressions (cancel mid-mint; cancel during lock-wait; revalidation- backoff stage only if existing mock knobs reach it): subsequent acquire succeeds, no cooldown entered by a cancellation.

  • Un-stamped parity: every verb’s deadline behavior identical with no deadline — same request path/timeouts/retry policy; the typed-error construction and call-outcome telemetry apply in both regimes (incl. put_idempotent headroom); 3-attempts-on-503 pin stays green.

  • Retry policy: 0.8×-budget success (fails under the old ÷3 split, passes now).

  • Streaming: default + explicit timeouts; body outlives a page deadline.

  • Pure math: proptests on remaining_at/clamp_at/floor/saturation with injected now — no wall clock.

  • States: queue_state totality proptest (never empty with failures; never populated with 0 items + failures); per-surface all-fail / partial / early-ok-late-hang with never-false-empty oracles; per-fixed-panel and per-fixed-section mixed-feed tests; ?program=all matrix.

  • Timing assertions are deadline-relative (elapsed ≤ deadline + fixed allowance).

  • Metrics: manual-reader exporter tests — exact counts + label vocabulary.

Harness (MR2 builds it): Tier a — fault matrix at the fetch pipelines (stamped test_service_clients + spawn_router mocks; the #1310/#1309 idioms; no DB). Tier b — full-handler proof per surface, split by what each handler transitively requires (deviation from rev 3, recorded 2026-08-04: the original "explicitly constructed CompositionState, no RabbitMQ" is unconstructible — canopy_mq::ConnectionManager::new connects to the broker eagerly and Publisher has no broker-less constructor, and load_composition propagates fetch_db_layers errors with no filesystem fallback, so any CompositionState demands live RabbitMQ + PG):

  • Cases family (GET /cases, GET /cases/search — no CompositionState): true full-router proof — real require_auth + Extension stack + MemoryStore sessions (write_authz_route_tests.rs precedent) + direct WebConfig literals + spawn_router mock upstreams. Zero infrastructure.

  • Dashboard family (get_dashboard, get_panel_fragment — need CompositionState only to load the composition): proof at the injected-composition seam — render_dashboard_inner / panel_fragment_inner take an already-ComposedSurface (all-pub struct, hand-constructible), so the panels-fanout + hero + render path is proven with zero infrastructure. Composition loading stays covered by tests/composition_api_test.rs (devstack-gated) and the e2e battery.

Verification

Per MR: cargo fmt --check --all · cargo clippy --all-targets — -D warnings · RUSTDOCFLAGS="-D rustdoc::broken_intra_doc_links" cargo doc --workspace --no-deps · cargo xtask quality-budgets · cargo xtask check-docs · full pre-push battery (cargo xtask validate; JUnit XML → test-results/) · api-docs no-op check (BFF-only). MR1’s gate: no render-path change + the new unit/proptest suite. MR2/MR3 gates: the fault matrices above.

Edit this page · default