ADR-031: Policy Coverage Assurance — Currency, Actions, Scenarios

On this page

Status

Accepted (2026-06-09). Implementation tracked by epic &58 (parent) and its children &59 / &60 / &61; per-child plans: policy-currency-drift, action-coverage-matrix, scenario-inventory-e2e.

NOTE
§3 amended by ADR-032 (2026-06-10): the "second jurisdiction fixture" is refined into the two-tier corpus — synthetic test-min/test-max engine fixtures + per-jurisdiction conformance packs.

Context

Canopy must be correct for any jurisdiction running any program subset (ADR-005/ADR-006). Today exactly one axis of policy fidelity is measurable and enforced: every value in jurisdiction.toml traces to a citation (ADR-011; cargo xtask policy audit; ~930 Georgia citations). The other axes that decide whether the platform is complete are asserted, not measured:

  • Currency. policy audit is forward-only with a 365-day staleness warning; it does not detect upstream drift. The policy drift tool ADR-011 sketched was never built (xtask/src/cmd/policy.rs has no Drift variant); policy sync-cache is a shallow clone with no commit pin or content hash, so "did PAMMS change since we verified?" is unanswerable mechanically. A rulesets/federal/citations.toml exists but no audit reads it — federal values (FPL, COLA allotments, SUAs, SMI) carry citations that nothing validates, and the annual federal indexing cycle (Oct 1 / Jan 1 / Jul 1) is tribal knowledge.

  • Action coverage. No artifact maps policy-mandated actions (what a worker/applicant/system MUST be able to do) to endpoints + verbs. federal-requirements.adoc is prose (regulation → service, no verb/path); ADR-007 CLI parity is documentary, unenforced. Epic &56 is the proven cost: an entire class of mandated action (worker fact-authoring) had zero endpoints and was found by tracing a demo, not by audit.

  • Scenario coverage. No inventory of the casework long tail exists; coverage is demo-driven (24 persona archetypes, 43 Playwright specs, 19 JDM math fixtures). We test what we stage, and a staged scenario is by definition one we already knew about. All ~23 integration tests hardcode jurisdiction: "georgia"; "works everywhere" is untested.

Decision

Extend the ADR-011 discipline to three further coverage artifacts. Each is a versioned, schema-validated, machine-checkable file in the repo, each is enforced by an xtask gate wired into CI via the staged advisory→blocking pattern ADR-011 proved out (adr-011-unwrap-audit → blocking after burn-down), and each treats policy as the spec: the artifact derives from the manuals + CFR, and the gate fails when the implementation does not cover the artifact — never the other way around.

1. Currency: pinned sources + drift detection (epic &59)

  • The Citation schema (crates/canopy-policy/src/citation.rs) gains source pinning: a content hash of the cited source section captured at verification time, so "upstream changed since we verified" becomes a mechanical comparison, not a guess. policy sync-cache records what it fetched (commit/hash manifest), replacing the unpinned shallow clone.

  • A cargo xtask policy drift command compares pinned hashes against the refreshed cache and reports changed-since-verified citations. Drift detection is mechanical and CI-visible; deciding whether a drifted value must change remains human judgment (ADR-011’s stance stands — drift never auto-edits values).

  • Federal becomes a first-class audited source family. policy audit extends to rulesets/federal/citations.toml with the same completeness/consistency/staleness checks the jurisdiction file gets, and the annual indexing cycle is modeled explicitly (effective-date windows per family: SNAP COLA Oct 1, FPL Jan 1, SMI Jul 1) so an out-of-window table is a finding, not a surprise.

  • Reverse completeness: orphaned citations (cited keys no longer present) and required-but-uncited parameters become findings instead of being silently ignored.

2. Action coverage: the mandated-action catalogue (epic &60)

  • A schema-validated action catalogue (canopy-policy crate schema; TOML data) enumerates policy-mandated actions: actor (worker/applicant/system), the action, the regulatory trigger (CFR/PAMMS citation), and the coverage binding — service, endpoint path, HTTP verb, operationId, CLI command (ADR-007), test reference. The structured analogue of citations.toml, for capabilities instead of values.

  • A cargo xtask policy action-coverage gate cross-checks the catalogue against the committed OpenAPI snapshots (docs/modules/ROOT/openapi/.json, 15 services / ~174 paths — already maintained by cargo xtask api-docs): a mandated action whose endpoint+verb is absent is a finding. Known-open gaps live in an allowlist with a mandatory reason + issue ref (the compliance/.toml pattern), so the gate stays green while honestly tracking debt.

  • ADR-007 parity becomes enforceable as a by-product: each catalogue row’s CLI column is checked against the CLI command registry; a missing CLI mirror is a finding of the same gate.

3. Scenario coverage: the scenario inventory (epic &61)

  • A schema-validated scenario inventory enumerates casework scenarios derived from the manuals + CFR (change-of-circumstance types per 7 CFR 273.12, expedited→regular transitions, ABAWD edges, hearings/IPV paths, recoupment, cross-program ELE/TMA/EE15, churn, mixed households …), each tagged with programs, life events, regulatory citations, and its coverage binding to E2E specs/fixtures.

  • A cargo xtask scenarios audit gate reports per-scenario coverage status (covered / partial / uncovered) — policy audit, but for behavior. Uncovered scenarios are tracked findings that become issues, not silence.

  • The inventory drives human-fidelity, multi-life-event E2E journeys (e.g. job loss → report change → adverse action → appeal with continued benefits → recert) consuming the endpoints proven by the action catalogue and the values kept current by drift detection — and at least one journey runs against a second jurisdiction fixture to keep "works everywhere" honest.

Consequences

  • Three new durable artifacts join citations.toml as repo-versioned compliance surfaces; reviewers see coverage changes in diffs, and the quality-bar for "complete" becomes mechanical on all four axes (values, currency, actions, scenarios).

  • Authoring the catalogues is a genuine policy-reading effort (the manuals + CFR are the source). The artifacts are seeded program-by-program (SNAP first, UAT-aligned) rather than boiling the ocean; an incomplete catalogue with an honest allowlist beats an aspirational complete one.

  • Gates follow the proven staged pattern: land advisory (allow_failure: true), burn down or allowlist findings with reasons, then flip blocking. The ratchet discipline of ADR-030 applies — coverage may only improve.

  • New &56-class gaps are found by audit, by construction — each uncovered action/scenario is enumerated and filed, not stumbled over.

  • The catalogues are jurisdiction-agnostic in schema; Georgia is the first dataset, not a special case.

Relationship to other ADRs

  • Extends ADR-011 (policy-to-rules traceability): same thesis, three new axes; the PolicySource abstraction and the staged-enforcement pattern carry over. ADR-011’s "drift is advisory, never a value-editing CI gate" stance is preserved and sharpened: hash drift detection is mechanical and CI-visible; value reinterpretation stays human.

  • Operationalizes ADR-007 (CLI/API/UI parity): the action catalogue’s CLI column is the first enforcement mechanism parity has had.

  • Builds on ADR-005/ADR-006 (modular profiles / jurisdiction-agnostic rulesets): the scenario inventory’s second-jurisdiction journeys are the first systematic exercise of that promise.

  • Epic &56 (ADR-027/028) is the motivating instance: the first catalogue rows for worker fact-authoring actions bind to the Track-1 endpoints, and the pre-&56 state (zero endpoints) is the canonical example of what the gate exists to catch.

Amendment 1 — Journey walkthrough pairing gate (#972, 2026-07-05)

Status unchanged (still Accepted). This refines §3’s "human-fidelity, multi-life-event E2E journeys" into a machine-enforced pairing contract. ADRs are immutable once accepted, so this is an in-document amendment, not an edit to the Decision.

§3 requires journeys but only mechanically checked the automated half. Epic &61 adds the requirement that every journey also ships a human-followable Antora walkthrough (a per-persona, click-by-click script under docs/modules/ROOT/pages/walkthroughs/, with spec-generated screenshots), so a human can reproduce what the spec automates. cargo xtask scenarios audit now enforces the pairing:

  • New walkthrough binding kind. A kind = "walkthrough" points file at the paired .adoc. It is a companion doc, not a coverage tier — it never lifts a scenario’s status. resolve_walkthrough requires the file to live under the walkthroughs module, name its scenario.id (the doc↔scenario cross-link), and reference at least one image::walkthroughs/… screenshot whose committed PNG exists (the screenshot contract; a page can’t claim a screenshot the spec never produced).

  • MissingWalkthrough finding. A covered journey-tier scenario with neither a resolving walkthrough binding nor a walkthrough_blocked_by marker fails the gate. Uncovered/partial journeys are exempt (nothing to pair yet).

  • Issue-backed walkthrough_blocked_by. When a journey’s walkthrough is blocked by missing UI/feature work, the row carries walkthrough_blocked_by = ["N", …] — a tracked, visible block, not an allowlist. Each entry must be a -prefixed issue ref, and a row may not carry both a walkthrough binding and the marker (a stale-marker guard); closing the gap issue must remove the marker and add the walkthrough (an acceptance criterion on each gap issue). This is the honest realization of "an incomplete catalogue with an honest allowlist beats an aspirational complete one" — here the allowlist entries are live issues.

  • Reverse audit (OrphanSpec). Every on-disk tests/e2e/specs/journey-*.spec.ts must be referenced by an e2e-spec binding, so a journey spec can never run invisibly to the coverage gate.

As-built note: at introduction, all nine existing SNAP journeys were UI-blocked (no worker cert-create, SNAP appeal file/decision, enrollment/issuance, or ELE-consent UI; SNAP-only intake; no backdating), so each is walkthrough_blocked_by a filed UI-gap issue (#973–#979) — themselves human-UAT blockers (#980 TSNAP-display and #981 the ADH claim-reclassification feature block the not-yet-authored #851/#852 journeys). See the scenario-inventory-e2e plan and Journey Walkthroughs. Consistent with ADR-032 (the corpus split is unchanged; this adds a binding kind + two findings).

Edit this page · default