ADR-032: Two-Tier Scenario Corpus — Synthetic Engine Fixtures + Per-Jurisdiction Conformance Packs

On this page

Status

Accepted (2026-06-10). Amends ADR-031 §3 (refines the "second jurisdiction fixture" into a synthetic pair + conformance packs) and extends ADR-006 (the option-space taxonomy gains test fixtures that exercise it). Implementation tracked by epic &61; the scenario-inventory-e2e plan’s MR structure reflects this ADR.

Context

Epic &61 builds a scenario inventory (the casework long tail, derived from policy) and the E2E suites that cover it. Before authoring begins, one architectural question decides the shape of every scenario artifact: what policy configuration does the canopy-native regression corpus run against?

Two distinct testing questions hide under "scenario-based E2E":

  1. The engine question — does canopy correctly implement the space of legal programs federal law defines? ADR-006 already names the taxonomy: federal floor rules, federal parameters, state options (explicit elections — BBCE, simplified reporting, interview waiver…), state-set values, and state-specific provisions. A real jurisdiction is one point in that space.

  2. The deployment question — does this configured jurisdiction behave according to its policy? Inherently per-jurisdiction: the scenarios Georgia ships are not the scenarios South Carolina ships, because their elections differ.

The expedient answer — reuse Georgia policy as the default test corpus — conflates the two questions and fails three ways:

  • Un-elected arms go permanently untested. Georgia elects BBCE, so the non-BBCE asset-test path never executes in CI; a regression there ships silently to the first jurisdiction that elected differently. Every option Georgia exercises one way leaves the other arm dark. "Works for Georgia" would silently become canopy’s definition of "works."

  • Real-value churn rots goldens. Georgia’s values change on the federal indexing cycle (COLA Oct 1, FPL Jan 1, SMI Jul 1 — modeled by &59’s indexing.toml) and at state policy revisions. Golden assertions tied to real values churn annually, and churned goldens get rubber-stamped — the regression suite decays exactly where it is needed most.

  • Georgia-isms accrete into core fixtures. Today rulesets/default/ is a verbatim Georgia copy and ~23 integration tests hardcode jurisdiction: "georgia" — the &58 program explicitly set out to remove this class of coupling, not entrench it.

Conversely, a purely synthetic corpus cannot answer the deployment question: it validates the engine, not the composition of real values, real elections, and real state provisions a jurisdiction actually ships — and SNAP UAT (September 2026) is a Georgia contract.

The raw material for both tiers now exists: the state manuals are pinned and catalogued (DPH WIC + DECAL CAPS corpora, #764; PAMMS for SNAP/TANF/Medicaid), and the federal option space has published enumerations (the FNS SNAP State Options Report, the CCDF state-plan preprint, Medicaid state-plan option lists).

Decision

Scenario testing is two corpora with distinct jobs, sharing one inventory schema.

1. The scenario inventory is tagged by scope

Every ScenarioEntry (epic &61 MR1 schema) carries:

  • scope = "universal" — mandated by the federal floor; every jurisdiction must pass it unchanged; or

  • scope = "election-dependent" — behavior branches on one or more state options, named via elections = ["snap.bbce", "snap.simplified-reporting", …].

Election keys resolve against a federal option registry (compliance/federal-options/{program}.toml): each option gets a key, the CFR/statute citation that authorizes the election, and its legal values. The registry is populated incrementally — an option enters the registry when a scenario first references it (no up-front ocean-boiling); unknown election keys are a schema error in cargo xtask scenarios audit. Over time the registry becomes its own completeness artifact in the ADR-031 sense: the machine-checkable enumeration of the federal option space, seeded from the published option reports.

2. The engine corpus runs against a synthetic adversarial pair

Two fixture jurisdictions, designed for branch coverage rather than realism:

  • rulesets/test-min/ — the smallest legal deployment: minimal program subset (exercising ADR-005 graceful degradation of absent optional services), strictest or declined elections (no BBCE, standard reporting, interviews required…).

  • rulesets/test-max/ — everything on: all programs, all optional surfaces, the most permissive elections.

The pair exists because some behaviors only manifest in contrast — a single fixture cannot exercise both arms of any election. Election-dependent scenarios run against whichever fixture (or both) elects the arm under test; universal scenarios run against both.

Design properties:

  • Round-number policy values (income limit $1,000, deduction $100, grant $500). Golden assertions become auditable by inspection, and the engine corpus is immune to real-world indexing churn by construction.

  • Citation discipline is preserved, not exempted. Each election in a test- jurisdiction cites the federal provision authorizing it (the registry entry). Each *value cites the fixture-design appendix of the scenario-inventory plan via a fixture citation source kind, which cargo xtask policy audit accepts only under rulesets/test-* — a fixture citation anywhere else is an audit error.

  • The engine corpus is canopy CI’s responsibility: universal + election-dependent scenarios in the standard pipelines, with the same tiering (unit/flow/journey) and gating the &61 plan defines.

3. Conformance packs are per-jurisdiction artifacts

A conformance pack at rulesets/{jurisdiction}/scenarios/ holds the scenario bindings a real deployment ships:

  • the election-dependent scenario set instantiated against that jurisdiction’s actual elections, with real-value assertions;

  • plus state-provision scenarios that exist only in that jurisdiction’s policy (DECAL’s 13-week job-search grace, Georgia Pathways — sourced from the pinned state-manual corpora and the state rows of the action catalogue).

Georgia’s pack is the first and is the onboarding template: bringing up a new jurisdiction means electing options, setting values (ADR-006), and instantiating the election-dependent scenario set against those choices — a bounded, documented work-list rather than an open-ended QA exercise. A jurisdiction’s conformance-pack coverage is that deployment’s ship gate; Georgia’s is the September 2026 UAT readiness measure.

The existing demo-driven suite (24 personas, 43 specs, 19 JDM fixtures, all Georgia-seeded) is grandfathered as the seed of the Georgia conformance pack — &61’s inventory binds them where they genuinely cover scenarios; nothing is rewritten for purity.

4. Supersessions and non-changes

  • The &61 plan’s MR5 (testland, a single divergent fixture) is superseded by the test-min/test-max pair; its mechanics survive (the --jurisdiction parameterization of seed/e2e, the scheduled smoke job, the asserted-divergence check).

  • rulesets/default/ is unchanged. It is an operator-bootstrap artifact (stage-6 #499), not a test fixture; engine testing never targets it. The ~23 hardcoded jurisdiction: "georgia" test literals still burn down to a canopy-test-lib helper (&61 MR6) so service tests can target the fixtures.

Consequences

  • Engine regressions become catchable in the arms no real jurisdiction has elected yet — the difference between "canopy works the way Georgia uses it" and "canopy works."

  • Goldens stop churning with policy updates. Real-value assertions live only in conformance packs, where churn is meaningful (the jurisdiction’s policy actually changed) rather than noise.

  • Jurisdiction onboarding gains a deliverable — instantiate the conformance pack — replacing an undefined QA burden with a checklist derived from the option registry.

  • Cost: authoring and maintaining two fixture jurisdictions (full per-jurisdiction artifact sets: jurisdiction.toml, citations, composition, workflows, notices) and the incremental option registry. The artifact set is known (Georgia’s is 34 files); round numbers and minimal elections keep test-min small.

  • The option registry is a new completeness surface: future audits can ask "which federal options does canopy not model at all?" — the engine-level analogue of the action catalogue’s unbound rows.

  • Scenario authoring (&61 MR2+) must tag scope from the start; the inventory schema (&61 MR1) carries scope/elections from its first version, avoiding a retrofit.

Edit this page · default