Plan: JDM Ruleset End-to-End Happy-Path Tests

On this page

Status

Step Description Status

1

Add shared canopy_test_lib::rules helper that posts to /v1/evaluate and asserts output shape

Done (2026-04-18) — RulesetFixture, FixtureExpect, evaluate_fixture, fixture_path, load_fixture + 3 unit tests

2

Author canonical happy-path fixtures per program under crates/canopy-test-lib/fixtures/rulesets/

Done (2026-04-27) — 12/12 fixtures fully populated. The two previously-ignored fixtures (snap-eligibility, medicaid-non-magi) had their full input schema authored: snap-eligibility added earned_income_deduction_pct (0.20), shelter_half_pct (0.50), au_net_income_pct (0.30), minimum_benefit ($23), minimum_benefit_max_hh_size (2), and renamed countable_resources/resource_limit → countable_assets/asset_limit to match the ruleset’s decision-table input expressions. medicaid-non-magi added abd_min_age (65), abd_mnil (317), mnil (317), medical_expenses_monthly (0), tefra_max_age (18), hospital_los_days_threshold (30), chafee_min_age/chafee_max_age (18/21), waiver_type ("none"), has_medicare_part_b (false), is_chafee_eligible/in_foster_care/has_adoption_assistance (false). cargo xtask rules check: 12 fixtures evaluated, 0 failed. Paired Rust tests dropped #[ignore] and the macro’s (ignore, …​) arm.

3

Write one integration test per active ruleset

Done (2026-04-18) — 12 tests in ruleset_happy_path_test.rs; 2 #[ignore] pair with the ignored fixtures

4

Wire the new tests into cargo xtask test --integration — no new profile required

Done (2026-04-18) — nextest auto-discovers; 10/10 non-ignored tests green in the pre-push hook

5

Add a cargo xtask rules check gate that compiles every ruleset against its fixture, catches drift pre-commit

Done (2026-04-18) — new "Fixture-driven drift gate" section runs Decision::evaluate in-process against each paired fixture; respects fixture "ignore": true flag

Branch: test/jdm-happy-path
Labels: type::test, priority::high, program::cross-program, service::rules, workflow::ready

Context

The 2026-04-18 repo review found that canopy-rules has "smoke tests only" — the current coverage verifies that the zen-engine wrapper starts and that its health/metrics endpoints respond. It does not verify that any production JDM ruleset actually evaluates correctly given a realistic input.

Per ADR-003, every eligibility decision traverses a JDM ruleset via canopy-rules. A broken ruleset — syntax error, renamed variable, removed expression node — surfaces only when the first downstream determination calls through. In practice this means ruleset regressions are detected at UAT time, far too late.

Active rulesets today (12 total):

Ruleset Program

snap-eligibility.json

SNAP

snap-benefit-calculation.json

SNAP

snap-categorical-eligibility.json

SNAP

tanf-eligibility.json

TANF

tanf-benefit-calculation.json

TANF

tanf-work-requirements.json

TANF

medicaid-magi.json

Medicaid (MAGI)

medicaid-non-magi.json

Medicaid (ABD/LTC/MN)

chip-eligibility.json

CHIP

medicaid-eligibility-hierarchy.json

Medicaid (EE15)

caps-eligibility.json

CAPS

wic-eligibility.json

WIC

Each ruleset gets one canonical happy-path fixture and one test. The goal is not exhaustive coverage (that belongs to the per-program integration tests); it is a drift gate.

Scope

In scope:

  • One happy-path fixture + test per active ruleset, 12 total.

  • Shared canopy_test_lib::rules helper that invokes canopy-rules over HTTP.

  • A pre-commit cargo xtask rules check gate validating all rulesets compile and their fixtures evaluate without panics.

Out of scope:

  • Denial-path and edge-case fixtures (belong to per-program plans, e.g. snap-categorical-eligibility.adoc).

  • Snapshot testing of output payload shapes (too brittle for data-driven rulesets that evolve with federal parameters).

  • Performance / load testing.

Dependencies

  • services/canopy-rules — already exposes POST /v1/evaluate.

  • crates/canopy-test-lib — already provides TestClient and infrastructure_available.

  • cargo xtask rules check — already exists per roadmap.adoc Tier 2 ("all 12 rulesets compile under zen-engine 0.55"); this plan extends it to run fixtures.

Design

Fixture layout

crates/canopy-test-lib/fixtures/rulesets/
├── snap-eligibility.json         # input + expected_output_shape
├── snap-benefit-calculation.json
├── snap-categorical-eligibility.json
├── tanf-eligibility.json
├── …
└── wic-eligibility.json

Each fixture file:

{
  "ruleset":  "snap-eligibility",
  "description": "Single-adult household at 100% FPL — standard approve",
  "input":    { … full ApplicationContext … },
  "expect": {
    "output_fields": ["status", "benefit_amount", "reasons"],
    "status":        "approved"
  }
}

expect.output_fields asserts presence, not value, for fields that vary with federal parameter updates (benefit amount, FPL percentages). expect.status is the strong check.

Test helper

// crates/canopy-test-lib/src/rules.rs (new)

pub async fn evaluate_fixture(fixture_path: &str) -> Result<serde_json::Value, RulesError> {
    let fixture: RulesetFixture = serde_json::from_str(&std::fs::read_to_string(fixture_path)?)?;
    let cfg = TestConfig::from_env();
    let client = TestClient::new(&cfg.rules_url);
    let resp = client.post_json(
        &format!("/v1/evaluate/{}", fixture.ruleset),
        &fixture.input,
    ).await;
    resp.assert_status(200);
    let out = resp.json_value();
    for field in &fixture.expect.output_fields {
        assert!(
            out.get(field).is_some(),
            "ruleset {} output missing expected field `{}`",
            fixture.ruleset, field,
        );
    }
    if let Some(expected_status) = &fixture.expect.status {
        assert_eq!(
            out.get("status").and_then(|v| v.as_str()),
            Some(expected_status.as_str()),
            "ruleset {} status mismatch", fixture.ruleset,
        );
    }
    Ok(out)
}

Test module layout

One integration test file:

// crates/canopy-rules-client/tests/ruleset_happy_path_test.rs (or services/canopy-rules/tests/...)

#[tokio::test]
async fn snap_eligibility_happy_path() {
    if !canopy_test_lib::infrastructure_available().await { return; }
    canopy_test_lib::rules::evaluate_fixture(
        "../../crates/canopy-test-lib/fixtures/rulesets/snap-eligibility.json"
    ).await.expect("SNAP eligibility happy path");
}

// … one per ruleset

The shared helper keeps each test body short. Adding a new ruleset means: drop a fixture, add a 4-line test, done.

cargo xtask rules check extension

Current behavior: loads every JSON ruleset under rulesets/ and confirms it parses. Extend it to:

  1. For every ruleset whose fixture exists under crates/canopy-test-lib/fixtures/rulesets/, invoke the zen-engine evaluator directly (no HTTP) with the fixture’s input.

  2. Fail the check if any fixture evaluation panics or the expect.status assertion fails.

The in-process invocation keeps the check fast (no devstack required) and makes it a legitimate pre-push gate. The HTTP tests in Step 3 remain the canonical end-to-end coverage.

Steps

Step 1: Shared helper

Files: crates/canopy-test-lib/src/rules.rs (new), crates/canopy-test-lib/src/lib.rs.

Implement RulesetFixture + evaluate_fixture per Design. Add pub mod rules; to lib.rs.

Step 2: Fixtures

Files: crates/canopy-test-lib/fixtures/rulesets/*.json — 12 files.

Each fixture targets a single "canonical approve" case per program. Use realistic but minimal inputs. Keep PII scrubbed — the fixture is checked into version control.

Step 3: Tests

Files: crates/canopy-rules-client/tests/ruleset_happy_path_test.rs (new).

One #[tokio::test] per ruleset. Guard each with infrastructure_available().

Step 4: CI wiring

Files: (none expected — nextest discovers the tests automatically).

Verify cargo xtask test --integration picks them up. Confirm .gitlab-ci.yml integration stage includes them (it should, since the integration profile is *).

Step 5: cargo xtask rules check extension

Files: xtask/src/cmd/rules.rs.

Extend the existing check subcommand. Loop over each fixture, call zen_engine::Engine::evaluate directly with the input, assert the output status matches expect.status. Preserves the check as a fast, in-process, pre-push gate.

Files Touched

File Change

crates/canopy-test-lib/src/rules.rs

New helper module

crates/canopy-test-lib/src/lib.rs

pub mod rules;

crates/canopy-test-lib/fixtures/rulesets/*.json

12 new fixtures

crates/canopy-rules-client/tests/ruleset_happy_path_test.rs

12 new tests

xtask/src/cmd/rules.rs

Extend check to run fixtures in-process

CHANGELOG.adoc

Entry under == Unreleased

Verification

  1. cargo xtask rules check — 12 fixtures evaluated in-process, all green

  2. cargo xtask test --integration — 12 new integration tests pass

  3. Deliberately break one ruleset (e.g., remove an expression node in a local copy), re-run cargo xtask rules check — the corresponding fixture fails, check exits non-zero

  4. cargo xtask validate — full pre-push battery green

Documentation Updates

  • .claude/docs/testing.md — ruleset fixture convention noted in the validate step description (2026-04-18)

  • CLI Reference — document extended cargo xtask rules check (deferred; note added to fixture README)

  • CHANGELOG.adoc — entry under == Unreleased (2026-04-18)

Errata

(Empty — the previously-recorded gap on the 2 ignored fixtures was closed in Step 2 on 2026-04-27.)

Potential Improvements

  • Expand expect.output_fields / expect.exact usage. Current fixtures leave expect empty; the drift gate catches the ruleset failing to evaluate, but does not catch an edit that changes the output shape. Authoring explicit output-field presence checks for each ruleset would tighten the gate at the cost of more per-fixture maintenance. Worth doing once per-program integration tests stabilise.

  • Auto-generate the input schema. The ruleset JSONs encode the expected input fields implicitly in expression bodies. A small parser over nodes[].content.expressions[].value could emit a JSON-schema or TypeScript type per ruleset; fixtures would then be validated against the schema at check time and a promotion step could produce a skeleton for new fixtures.

Tracked follow-ups (filed 2026-04-24 after audit of plan Errata + Potential Improvements sections across the repo):

  • #330 — Auto-generate JSON-schema from JDM inputs (from Potential Improvements)

Edit this page · default