Plan: Orchestrator Parallel-Dispatch and Circuit-Breaker Tests

On this page

Status

Step Description Status

1

Extract an OrchestratorHarness test helper that wraps the orchestrator with pluggable ProgramClient mocks

Done (2026-04-19) — Revised — no trait extraction. The orchestrator dispatches via raw reqwest::Client, so an in-process axum mock server per program gives real HTTP semantics (timeout, circuit breaker, signature verify) without a production-code refactor. See errata.

2

Test: parallel dispatch — N slow program responses complete in ~max(latency), not sum(latency)

Done (2026-04-19) — 3 × 400ms delayed mocks; assertion: total elapsed < 900ms.

3

Test: per-service timeout — one slow program does not block combined result beyond the configured cutoff

Done (2026-04-19) — 3s timeout + 5s mock; assertion: elapsed < 6s, slow program → pending_verification.

4

Test: circuit breaker opens after threshold consecutive failures

Done (2026-04-24) — CircuitBreaker::state() + ProgramServiceRegistry::breaker_state_for() landed in the same MR; integration test breaker_trips_open_after_threshold_failures asserts Closed → Closed (1/2) → Open (2/2) and that subsequent short-circuit calls emit circuit breaker open in the pending basis.

5

Test: circuit breaker recovery — half-open probe succeeds, closed state resumes

Done (2026-04-24) — integration test breaker_recovers_on_successful_probe pre-trips a threshold=1/recovery=0s breaker via direct record_failure(), then runs determine(): the orchestrator’s can_call() transitions Open→HalfOpen, the probe hits a happy mock, record_success() closes the circuit, final state asserted Closed and probe landed approved.

6

Test: signature-verification failure → program result dropped, audit event emitted

Done (2026-04-19) — tampered JWS mock; assertion: no approved entry, pending basis cites signature rejection.

7

Test: optional-service degradation — a profile-skipped program returns early without contacting the service (covers ADR-005)

Deferred to adr-005 plan — that plan is the natural home for Compose-profile-matrix tests.

Branch: test/orchestrator-dispatch
Labels: type::test, priority::high, program::cross-program, service::eligibility, workflow::ready

Context

Per ADR-002, canopy-eligibility orchestrates parallel calls to each program service, verifies signed determinations, and assembles the combined result. Current coverage is limited to the happy path — every program service responds successfully within latency budget, signatures verify, and the combined result is correct.

Four high-value failure modes have no explicit test:

  1. Parallel fan-out regression — if someone replaces tokio::join! with a sequential loop, happy-path tests still pass but wall-clock latency balloons in production.

  2. Slow-service isolation — a single unresponsive program service must not delay the combined result beyond the per-call timeout.

  3. Circuit-breaker state machine — the breaker exists in crates/canopy-api/src/circuit_breaker.rs but its integration with the orchestrator is unvalidated.

  4. Signature verification — ADR-002 mandates reject-on-invalid-signature; the reject path is untested.

A fifth failure mode — optional-service graceful degradation — is called out by ADR-005 and is currently assumed to work based on reading the code. This plan proves it.

Scope

In scope:

  • A small harness that lets a test drive the orchestrator with mock ProgramClient implementations that can be instrumented (delays, failures, signatures).

  • Six new unit tests covering the failure modes above.

  • One integration test that exercises the optional-service degradation path end-to-end against the devstack.

Out of scope:

  • Changes to the circuit breaker or orchestrator production code. If a test reveals a bug, file a follow-up plan.

  • Load testing (belongs in a pre-1.0 operational plan).

  • Full chaos testing (belongs to the operational-infrastructure plan).

Dependencies

  • services/canopy-eligibility/src/orchestrator.rs — orchestrator entry point.

  • services/canopy-eligibility/src/clients.rsServiceClients and the per-program client traits.

  • crates/canopy-api/src/circuit_breaker.rs — existing breaker with CircuitBreakerState.

  • crates/canopy-signing — signing/verification helpers for crafting mock signed payloads.

Design

OrchestratorHarness

The orchestrator’s ServiceClients struct couples every program client. A harness lets a test swap individual clients for mocks:

// services/canopy-eligibility/src/orchestrator_harness.rs (new, test-only)

#[cfg(any(test, feature = "test-harness"))]
pub struct OrchestratorHarness {
    pub clients: ServiceClients,
    breaker: CircuitBreaker,
}

impl OrchestratorHarness {
    pub fn new() -> Self { /* construct with NoopProgramClient defaults */ }

    pub fn with_snap(mut self, c: impl ProgramClient + 'static) -> Self { … }
    pub fn with_tanf(mut self, c: impl ProgramClient + 'static) -> Self { … }
    pub fn with_medicaid(mut self, c: impl ProgramClient + 'static) -> Self { … }

    pub async fn determine(&self, ctx: ApplicationContext) -> CombinedResult { … }
}

The harness is gated behind #[cfg(test)] to avoid bloating release builds.

Instrumented mocks

struct DelayedProgramClient { latency: Duration, inner: NoopProgramClient }
struct FailingProgramClient { error_count: AtomicU32, max_errors: u32 }
struct BadSignatureProgramClient { /* returns determination with tampered signature */ }
struct DisabledProgramClient;  // simulates ADR-005 optional-service-off

Each is ~20 lines. Kept under tests/support/ or the harness module.

Parallel-fan-out proof

#[tokio::test]
async fn dispatch_is_parallel_not_sequential() {
    let harness = OrchestratorHarness::new()
        .with_snap(DelayedProgramClient::new(Duration::from_millis(400)))
        .with_tanf(DelayedProgramClient::new(Duration::from_millis(400)))
        .with_medicaid(DelayedProgramClient::new(Duration::from_millis(400)));
    let start = Instant::now();
    let _ = harness.determine(seed_application_context()).await;
    let elapsed = start.elapsed();
    // If sequential, elapsed ≈ 1200ms. Budget in parallel: 400 + 200 overhead.
    assert!(elapsed < Duration::from_millis(700), "actual: {elapsed:?}");
}

The 700 ms budget is generous; tighten if needed once the test is stable.

Timeout isolation

#[tokio::test]
async fn slow_program_does_not_block_combined_result() {
    let harness = OrchestratorHarness::new()
        .with_snap(HappyProgramClient::new())
        .with_medicaid(DelayedProgramClient::new(Duration::from_secs(30)));
    let start = Instant::now();
    let result = harness.determine(seed_application_context()).await;
    let elapsed = start.elapsed();
    assert!(elapsed < Duration::from_secs(6));  // per-service timeout is 5s
    assert!(result.medicaid.is_none());
    assert!(result.snap.is_some());
}

Circuit breaker

Two tests:

#[tokio::test]
async fn breaker_opens_after_threshold_failures() {
    let failing = FailingProgramClient { /* always fails */ };
    let harness = OrchestratorHarness::new().with_snap(failing);
    for _ in 0..3 { let _ = harness.determine(ctx.clone()).await; }
    assert_eq!(harness.breaker_state_for("snap"), CircuitBreakerState::Open);
}

#[tokio::test]
async fn breaker_closes_after_successful_probe() {
    let recovering = RecoveringProgramClient::new(errors_before_recovery: 3);
    let harness = OrchestratorHarness::new().with_snap(recovering);
    for _ in 0..3 { let _ = harness.determine(ctx.clone()).await; }
    assert_eq!(harness.breaker_state_for("snap"), CircuitBreakerState::Open);
    tokio::time::sleep(breaker_cooldown()).await;
    let _ = harness.determine(ctx.clone()).await;  // half-open probe
    assert_eq!(harness.breaker_state_for("snap"), CircuitBreakerState::Closed);
}

Signature verification

#[tokio::test]
async fn bad_signature_is_rejected_and_audited() {
    let bad = BadSignatureProgramClient;
    let (audit_tx, mut audit_rx) = tokio::sync::mpsc::unbounded_channel();
    let harness = OrchestratorHarness::new()
        .with_snap(bad)
        .with_audit_sink(audit_tx);
    let result = harness.determine(seed_application_context()).await;
    assert!(result.snap.is_none());
    let event = audit_rx.try_recv().unwrap();
    assert_eq!(event.kind, "signature_verification_failed");
}

Optional-service degradation (integration)

Separate test file because it needs a real devstack profile:

// services/canopy-eligibility/tests/optional_service_degradation_test.rs

#[tokio::test]
async fn caps_disabled_profile_skips_caps() {
    if !infrastructure_available().await { return; }
    // Requires devstack started with COMPOSE_PROFILES=snap-only
    let cfg = TestConfig::from_env();
    if !cfg.caps_url.is_empty() { return; /* caps is deployed; skip */ }
    let client = TestClient::new(&cfg.eligibility_url);
    let resp = client.post_json("/v1/eligibility/determine", &seed_ctx_json()).await;
    resp.assert_status(200);
    let body = resp.json_value();
    assert!(body["caps"].is_null());
    assert!(body["snap"].is_object());
}

Steps

Step 1: Harness

Files: services/canopy-eligibility/src/orchestrator_harness.rs (new, #[cfg(test)] guarded), services/canopy-eligibility/src/lib.rs.

Build the harness and mock clients per Design. Keep the API stable — every test depends on it.

Step 2–6: Unit tests

Files: services/canopy-eligibility/tests/orchestrator_dispatch_test.rs (new).

One test per Step. Use the harness from Step 1. Keep each test body under 40 lines.

Step 7: Optional-service degradation integration test

Files: services/canopy-eligibility/tests/optional_service_degradation_test.rs (new).

Drives the orchestrator HTTP API against a snap-only devstack. Skips if CAPS is deployed (the test is only meaningful when the optional service is absent).

CI implication: .gitlab-ci.yml may need a second integration stage that runs with COMPOSE_PROFILES=snap-only. File a follow-up plan (ci-profile-integration-stage.adoc) if this is non-trivial — do not block this plan on it.

Files Touched

File Change

services/canopy-eligibility/src/orchestrator_harness.rs

New test harness + mock clients

services/canopy-eligibility/src/lib.rs

#[cfg(test)] mod orchestrator_harness;

services/canopy-eligibility/tests/orchestrator_dispatch_test.rs

6 new unit tests

services/canopy-eligibility/tests/optional_service_degradation_test.rs

1 new integration test

CHANGELOG.adoc

Entry under == Unreleased

Verification

  1. cargo nextest run -p canopy-eligibility --lib — 6 new unit tests pass

  2. cargo xtask test --integration — existing integration tests still green

  3. With COMPOSE_PROFILES=snap-only cargo xtask dev start, run the optional-service test manually — passes

  4. Deliberately break parallel dispatch (replace tokio::join! with sequential awaits in a local branch) — the parallel test fails with a clear elapsed-time assertion

  5. Deliberately weaken signature verification (always-ok stub) — the bad-signature test fails

Documentation Updates

  • Testing — deferred; the fixture pattern is already documented for cargo xtask rules check, and the dispatch tests follow the same "integration test against in-process mocks" approach without requiring new general-purpose docs

  • CHANGELOG.adoc — entry under == Unreleased (2026-04-19)

Errata

Approach pivot: HTTP mocks instead of ProgramClient trait

The plan as originally written called for extracting a ProgramClient trait to swap mock implementations for the real HTTP client. In practice, canopy-eligibility/src/orchestrator.rs dispatches by constructing URLs from ProgramServiceRegistry and calling reqwest::Client::post directly — there is no trait to implement. Extracting one would be a meaningful production-code refactor and violates this plan’s "out of scope: changes to the orchestrator production code" rule.

In-process axum mock servers on ephemeral ports give real HTTP semantics for the same tests: the orchestrator’s circuit breaker, per-program timeout, signature verification, and quarantine logic all run against real sockets. The only production-code change required is one line on ProgramServiceRegistry — a from_services(HashMap<…​>) constructor so a test can build a registry pointing at the mock addresses. Existing from_env() path is untouched.

Circuit-breaker tests deferred

Steps 4 and 5 were dropped from this MR. The CircuitBreaker type in canopy-api has can_call(), record_success(), record_failure() — but no way to inspect the state from outside. To assert "breaker is open after 5 failures" a test needs breaker_state_for(program: Program) → CircuitBreakerState or similar on ProgramServiceRegistry, which does not currently exist.

Building that inspection helper is a small addition to canopy-api::circuit_breaker (expose the current state) plus a passthrough on ProgramServiceRegistry. Filed as a follow-up because the inspection API is a distinct piece of production code that does not belong under "orchestrator dispatch tests" and would expand this MR’s surface.

Until those helpers exist, the happy-path and failing-service coverage comes from real-service integration tests in the per-program crates (e.g., canopy-snap::snap_test) that exercise the orchestrator indirectly.

Test-process shutdown takes ~30s

Each test reports ~30s wall-clock even though the orchestrator assertion fires in a few hundred milliseconds. The delay is the tokio runtime waiting for background tasks (telemetry exporters, sqlx pool drain, axum server join handles) to finish draining before the process exits. MockHandle::drop aborts the axum tasks, but something else (likely canopy-common::telemetry or the sqlx pool) holds the runtime for a fixed 30s shutdown window.

This is cosmetic — the test assertions complete in <1s and three tests still finish in 30s wall-clock when run in parallel (not 90s). Filed as a quality-of-life follow-up; not blocking the drift-gate value.

Potential Improvements

  • Expose circuit-breaker state on ProgramServiceRegistry via state_for(program: Program) → CircuitBreakerState so the two deferred breaker tests can be written without inspecting private fields. Small PR (~30 lines) once scoped.

  • Controlled runtime shutdown via a test helper that `select!`s on a oneshot channel inside the mock handlers, so test end flushes in <1s instead of 30s.

  • Shared mock helper in canopy-test-lib. The mock_program / mock_persons pattern will be needed again by ADR-005 graceful-degradation tests (per that plan’s unit-test design). Extracting them to canopy_test_lib::orchestrator_mocks before the ADR-005 work starts would avoid a copy-paste. Do this alongside the next consumer.


Tracked follow-ups (filed 2026-04-24 after audit of plan Errata + Potential Improvements sections across the repo):

  • #332 — Reduce test-process shutdown from 30s to <1s (from Errata)

  • #333 — Controlled runtime shutdown helper (from Potential Improvements)

  • #334 — Shared mock helper in canopy-test-lib (from Potential Improvements)

Edit this page · default