Plan: Orchestrator Parallel-Dispatch and Circuit-Breaker Tests
On this page
Status
| Step | Description | Status |
|---|---|---|
1 |
Extract an |
Done (2026-04-19) — Revised — no trait extraction. The orchestrator dispatches via raw |
2 |
Test: parallel dispatch — N slow program responses complete in ~max(latency), not sum(latency) |
Done (2026-04-19) — 3 × 400ms delayed mocks; assertion: total elapsed < 900ms. |
3 |
Test: per-service timeout — one slow program does not block combined result beyond the configured cutoff |
Done (2026-04-19) — 3s timeout + 5s mock; assertion: elapsed < 6s, slow program → pending_verification. |
4 |
Test: circuit breaker opens after threshold consecutive failures |
Done (2026-04-24) — |
5 |
Test: circuit breaker recovery — half-open probe succeeds, closed state resumes |
Done (2026-04-24) — integration test |
6 |
Test: signature-verification failure → program result dropped, audit event emitted |
Done (2026-04-19) — tampered JWS mock; assertion: no approved entry, pending basis cites signature rejection. |
7 |
Test: optional-service degradation — a profile-skipped program returns early without contacting the service (covers ADR-005) |
Deferred to adr-005 plan — that plan is the natural home for Compose-profile-matrix tests. |
Branch: test/orchestrator-dispatch
Labels: type::test, priority::high, program::cross-program, service::eligibility, workflow::ready
Context
Per ADR-002, canopy-eligibility orchestrates parallel calls to each program service, verifies signed determinations, and assembles the combined result. Current coverage is limited to the happy path — every program service responds successfully within latency budget, signatures verify, and the combined result is correct.
Four high-value failure modes have no explicit test:
-
Parallel fan-out regression — if someone replaces
tokio::join!with a sequential loop, happy-path tests still pass but wall-clock latency balloons in production. -
Slow-service isolation — a single unresponsive program service must not delay the combined result beyond the per-call timeout.
-
Circuit-breaker state machine — the breaker exists in
crates/canopy-api/src/circuit_breaker.rsbut its integration with the orchestrator is unvalidated. -
Signature verification — ADR-002 mandates reject-on-invalid-signature; the reject path is untested.
A fifth failure mode — optional-service graceful degradation — is called out by ADR-005 and is currently assumed to work based on reading the code. This plan proves it.
Scope
In scope:
-
A small harness that lets a test drive the orchestrator with mock
ProgramClientimplementations that can be instrumented (delays, failures, signatures). -
Six new unit tests covering the failure modes above.
-
One integration test that exercises the optional-service degradation path end-to-end against the devstack.
Out of scope:
-
Changes to the circuit breaker or orchestrator production code. If a test reveals a bug, file a follow-up plan.
-
Load testing (belongs in a pre-1.0 operational plan).
-
Full chaos testing (belongs to the operational-infrastructure plan).
Dependencies
-
services/canopy-eligibility/src/orchestrator.rs— orchestrator entry point. -
services/canopy-eligibility/src/clients.rs—ServiceClientsand the per-program client traits. -
crates/canopy-api/src/circuit_breaker.rs— existing breaker withCircuitBreakerState. -
crates/canopy-signing— signing/verification helpers for crafting mock signed payloads.
Design
OrchestratorHarness
The orchestrator’s ServiceClients struct couples every program client. A harness lets a test swap individual clients for mocks:
// services/canopy-eligibility/src/orchestrator_harness.rs (new, test-only)
#[cfg(any(test, feature = "test-harness"))]
pub struct OrchestratorHarness {
pub clients: ServiceClients,
breaker: CircuitBreaker,
}
impl OrchestratorHarness {
pub fn new() -> Self { /* construct with NoopProgramClient defaults */ }
pub fn with_snap(mut self, c: impl ProgramClient + 'static) -> Self { … }
pub fn with_tanf(mut self, c: impl ProgramClient + 'static) -> Self { … }
pub fn with_medicaid(mut self, c: impl ProgramClient + 'static) -> Self { … }
pub async fn determine(&self, ctx: ApplicationContext) -> CombinedResult { … }
}
The harness is gated behind #[cfg(test)] to avoid bloating release builds.
Instrumented mocks
struct DelayedProgramClient { latency: Duration, inner: NoopProgramClient }
struct FailingProgramClient { error_count: AtomicU32, max_errors: u32 }
struct BadSignatureProgramClient { /* returns determination with tampered signature */ }
struct DisabledProgramClient; // simulates ADR-005 optional-service-off
Each is ~20 lines. Kept under tests/support/ or the harness module.
Parallel-fan-out proof
#[tokio::test]
async fn dispatch_is_parallel_not_sequential() {
let harness = OrchestratorHarness::new()
.with_snap(DelayedProgramClient::new(Duration::from_millis(400)))
.with_tanf(DelayedProgramClient::new(Duration::from_millis(400)))
.with_medicaid(DelayedProgramClient::new(Duration::from_millis(400)));
let start = Instant::now();
let _ = harness.determine(seed_application_context()).await;
let elapsed = start.elapsed();
// If sequential, elapsed ≈ 1200ms. Budget in parallel: 400 + 200 overhead.
assert!(elapsed < Duration::from_millis(700), "actual: {elapsed:?}");
}
The 700 ms budget is generous; tighten if needed once the test is stable.
Timeout isolation
#[tokio::test]
async fn slow_program_does_not_block_combined_result() {
let harness = OrchestratorHarness::new()
.with_snap(HappyProgramClient::new())
.with_medicaid(DelayedProgramClient::new(Duration::from_secs(30)));
let start = Instant::now();
let result = harness.determine(seed_application_context()).await;
let elapsed = start.elapsed();
assert!(elapsed < Duration::from_secs(6)); // per-service timeout is 5s
assert!(result.medicaid.is_none());
assert!(result.snap.is_some());
}
Circuit breaker
Two tests:
#[tokio::test]
async fn breaker_opens_after_threshold_failures() {
let failing = FailingProgramClient { /* always fails */ };
let harness = OrchestratorHarness::new().with_snap(failing);
for _ in 0..3 { let _ = harness.determine(ctx.clone()).await; }
assert_eq!(harness.breaker_state_for("snap"), CircuitBreakerState::Open);
}
#[tokio::test]
async fn breaker_closes_after_successful_probe() {
let recovering = RecoveringProgramClient::new(errors_before_recovery: 3);
let harness = OrchestratorHarness::new().with_snap(recovering);
for _ in 0..3 { let _ = harness.determine(ctx.clone()).await; }
assert_eq!(harness.breaker_state_for("snap"), CircuitBreakerState::Open);
tokio::time::sleep(breaker_cooldown()).await;
let _ = harness.determine(ctx.clone()).await; // half-open probe
assert_eq!(harness.breaker_state_for("snap"), CircuitBreakerState::Closed);
}
Signature verification
#[tokio::test]
async fn bad_signature_is_rejected_and_audited() {
let bad = BadSignatureProgramClient;
let (audit_tx, mut audit_rx) = tokio::sync::mpsc::unbounded_channel();
let harness = OrchestratorHarness::new()
.with_snap(bad)
.with_audit_sink(audit_tx);
let result = harness.determine(seed_application_context()).await;
assert!(result.snap.is_none());
let event = audit_rx.try_recv().unwrap();
assert_eq!(event.kind, "signature_verification_failed");
}
Optional-service degradation (integration)
Separate test file because it needs a real devstack profile:
// services/canopy-eligibility/tests/optional_service_degradation_test.rs
#[tokio::test]
async fn caps_disabled_profile_skips_caps() {
if !infrastructure_available().await { return; }
// Requires devstack started with COMPOSE_PROFILES=snap-only
let cfg = TestConfig::from_env();
if !cfg.caps_url.is_empty() { return; /* caps is deployed; skip */ }
let client = TestClient::new(&cfg.eligibility_url);
let resp = client.post_json("/v1/eligibility/determine", &seed_ctx_json()).await;
resp.assert_status(200);
let body = resp.json_value();
assert!(body["caps"].is_null());
assert!(body["snap"].is_object());
}
Steps
Step 1: Harness
Files: services/canopy-eligibility/src/orchestrator_harness.rs (new, #[cfg(test)] guarded), services/canopy-eligibility/src/lib.rs.
Build the harness and mock clients per Design. Keep the API stable — every test depends on it.
Step 2–6: Unit tests
Files: services/canopy-eligibility/tests/orchestrator_dispatch_test.rs (new).
One test per Step. Use the harness from Step 1. Keep each test body under 40 lines.
Step 7: Optional-service degradation integration test
Files: services/canopy-eligibility/tests/optional_service_degradation_test.rs (new).
Drives the orchestrator HTTP API against a snap-only devstack. Skips if CAPS is deployed (the test is only meaningful when the optional service is absent).
CI implication: .gitlab-ci.yml may need a second integration stage that runs with COMPOSE_PROFILES=snap-only. File a follow-up plan (ci-profile-integration-stage.adoc) if this is non-trivial — do not block this plan on it.
Files Touched
| File | Change |
|---|---|
|
New test harness + mock clients |
|
|
|
6 new unit tests |
|
1 new integration test |
|
Entry under |
Verification
-
cargo nextest run -p canopy-eligibility --lib— 6 new unit tests pass -
cargo xtask test --integration— existing integration tests still green -
With
COMPOSE_PROFILES=snap-only cargo xtask dev start, run the optional-service test manually — passes -
Deliberately break parallel dispatch (replace
tokio::join!with sequential awaits in a local branch) — the parallel test fails with a clear elapsed-time assertion -
Deliberately weaken signature verification (always-ok stub) — the bad-signature test fails
Documentation Updates
-
Testing — deferred; the fixture pattern is already documented for
cargo xtask rules check, and the dispatch tests follow the same "integration test against in-process mocks" approach without requiring new general-purpose docs -
CHANGELOG.adoc— entry under== Unreleased(2026-04-19)
Errata
Approach pivot: HTTP mocks instead of ProgramClient trait
The plan as originally written called for extracting a ProgramClient trait to swap mock implementations for the real HTTP client. In practice, canopy-eligibility/src/orchestrator.rs dispatches by constructing URLs from ProgramServiceRegistry and calling reqwest::Client::post directly — there is no trait to implement. Extracting one would be a meaningful production-code refactor and violates this plan’s "out of scope: changes to the orchestrator production code" rule.
In-process axum mock servers on ephemeral ports give real HTTP semantics for the same tests: the orchestrator’s circuit breaker, per-program timeout, signature verification, and quarantine logic all run against real sockets. The only production-code change required is one line on ProgramServiceRegistry — a from_services(HashMap<…>) constructor so a test can build a registry pointing at the mock addresses. Existing from_env() path is untouched.
Circuit-breaker tests deferred
Steps 4 and 5 were dropped from this MR. The CircuitBreaker type in canopy-api has can_call(), record_success(), record_failure() — but no way to inspect the state from outside. To assert "breaker is open after 5 failures" a test needs breaker_state_for(program: Program) → CircuitBreakerState or similar on ProgramServiceRegistry, which does not currently exist.
Building that inspection helper is a small addition to canopy-api::circuit_breaker (expose the current state) plus a passthrough on ProgramServiceRegistry. Filed as a follow-up because the inspection API is a distinct piece of production code that does not belong under "orchestrator dispatch tests" and would expand this MR’s surface.
Until those helpers exist, the happy-path and failing-service coverage comes from real-service integration tests in the per-program crates (e.g., canopy-snap::snap_test) that exercise the orchestrator indirectly.
Test-process shutdown takes ~30s
Each test reports ~30s wall-clock even though the orchestrator assertion fires in a few hundred milliseconds. The delay is the tokio runtime waiting for background tasks (telemetry exporters, sqlx pool drain, axum server join handles) to finish draining before the process exits. MockHandle::drop aborts the axum tasks, but something else (likely canopy-common::telemetry or the sqlx pool) holds the runtime for a fixed 30s shutdown window.
This is cosmetic — the test assertions complete in <1s and three tests still finish in 30s wall-clock when run in parallel (not 90s). Filed as a quality-of-life follow-up; not blocking the drift-gate value.
Potential Improvements
-
Expose circuit-breaker state on
ProgramServiceRegistryviastate_for(program: Program) → CircuitBreakerStateso the two deferred breaker tests can be written without inspecting private fields. Small PR (~30 lines) once scoped. -
Controlled runtime shutdown via a test helper that `select!`s on a oneshot channel inside the mock handlers, so test end flushes in <1s instead of 30s.
-
Shared mock helper in
canopy-test-lib. The mock_program / mock_persons pattern will be needed again by ADR-005 graceful-degradation tests (per that plan’s unit-test design). Extracting them tocanopy_test_lib::orchestrator_mocksbefore the ADR-005 work starts would avoid a copy-paste. Do this alongside the next consumer.
Tracked follow-ups (filed 2026-04-24 after audit of plan Errata + Potential Improvements sections across the repo):