Known Issues and Lessons Learned

Garage S3 crashes on startup

Garage v2.2.0+ requires rpc_bind_addr AND rpc_secret in garage.toml. Both fields are mandatory.

Set both fields (already configured in devstack/garage/garage.toml).

container name already in use on cargo xtask dev start

Orphan containers from a prior compose project name shadowing the current one.

The xtask devstack guard now passes --remove-orphans to all up/down calls. If you see this manually, run docker compose --profile full down --remove-orphans.

Port conflicts after a crashed devstack

Previous run didn’t shut down cleanly; ports stay bound.

docker compose --profile full down --remove-orphans. If still stuck, docker ps -a and docker rm -f the leftovers.

docker compose stop --profile X errors out

stop doesn’t accept --profile flag (unlike up / down).

cargo xtask dev stop calls docker compose stop without a profile flag. Match that pattern in scripts.

Cold devstack start takes 10+ minutes

Building the workspace from scratch (Alpine target, full dep graph). Cached builds are seconds.

First-time setup is unavoidable. Subsequent restarts use the named-volume cache. Don’t docker system prune -af between runs.

Compose / env edits don’t take effect after docker compose up -d

Devstack compose + env changes are applied by the xtask devstack guard, not a bare compose up.

Drive the devstack with cargo xtask dev refresh (applies compose edits + minimum rebuild), never raw docker compose up -d.

canopy-persons comes up with ssn_last_four null / empty CANOPY_ENCRYPTION_KEY after a compose recreate (e.g. create_person_returns_201 fails on the push after a green one)

Secrets were delivered only via per-call process-env injection. Any compose recreate that didn’t thread the decrypted secret env — notably the post-validate seed/e2e steps in the pre-push battery — rebuilt canopy-persons with an empty ${CANOPY_ENCRYPTION_KEY:-}, leaving it broken for the next push. Only canopy-persons is affected: it reads CANOPY_ENCRYPTION_KEY with no on-disk fallback, whereas the ${CANOPY_*__SIGNING_KEY:-} keys fall back to .keys/.

Fixed (#734): the shared: secrets are written to a gitignored .env floor (docker::write_shared_secret_floor) that docker compose auto-reads on every invocation, so recreates interpolate the real key regardless of which path triggers them. Per-call process-env injection still overrides it. If it somehow recurs, cargo xtask dev reload; confirm with docker exec canopy-canopy-persons-1 printenv CANOPY_ENCRYPTION_KEY.

Test users can’t log in

Password hashes in definitions.json must be bcrypt ($2a$…​); argon2 is silently rejected.

The seeded definitions.json already uses bcrypt. If adding new test users, hash with htpasswd -bnBC 10 "" password | tr -d ':\n'.

Password grant flow fails for seeded users

emailVerified defaults to false for new realm users; Keycloak rejects unverified emails on password grant.

Set "emailVerified": true in definitions.json for every test user.

JWT validation fails with "issuer mismatch"

When running in Docker, the JWT’s iss claim uses the external URL while services try to fetch JWKS from a Docker-internal URL.

Configure CANOPY_<SERVICE>KEYCLOAK_ISSUER (external) and CANOPY_<SERVICE>KEYCLOAK_URL (internal) separately. The split-URL JwksProvider::with_fetch_url in canopy-auth handles this.

Repeated JWKS fetches when an unknown kid arrives

The JwksProvider force-refreshes on unknown-kid; without backoff, a stale token can hammer Keycloak.

30-second debounce on forced refreshes is built into JwksProvider. No action needed; documented for incident reviewers.

Integration tests skipped silently in CI

infrastructure_available() returns false when devstack isn’t reachable; without a CI guard, tests skip instead of fail.

In CI, set CANOPY_CI=true. The helper then panics instead of returning false, so missing devstack fails the build.

Pre-push validate takes minutes longer than expected

cargo clippy and cargo nextest use different cargo profiles (dev vs test) by default, forcing a recompile between them.

cargo xtask validate runs clippy with --profile test so artefacts share with nextest. Manual invocations should match.

evaluation_creates_audit_trail test occasionally fails

Test publishes an event then queries canopy-security; RabbitMQ delivery timing can race the query.

Re-run usually passes. Tracked as a known transient. If persistent, increase the polling interval in the test.

Pre-push hook runs E2E unconditionally

Playwright runs in a Docker container; no node_modules shortcut.

Container builds + runs in ~50 s with warm cache. To skip locally: git push --no-verify (only for non-functional changes).

First container test run is slow

cargo xtask test --integration runs in the new in-network test runner (ADR-015). First run compiles all test binaries inside the container.

Subsequent runs hit the canopy_integration_target named volume. Don’t prune that volume between runs. To skip: cargo xtask test --integration --host (legacy host-direct path).

An E2E page.request.post to a canopy-web handler returns 403 before the handler runs

csrf::csrf_middleware is a route_layer mounted before the handlers; a raw POST without the CSRF token is rejected by the middleware, so the test exercises CSRF, not your handler.

Grab input[name="_csrf"] from an in-scope page and send it as the X-CSRF-Token header. Canonical pattern: tests/e2e/specs/intake-partial-demo.spec.ts.

Can’t reproduce a server-side error/outage state in a Playwright test

The failing upstream call is server-to-server, so route-interception in the browser can’t trigger it — the service must actually be down.

docker stop canopy-canopy-<svc>-1, then a temp Playwright project with dependencies: ['auth-setup'] + a spec that page.screenshot({ path: 'results/foo.png' }) (only results/ is bind-mounted, to host test-results/e2e/); run cargo xtask e2e --no-refresh — --project <tmp>; restart the service; revert the temp project.

404 rule set not found from canopy-rules

NamedFilesystemLoader loads by the top-level name in each JSON, not the filename.

Use georgia-<program>-<purpose> (e.g. georgia-caps-eligibility), not caps-eligibility. Match the filename for sanity.

Decision-table cell with quoted-string condition fails to match

zen-engine 0.55 string matching in DT cells is unreliable.

Use an expressionNode with field == 'value' instead of a DT cell.

Ternary condition ? "string" : "string" fails at eval time

zen-engine’s expression evaluator chokes on string literals inside ternaries.

Build string outputs in a separate expression node. Or move the branching into a DT.

DT output cell typed as bool is actually a string in JSON

zen-engine emits all DT outputs as strings.

Parse with v.as_bool().unwrap_or_else(|| v.as_str() == Some("true")).

Downstream nodes see null for fields the upstream emitted

Transform nodes drop input fields by default.

Set passThrough: true on every transform node where the output should retain unmodified inputs.

Subscriber struct deserialisation fails on FTI fields

ADR-004 publisher-side wire scrubbing drops FTI field names before publishing.

Subscriber structs must mark possibly-scrubbed fields #[serde(default)] Option<T> and treat absent as zero-value.

Field-name drift between publisher and subscriber goes uncaught

Schema-less JSON envelope; no compile-time check.

Cross-service integration tests are the backstop. When adding a wire field, pick the subscriber-side name first and use it on the wire.

Service publishes events but subscribers don’t receive

Connection dropped (e.g. docker compose restart rabbitmq) and the in-process channel went stale.

canopy-mq::ConnectionManager ships exponential-backoff reconnect (issue #313). If a service is wedged, restart it. The reconnect is automatic for transient broker outages.

pool timed out while waiting for an open connection under shared-db profile

17 services + integration tests against one Postgres instance exhaust the default 100-connection cap.

command: ["postgres", "-c", "max_connections=400"] in compose. Already configured for the shared-db profile.

Build fails with "DATABASE_URL not set" during cargo build

sqlx compile-time query verification needs a live DB or cached metadata.

Production builds set SQLX_OFFLINE=true and check in .sqlx/ query metadata. For ad-hoc local builds without devstack, set SQLX_OFFLINE=true and run cargo sqlx prepare after schema changes.

Want to roll back a migration

Migrations are forward-only by convention. No down.sql files.

Write a corrective migration (additive fix). For data corruption, restore from backup per the runbook.

Warning about DATABASE_URL not matching service name

Under --shared-db mode, all services use one Postgres instance, so the URL hostname doesn’t match the service name.

Warning is intentional. Errors only on truly inconsistent configurations.

Program determinations bucket as signature_quarantined in dev

Per-program signing keys must be loaded both by the program service (signing) and canopy-eligibility (verification). Both env vars or both files.

cargo xtask dev start calls ensure_signing_keys which generates .keys/<program>-{private,public}.pem for snap/tanf/medicaid/caps/wic. The orchestrator’s VerifyingKeyRegistry::from_env_or_keys_dir falls back to the public PEM on disk when CANOPY_VERIFY_KEY_* is unset. Fixed in #338 / !138.

canopy-medicaid signature verification fails post-DB-roundtrip

Service signed serde_json::to_vec(&determination) with full nanosecond created_at. PostgreSQL TIMESTAMPTZ truncates to microseconds; wire JSON has the truncated value, so re-serialised bytes differ from signed bytes.

canopy-snap normalises created_at to microsecond precision and binds it explicitly (see services/canopy-snap/src/store/mod.rs::create_snap_determination). Other program services that use this pattern need the same treatment. #338 / !138.

Decimal benefit_amount serialises differently in-memory vs from DB

Decimal::from(298) serialises as "298" but PostgreSQL NUMERIC(10,2) returns "298.00".

Call .rescale(2) on benefit Decimals before signing. canopy-snap does this since !138.

Orchestrator-dispatched Medicaid determinations scored the applicant at a hardcoded age 30 (and $0 resources)

The orchestrator builds one generic ApplicationContext, threads each member’s real age onto members[] and sets applicant_person_id to the head, but never sets a top-level age/disability_status/countable_resources. The Medicaid handler read the top-level fields (ctx.age.unwrap_or(30), etc.), so the real age — present on members[] — was never read, silently mis-scoring every age-banded COA. Root cause: no rule that a determination’s wire-shape must be satisfiable from the orchestrator context.

Resolve the applicant’s age/disability from ctx.members[] (top-level kept as an override channel). Fixed in epic &63’s first slice. Medical-expense aggregation now fixed — the Medicaid handler sums ctx.expenses (expense_type=="medical") into the MN-spenddown medical_expenses_monthly (medical half of #856). The per-program context-building is now governed by a ratified implementation design — ADR-035 (Per-Subject Determination + Per-Program Context Mappers, Accepted 2026-06-16) — which replaces the broadcast with typed per-program mappers and makes determinations per-subject (Medicaid per member, CAPS per child, WIC per participant; SNAP/TANF stay household), staged CAPS (#857) → Medicaid (#860) → WIC (#769). CAPS Slice 1 is itself sub-sliced (MR1, MR2, MR3a, MR3b, MR4); MR1 (carrier foundation) + MR2 (per-program map_context send seam) + MR3a (orchestrator per-subject receive plumbing) + MR3b (CAPS per-child determinations + the complete-or-provisional seam) are merged — MR1 added person_id on the signed envelope + ProgramResult, the MissingInput/missing_inputs carrier, and a nullable program_determinations.person_id column; MR2 routed dispatch through a per-program map_context mapper in place of the generic broadcast; MR3a made the orchestrator accept a bare envelope or a {determinations:[…​]} list (normalize to a Vec, verify/persist/bucket per determination); MR3b reshaped CapsApplicationContext to a per-child children[] household context (canopy-caps returns one signed determination per child, persisted atomically in one transaction), widened map_context to Result<ApplicationContext, ContextError> so the orchestrator→CAPS outcome is an honest household-level input_unsatisfiable (its worker-facts are unsourceable pre-corpus) instead of a silent 422, and landed the (eligibility_request_id, program, person_id) NULLS NOT DISTINCT unique key + a persist-failure→pending guard. MR1/MR2/MR3a are behavior-inert; MR3b is the first per-subject behavior. The untagged ProgramInput enum + per-child input_unsatisfiable surfacing are deferred (dead-code / unreachable until the ADR-027 corpus, #56 / #860). MR4 (the household-wide grouped-roster Determination tab — closes #857) is merged: canopy-web’s five per-program determination tabs become one scope-filtered roster grouped by program → subject (SNAP/TANF strip, Medicaid/CAPS/WIC per-subject rows), a cash-only summary (no false grand total), and the amber input_unsatisfiable checklist from the immediate determine response; the 16 #392 caseworker action forms are re-homed into a per-program-group "Actions ▾" disclosure, with Medicaid resolve-quarantined UI-hidden + backend-403’d to operator roles (is_determination_operator ⊆ can_write). Slice 2 (#860, Medicaid per-member) is merged: canopy-medicaid’s /v1/determine now enumerates ctx.members and returns one signed determination per member (the {determinations:[…​]} body the orchestrator already accepts), each scored on its own age + disability, persisted atomically; an invalid/duplicate members[] person_id is 422, an empty list falls back to a single applicant determination. The income test stays household-level (per-member budget-group composition — compose_magi_budget_group — is deferred to #864, blocked on the tax_filing_status worker-fact, 42 CFR 435.603(f) / #858 / &56). Slice 3 (#769, WIC per-participant) is merged — ADR-035 Slice 1 (CAPS / Medicaid / WIC) is now COMPLETE: canopy-wic’s /v1/determine reshapes WicApplicationContext to a per-participant participants[] household context and returns one signed determination per participant (the {determinations:[…​]} body, persisted + events in one tx; person_id set before signing), each scored on its own category against the shared economic-unit income (7 CFR 246.7); map_wic_context joins CAPS as a complete-or-provisional arm, erring input_unsatisfiable naming the three worker-facts (participant_category / nutritional_risk_documented / is_breastfeeding_fully) until the &56 corpus (#858) — replacing the silent 422; an empty participants[] or a duplicate person_id is 422, and nutritional risk stays service-verified from wic_nutritional_risk_assessments (don’t-trust-caller). The countable_resources aggregation (#856 resource half) is now fixed — the handler conservatively aggregates the threaded ctx.assets under 42 CFR 435.601(b) (only unambiguously-countable categories sum, so errors only under-count; the scalar stays an override channel); the full SSI methodology (equity valuation, first-moment-of-month, per-asset homeplace/vehicle designation) remains the unbound resource-counting-methodology action under #778. Still remaining under epic &63 (now ADR-035 slices): per-member budget-group income precision (#864); the EE15/ELE per-member propagation reconciliation (the household medicaid_assigned_group scalar stays interim — deferred within the Medicaid slice).

Orchestrator-dispatched income/expense amounts reached programs un-normalized (weekly/annual mislabeled as monthly)

The orchestrator threaded raw canopy-persons {amount, frequency} records; the medicaid/tanf determine contracts drop frequency (deserialize amount→monthly_amount), so a non-monthly amount silently became a "monthly" value — under-counting ~4.3× (weekly) or over-counting 12× (annual). Three divergent converters (SNAP/ELE/web) each re-implemented the conversion. Root cause: normalization happened per-consumer instead of at the orchestrator input-building seam (ADR-034).

Shared canopy_reference::money::to_monthly (#861, factors from the cited federal snap-budgeting-factors.json); the orchestrator normalizes income/expenses to monthly before dispatch (ADR-034 seam); SNAP + ELE delegate. SNAP byte-identical. canopy-web’s display-only f64 converter deferred to a follow-up.