Troubleshooting Guide

Symptom: cargo xtask dev start fails with "port already in use"

Fix:

Symptom: "The container name /canopy-canopy-tanf-1 is already in use"

Cause: Docker has orphaned containers from a prior run that weren’t fully removed.

Fix:

The devstack staleness guard now passes --remove-orphans to all compose calls automatically.

Symptom: Tests pass locally but fail against devstack, or devstack behavior doesn’t match recent code changes.

Cause: Containers running old binaries. Docker cached the previous build.

Fix:

Or use the staleness guard: cargo xtask dev status shows whether devstack is current. cargo xtask dev refresh performs the minimum rebuild needed.

Symptom: Garage container exits immediately with configuration error.

Cause: Missing rpc_bind_addr field in devstack/garage/garage.toml (required by Garage v2.2.0+).

Fix: Ensure garage.toml contains:

Symptom: Program service fails to start in --shared-db mode.

Cause: The shared PostgreSQL instance may not have all program databases created.

Fix: Databases are created by each service’s sqlx::migrate!() call, but the database itself must exist first. Check that all canopy_* databases are created on the shared instance.

Symptom: cargo xtask dev start or cargo xtask dev restart aborts with Error: parse sops JSON: expected value at line 1 column 1.

Cause: Only a cold dev start or dev restart re-decrypts secrets via SOPS (dev refresh and dev clean reuse the running stack and do not re-decrypt). The error means SOPS isn’t resolving the age key — it ran but emitted nothing. The decrypt path is sops --decrypt --output-type json secrets/dev.yaml, which needs the age key at ~/.config/sops/age/keys.txt.

Fix: 1. Confirm SOPS can decrypt directly — it should emit JSON:

Symptom: acquire_token_for("jane.doe", "password") returns None.

Causes: 1. Keycloak not running: check docker compose ps keycloak 2. emailVerified not set to true in devstack/keycloak/definitions.json 3. Keycloak realm not imported: check Keycloak admin console at http://localhost:8180

Symptom: All authenticated requests return 401 even with a valid token.

Cause: JWKS cache is stale or Keycloak rotated keys.

Fix: The JwksProvider auto-refreshes on unknown kid with a 30-second debounce. Wait 30 seconds and retry. If persistent, restart the affected service to force a fresh JWKS fetch.

Symptom: JWT iss claim doesn’t match KEYCLOAK_ISSUER setting.

Cause: In Docker, the public issuer URL (how browsers see Keycloak) differs from the internal Docker URL (how services fetch JWKS).

Fix: Set both:

Symptom: Integration tests show 0 passed, 0 failed (all skipped).

Cause: Devstack not running. infrastructure_available() returns false and tests skip.

Fix: Start devstack first: cargo xtask dev start --shared-db

In CI, set CANOPY_CI=true — the guard panics instead of skipping, ensuring tests never silently skip.

Symptom: cargo nextest run takes 1-2 minutes even when code hasn’t changed.

Cause: clippy ran with dev profile, nextest uses test profile — separate compilation targets.

Fix: cargo xtask validate runs clippy with --profile test to share artifacts. If running manually, use:

Symptom: canopy-rules::rules_test evaluation_creates_audit_trail fails intermittently.

Cause: Timing-dependent test — publishes an event and checks if canopy-security persisted it. RabbitMQ delivery can be delayed.

Fix: Rerun. If persistent, check RabbitMQ health: docker compose logs rabbitmq.

Symptom: After a git push, a custom seed is gone — the database holds the default 50-household seed and the E2E-run mutations instead.

Cause: The .githooks/pre-push hook runs a bare cargo xtask e2e (which re-seeds internally via xtask::cmd::seed::run) against the same long-lived canopy devstack — not ephemeral testcontainers. Every push therefore re-seeds and mutates the DB. This is unavoidable: the hook always does it, and --no-verify is forbidden.

Implication: Do any manual setup after pushing, or expect to re-seed. The safe order is merge first, re-seed last:

Mitigations (cheapest first): 1. Snapshot/restore around the E2E run using the existing tooling, preserving dev/demo state with near-zero perf hit (see ADR-016):

Note: Docker-in-Docker (DinD) is not recommended locally. State isolation, not daemon isolation, is what’s wanted — and DinD loses the Rust/musl build cache (cold ~10-minute builds every run) and doubles resource pressure across the ~29 services. "Fast DinD" via a mounted docker.sock is just sibling containers on the host daemon, which gives no isolation over a separate compose project.

Symptom: cargo audit reports RUSTSEC advisories.

Cause: Transitive dependencies via typst (document generation). 4 advisories are suppressed in deny.toml: - RUSTSEC-2024-0320 (yaml-rust unmaintained) - RUSTSEC-2025-0141 (bincode unmaintained) - RUSTSEC-2024-0436 (paste unmaintained) - RUSTSEC-2023-0071 (rsa Marvin Attack)

Fix: These are all transitive via typst and have no upstream fix. cargo deny check is the authoritative tool (subsumes cargo audit). The advisories are documented and monitored — when typst releases a fix, remove the ignore entries from deny.toml.

Symptom: Compilation fails with "expected identifier, found keyword `gen`"

Cause: Rust 2024 reserves gen as a keyword. Cannot use as variable name.

Fix: Rename the variable (e.g., gen_for_handler → generator_for_handler).

Symptom: git push exits 141 immediately after the pre-push hook reports success — the validation passed but the branch never lands on the remote.

Cause: The large pre-push E2E output (341 specs) floods the output capture and SIGPIPEs the push after the hook passed but before the transfer completes. The hook’s work is done; only the final transfer is killed.

Fix: Redirect the push output to a file so the capture only receives a tiny line, then re-push if the branch isn’t on the remote yet:

Validation already passed, so a plain re-push lands the branch.

Symptom: Build fails with "error returned from database: relation does not exist"

Cause: sqlx verifies SQL queries at compile time against DATABASE_URL. If the database is down or the migration hasn’t run, verification fails.

Fix: 1. Ensure PostgreSQL is running and the database exists 2. Run the service once to apply migrations: cargo xtask dev start --shared-db 3. Alternatively, use offline mode: SQLX_OFFLINE=true cargo build

Symptom: Service panics with "migrations failed"

Cause: Migration SQL has a syntax error, or a migration was modified after it was already applied.

Fix: - Check the migration file for SQL errors - If a migration was modified: migrations are forward-only. Write a corrective migration, or restore from backup (see Deployment Rollback) - If the database is corrupted: cargo xtask dev restart --shared-db wipes all data and starts fresh (development only)