ADR-015: Containerized Integration Tests

On this page

Context

Canopy’s Rust integration test suite (~1500+ tests across 14 services + shared crates) currently executes on the developer’s host, reaching devstack services through localhost:<ephemeral-port> mappings written to .ports.env by cargo xtask dev start. The host-execution model produces three operational gaps that surface inconsistently across developer environments and CI runners:

  1. Linux Docker Engine vs Docker Desktop divergence. Three canopy-web::session_test cases follow a 303 redirect to host.docker.internal:8180 (Keycloak). host.docker.internal is auto-injected into the host’s resolver by Docker Desktop on Mac/Windows but not by Docker Engine on Linux. Tests that resolve the redirect target on the host pass on Desktop and fail on Engine with failed to lookup address information: Name or service not known. The immediate workaround (TestClient::new_no_redirect() + assert_status(303) shipped in MR !65) sidesteps the resolution but blocks any redirect-following coverage.

  2. Hardcoded localhost in URL builders. canopy_test_lib::infrastructure::infrastructure_available(), canopy-db::pg_url(), and canopy-mq::amqp_url() read CANOPY_PORT_POSTGRES_5432 / CANOPY_PORT_RABBITMQ_5672 for the port but hardcode localhost as the hostname. Inside the docker network the canonical hostname is postgres / rabbitmq — there is no env hook that lets a containerized test runner override the host without rewriting these call sites.

  3. Host-side toolchain reproducibility. Running the suite at all requires cargo + cargo-nextest on every developer machine and every CI runner image, with host-side resolver quirks (systemd-resolved synthetic records, IPv6 preferences, /etc/hosts overrides) affecting reproducibility. ADR-001 program-service-isolation already mandates that program services see each other only by compose service name; the integration runner should sit on the same network for the same parity guarantee.

The pattern precedent already exists in the repo: docker-compose.yml defines canopy-e2e (Playwright) under profiles: [e2e], run via docker compose --profile e2e run --rm canopy-e2e. The Playwright suite has had zero environment-divergence failures since it shipped because every test runs against in-network DNS names.

A counter-pattern also exists: the testcontainers-rs crate is used inside unit-level tests for crates/canopy-db (PostgreSQL) and crates/canopy-mq (RabbitMQ) to spin per-test ephemeral containers. Those tests test individual crate behaviour against an isolated dependency. The full-stack HTTP integration tests in services/canopy-/tests/.rs and crates/canopy-test-lib-using crates are different in kind: they exercise the running canopy service mesh, not isolated infrastructure.

Decision

Run the Rust integration test suite in the docker network, against compose-service-name targets, by default — same as canopy-e2e.

Specifically:

  1. Add a Dockerfile.integration at the workspace root: multi-stage Alpine, non-root app user, pre-built nextest, source COPY`ed in. ENTRYPOINT runs `cargo nextest run --workspace --test '*' --profile integration. Per coding-conventions Container Runtime section.

  2. Add a canopy-integration service to docker-compose.yml under profiles: [integration]. Environment block hardcodes every CANOPY_TEST__*_URL to in-network address (http://canopy-rules:8001, …, postgres://canopy:canopy@postgres:5432/canopy, amqp://canopy:canopy@rabbitmq:5672/%2f). depends_on lists every service the suite probes with condition: service_healthy. Cache volume canopy-integration-target.

  3. Refactor three URL builders (pg_url, amqp_url, infrastructure_available) to read full URL env vars (CANOPY_TESTDATABASE_URL, CANOPY_TESTRABBITMQ_URL) with the existing localhost:<port> fallback. xtask::docker::write_ports_env emits the new vars for host-side runs (parity).

  4. xtask/src/cmd/{test,validate}.rs route the integration nextest step through docker compose --profile integration run --build --rm canopy-integration by default. A new --host flag preserves the existing host-side path for IDE iteration or single-test debugging.

  5. .gitlab-ci.yml adds integration-tests under the test stage, tagged dhs-aws-autoscaler-docker.xlarge, with DinD service, CANOPY_CI=true, JUnit artifact collection.

  6. testcontainers-rs continues to be the right choice for unit-level integration tests against isolated DB / broker dependencies (e.g. canopy-db, canopy-mq internals). Full-stack HTTP suites against the canopy mesh use docker-compose. ADR-015 documents this split so future contributors don’t conflate the two patterns.

Cross-cutting decisions

  • No shared target/ between host and container. Cache lives in the named volume canopy-integration-target to keep host-vs-musl artefacts segregated. First container build is slow; subsequent runs hit the cache.

  • Filesystem-bound tests. canopy-typst reads system fonts → Dockerfile.integration adds apk add --no-cache font-noto. canopy-seed uses tempfile::tempdir(), works in-container unchanged.

  • .dockerignore at workspace root excludes target/, .git/, node_modules/, test-results/, .devstack/, .ports.env. Keeps the build context lean.

Consequences

Positive

  • Production parity. Tests reach services by compose service name — the same DNS path program services use to reach each other (ADR-001). Host-vs-container resolution divergence stops being a class of failure.

  • No host toolchain bootstrap. Developer machines and CI runners need only Docker. cargo + cargo-nextest move into the test container’s build stage.

  • Linux/Mac/Windows parity. host.docker.internal resolution differences disappear because tests no longer touch the host DNS.

  • Aligns with canopy-e2e. Same shape — --profile X --rm, env-var URL overrides, depends-on health gates. Reduces compose surface novelty.

  • No public-API changes. The TestClient API, the TestConfig::from_env() schema, and the per-test code shape stay identical. Internal env-var sourcing is the only thing that changes.

Negative

  • Slower first run. cargo nextest list --workspace inside the container takes 3-5 minutes uncached; the named cache volume amortises subsequent runs, but a --build after a Cargo.toml change pays the cost again. CI runners get a fresh image per pipeline.

  • --host escape hatch. IDE-driven iteration (run-one-test, attach debugger) wants the host runner. The --host flag preserves it but adds a code path divergence the fix needs to cover. Documented in the plan’s Verification section.

  • DinD in CI. GitLab CI’s existing pipeline uses container jobs without DinD. Adding services: [docker:dind] is supported but increases per-job cost and image-pull time. The trade is paid once per pipeline.

  • Re-auditing host.docker.internal references. Any leftover host-only assumptions surface only after the cutover. Risk mitigated by leaving --host as an opt-back path during the migration window.

Neutral

  • Out-of-network access. Tests that need internet (rare; the mock IEVS / SAVE adapters are local) work fine — Docker’s default bridge gives the container outbound DNS.

  • Test result collection. JUnit XML is written to a host-mounted volume (./test-results/); CI reads it from the same path.

Alternatives Considered

  1. Inject host.docker.internal via extra_hosts in compose. Works on Mac/Windows but Linux Docker Engine’s host-gateway substitution is opt-in per-service, fragile across Docker versions, and doesn’t address the toolchain-bootstrap or hostname-hardcoding problems. Rejected.

  2. Migrate the full integration suite to testcontainers-rs. Per-test ephemeral DB/broker plus dynamically-spawned canopy-snap etc. would replace docker-compose entirely. Disproportionate effort for a suite that’s already aligned with compose semantics — every program service has its own health-gated lifecycle that compose orchestrates. Reserved for unit-level integration tests where per-test isolation is the value.

  3. Keep host-only with documentation. "Run on Docker Desktop only" is not a viable stance — Linux is the canonical CI environment. Rejected.

Amendment — CI image sourcing (2026-07-15, #1073)

The in-network model is unchanged, but CI no longer compiles the devstack inside its docker daemon: the from-source build (service build stage
dioxus portal toolchain, in parallel under dind) exceeded every runner disk. The integration-tests job (now tagged 2xlarge, main + tag pipelines only) sets COMPOSE_FILE=docker-compose.yml:docker-compose.prebuilt.yml + CANOPY_PREBUILT_IMAGES=true, which maps every service’s build: block to the immutable commit-SHA staging refs the pipeline’s build jobs already pushed (ADR-040 build-once, extended to test consumption) — so the integration suite exercises the exact digests docker-promote later retags. xtask (devstack_guard) pulls instead of --build under that flag. Only the test-runner image (Dockerfile.integration, which compiles nothing at build time) still builds in-job; local development keeps building from source.

Related ADRs

Edit this page · default