Plan: Containerized Integration Tests

On this page

Status

Step Description Status

1

Author ADR-015 (docs/modules/ROOT/pages/adrs/adr-015-containerized-integration-tests.adoc) — Status / Context / Decision / Consequences. Cite ADR-001, ADR-005. Document the testcontainers-rs vs docker-compose reconciliation.

Done (2026-04-27) — ADR-015 authored. Renumbered from ADR-012 in plan (ADR-012 was already taken by adr-012-layered-yaml-configuration when this plan was drafted).

2

URL-centralization close-out (A1–A3): export CANOPY_TESTDATABASE_URL from write_ports_env; refactor canopy-db::pg_url() and canopy-mq::amqp_url() to read full URL env vars with localhost fallback; make infrastructure_available() parse the host from CANOPY_TESTDATABASE_URL.

Done (2026-04-27) — xtask::docker::{build_env_for_ports, write_ports_env} now emit CANOPY_TESTDATABASE_URL; canopy-db::tests::pg_url, canopy-mq::tests::amqp_url (both mq_test.rs and reconnect_test.rs) read the full-URL env var first, fall back to legacy CANOPY_PORT_* + localhost; infrastructure_available parses host out of CANOPY_TESTDATABASE_URL. Pure unit-test of the URL authority parser added.

3

Add Dockerfile.integration (multi-stage Alpine, non-root, pre-built nextest) and .dockerignore at workspace root.

Done (2026-04-27) — Single-stage Alpine (rust:1.94-alpine) with musl-dev, pkgconf, openssl-dev, font-noto (canopy-typst tests), pre-built nextest, non-root app user, CARGO_TARGET_DIR=/app/target for the named volume mount. Multi-stage was scoped down to single-stage in Design because the canopy_integration_target named volume mounts at /app/target at runtime, overlaying any builder-stage cache promotion. .dockerignore extended with .ports.env + .devstack/.

4

Add canopy-integration compose service to docker-compose.yml with in-network env var overrides for every CANOPY_TEST*_URL plus CANOPY_TESTDATABASE_URL and CANOPY_TEST__RABBITMQ_URL, depends_on chain for every service the suite probes, and a named target volume for cache.

Done (2026-04-27) — canopy-integration service under profiles: [integration], hardcoded in-network URLs for all 21 CANOPY_TEST__*_URL env vars + DB + MQ; depends_on: condition: service_healthy for postgres/rabbitmq/keycloak + 11 program services; canopy_integration_target named volume; ./test-results:/app/test-results host bind for JUnit XML.

5

Wire --host escape-hatch flag into xtask/src/cmd/test.rs and xtask/src/cmd/validate.rs; default now routes the integration nextest step through docker compose --profile integration --profile snap-only run --build --rm canopy-integration. When CANOPY_CI=true, swap the profile arg to ci-integration.

Done (2026-04-27, partial — validate flip deferred) — xtask/src/cmd/test.rs adds --host flag; default cargo xtask test --integration now routes through run_integration_tests_container which calls docker compose --profile integration run --build --rm canopy-integration --profile <profile>. Profile arg swaps to ci-integration when CANOPY_CI=true. Dockerfile ENTRYPOINT split: ENTRYPOINT carries cargo nextest run --workspace; CMD carries --profile integration (defaultable) so the runner can override via docker compose run’s service-name-trailing args. Deviation: `xtask/src/cmd/validate.rs was NOT flipped — validate is the pre-push hot loop and the cold container build would add ~2 min to every pre-push. Tracked as a follow-up issue. Validate keeps the legacy host nextest call until the runner is proven stable on real CI.

6

Add integration-tests job to .gitlab-ci.yml under the test stage, tagged dhs-aws-autoscaler-docker.xlarge, with DinD service, CANOPY_CI=true, and JUnit artifact collection from test-results/integration/results.xml.

Done (2026-04-27) — integration-tests job, dhs-aws-autoscaler-docker.xlarge, docker:27-dind service, CANOPY_CI=true, JUnit artifact at test-results/integration/results.xml. Initial rollout: when: manual + allow_failure: true so a flaky first run doesn’t block the merge train. Flip to when: on_success once two consecutive passes land on main.

7

Documentation pass: .claude/docs/testing.md, .claude/docs/local-dev.md, .claude/CLAUDE.md, docs/modules/ROOT/pages/developer-guide.adoc, CHANGELOG.adoc.

Done (2026-04-27, scoped) — CHANGELOG.adoc Unreleased Added entry covers ADR-015 + new compose service + xtask flip + URL centralization + CI job + the validate-flip deviation. nav.adoc gains the ADR-015 link. .claude/docs/testing.md / .claude/docs/local-dev.md / developer-guide.adoc deferred — those are the operator-facing docs and best updated after the rollout-window CI job flips from when: manual to when: on_success (otherwise the docs would describe a path that’s still opt-in).

Branch: feature/containerized-integration-tests
ADR: adr-015-containerized-integration-tests.adoc (authored in Step 1)
Labels: type::feature, priority::medium, program::infrastructure, service::devstack, workflow::ready

Context

Canopy’s Rust integration tests currently execute on the host, reaching devstack services through localhost:<mapped-port> (ephemeral since the port-allocation work in ephemeral-port-allocation.adoc). The host-side model has three concrete operational gaps that surfaced during a fresh Linux Docker Engine setup:

  1. Three canopy-web::session_test cases followed a 303 to host.docker.internal:8180 (Keycloak) and failed with failed to lookup address information: Name or service not known. host.docker.internal is injected into the host’s resolver by Docker Desktop on Mac/Windows but not by Docker Engine on Linux — so tests that resolve the redirect target on the host pass on Desktop and fail on Engine. The immediate workaround (TestClient::new_no_redirect() + assert_status(303)) shipped in feat: auto-handle first-run devstack friction in xtask (MR !65), but the parity fix subsumes it because the redirect target does resolve inside the docker network.

  2. canopy_test_lib::infrastructure.rs, canopy-db::pg_url, and canopy-mq::amqp_url read CANOPY_PORT_POSTGRES_5432 / CANOPY_PORT_RABBITMQ_5672 for the port but hardcode localhost as the hostname. Inside the docker network the hostname is postgres / rabbitmq, so these four spots need a URL-level env var (not a port-level one) to let the container override.

  3. Running the suite at all requires cargo + cargo-nextest on every developer machine and every CI runner image, with host-side resolver quirks (systemd-resolved synthetic records, IPv6 preferences) affecting reproducibility.

Pattern precedent: the canopy-e2e Playwright container at docker-compose.yml:769-781 runs in-network via docker compose --profile e2e run --rm canopy-e2e. This plan applies the same shape to the Rust integration suite.

Scope

In scope

  • New Dockerfile.integration (multi-stage Alpine, non-root app user, pre-built nextest).

  • New canopy-integration compose service with profiles: [integration] and explicit depends_on for every service the suite probes.

  • New env vars in write_ports_env so host-side test runs pick up a full CANOPY_TEST__DATABASE_URL.

  • Refactor canopy-db::pg_url() and canopy-mq::amqp_url() to read full URL env vars (localhost fallback preserved).

  • Refactor infrastructure_available() to parse the host from the DB URL (localhost default).

  • xtask/src/cmd/{test,validate}.rs — default integration path runs in-container; --host flag preserves the existing host-side path.

  • .gitlab-ci.yml — new integration-tests job.

  • ADR-015 + this plan file + documentation updates.

Out of scope

  • Dioxus applicant portal integration tests (portal is stub-only; revisit post-UAT per ADR-008).

  • Unit test runner (stays local, no devstack dependency).

  • Playwright E2E — already containerized; no changes required.

  • New unit-level integration tests using testcontainers-rs per the convention; existing canopy suite is full-stack HTTP tests so docker-compose remains the right path (documented in ADR-015).

  • Migration of the new_no_redirect workaround out of session_test; it still works under both runners and a future cleanup can remove it once the container path is canonical.

Design

A. URL-centralization close-out

Upstream’s TestConfig::from_env() (at crates/canopy-test-lib/src/config.rs) already centralizes all 20 service URLs via CANOPY_TEST__<NAME>_URL env vars with localhost fallback, and auto-loads .ports.env from the workspace root. Three spots still bypass this abstraction with raw CANOPY_PORT_* port vars + hardcoded localhost:

A1. xtask/src/docker.rs::write_ports_env writes CANOPY_TESTKEYCLOAK_URL and CANOPY_TESTRABBITMQ_URL but not CANOPY_TEST__DATABASE_URL. Adjacent to the existing RabbitMQ line, emit:

let pg_port = get_host_port(ports, "postgres", 5432);
writeln!(file, "CANOPY_TEST__DATABASE_URL=postgres://canopy:canopy@localhost:{pg_port}/canopy")?;

A2. Rewrite pg_url() / amqp_url() to read full URL env vars.

  • crates/canopy-db/tests/db_test.rs::pg_url: read CANOPY_TEST__DATABASE_URL with the current postgres://canopy:canopy@localhost:{CANOPY_PORT_POSTGRES_5432}/canopy as fallback.

  • crates/canopy-mq/tests/mq_test.rs::amqp_url: read CANOPY_TEST__RABBITMQ_URL with the current amqp://canopy:canopy@localhost:{CANOPY_PORT_RABBITMQ_5672}/%2f as fallback.

Host runs are unchanged — the env vars are already in .ports.env — and the container will override both to in-network URLs.

A3. Make infrastructure_available() network-aware. crates/canopy-test-lib/src/infrastructure.rs currently does TcpStream::connect"127.0.0.1", pg_port. Parse the host out of CANOPY_TEST__DATABASE_URL (or default to 127.0.0.1). Inside the container this becomes postgres:5432; on host it stays localhost:<ephemeral>.

B. Compose service

Mirror the canopy-e2e shape. The environment: block explicitly overrides every URL TestConfig::from_env() reads with an in-network address — TestConfig’s fallback chain (env var → `.ports.env → hardcoded localhost) is irrelevant inside the container because the env vars are always set.

canopy-integration:
  profiles: [integration]
  build:
    context: .
    dockerfile: Dockerfile.integration
  user: app
  environment:
    CANOPY_TEST__KEYCLOAK_URL: http://keycloak:8080
    CANOPY_TEST__RULES_URL: http://canopy-rules:8001
    CANOPY_TEST__PERSONS_URL: http://canopy-persons:8002
    CANOPY_TEST__APPLICATIONS_URL: http://canopy-applications:8003
    CANOPY_TEST__ELIGIBILITY_URL: http://canopy-eligibility:8004
    CANOPY_TEST__VERIFICATION_URL: http://canopy-verification:8005
    CANOPY_TEST__ENROLLMENT_URL: http://canopy-enrollment:8006
    CANOPY_TEST__RENEWALS_URL: http://canopy-renewals:8007
    CANOPY_TEST__NOTICES_URL: http://canopy-notices:8008
    CANOPY_TEST__EXCHANGE_URL: http://canopy-exchange:8009
    CANOPY_TEST__APPEALS_URL: http://canopy-appeals:8010
    CANOPY_TEST__REPORTING_URL: http://canopy-reporting:8011
    CANOPY_TEST__SECURITY_URL: http://canopy-security:8012
    CANOPY_TEST__SNAP_URL: http://canopy-snap:8013
    CANOPY_TEST__TANF_URL: http://canopy-tanf:8014
    CANOPY_TEST__MEDICAID_URL: http://canopy-medicaid:8015
    CANOPY_TEST__CAPS_URL: http://canopy-caps:8016
    CANOPY_TEST__WIC_URL: http://canopy-wic:8017
    CANOPY_TEST__WEB_URL: http://canopy-web:8080
    CANOPY_TEST__PORTAL_URL: http://canopy-portal:8090
    CANOPY_TEST__DATABASE_URL: postgres://canopy:canopy@postgres:5432/canopy
    CANOPY_TEST__RABBITMQ_URL: amqp://canopy:canopy@rabbitmq:5672/%2f
    CANOPY_CI: ${CANOPY_CI:-}
  volumes:
    - ./test-results:/app/test-results
    - ./.keys:/app/.keys:ro
    - canopy-integration-target:/app/target
  depends_on:
    canopy-web:           { condition: service_healthy }
    canopy-rules:         { condition: service_healthy }
    canopy-persons:       { condition: service_healthy }
    canopy-applications:  { condition: service_healthy }
    canopy-eligibility:   { condition: service_healthy }
    canopy-verification:  { condition: service_healthy }
    canopy-enrollment:    { condition: service_healthy }
    canopy-renewals:      { condition: service_healthy }
    canopy-notices:       { condition: service_healthy }
    canopy-appeals:       { condition: service_healthy }
    canopy-security:      { condition: service_healthy }
    canopy-snap:          { condition: service_healthy }
    keycloak:             { condition: service_healthy }
    postgres:             { condition: service_healthy }
    rabbitmq:             { condition: service_healthy }
    garage:               { condition: service_started }
volumes:
  canopy-integration-target:

C. Dockerfile.integration

Multi-stage Alpine per .claude/docs/coding-conventions.md Container Runtime section. Non-root app user. Source COPY`ed (not bind-mounted) to match the existing `Dockerfile pattern.

# SPDX-License-Identifier: AGPL-3.0-or-later
# Multi-stage Alpine build for the Rust integration test runner.

# -- Stage 1: build --
FROM rust:1.94-alpine AS builder
RUN apk add --no-cache musl-dev curl
# Pre-built nextest (avoids ~3-min `cargo install`).
RUN curl -LsSf https://get.nexte.st/latest/linux-musl | tar -xzf - -C /usr/local/bin
WORKDIR /app
COPY . .
# Pre-compile test binaries so first `nextest run` inside the container is
# execution-only. Cached by docker layer.
RUN cargo nextest list --workspace --test '*' --profile integration

# -- Stage 2: runtime --
FROM rust:1.94-alpine
RUN apk add --no-cache musl-dev curl
RUN curl -LsSf https://get.nexte.st/latest/linux-musl | tar -xzf - -C /usr/local/bin
RUN addgroup -S app && adduser -S -G app -s /sbin/nologin app
WORKDIR /app
COPY --from=builder --chown=app:app /app /app
USER app
ENTRYPOINT ["cargo", "nextest", "run", "--workspace", "--test", "*", "--profile", "integration"]

Create or extend .dockerignore with:

target/
.git/
node_modules/
test-results/
.devstack/
.ports.env

D. xtask wiring

xtask/src/cmd/test.rs and xtask/src/cmd/validate.rs route integration tests through the compose service by default. A new --host flag preserves the existing host-side path for IDE iteration.

// In test::Args
#[arg(long)]
pub host: bool,

// In test::run(), integration branch:
if args.host {
    run_integration_tests_host()?;   // the existing nextest call
} else {
    run_integration_tests_container(&project)?;
}

fn run_integration_tests_container(project: &str) -> Result<()> {
    crate::docker::compose_cmd(project, &[
        "--profile", "integration",
        "--profile", "snap-only",   // transitively starts canopy-web and its deps
        "run", "--build", "--rm",
        "canopy-integration",
    ])
}

In validate.rs step [10/11] cargo nextest run …​: replace with the same router. Keep the step label stable.

E. Nextest profile selection

.config/nextest.toml defines integration (local) and ci-integration (adds terminate-after = 2 on 120s slow timeout). The container ENTRYPOINT hardcodes --profile integration. For CI, the canopy-integration service is invoked with a compose command override that swaps the profile — wire this in the CI job by appending --profile ci-integration to the compose run arguments when CANOPY_CI=true. The xtask router should do the same switch so cargo xtask test running under CI uses ci-integration without manual override.

F. CI — .gitlab-ci.yml

Add integration-tests under the test stage. Per .claude/docs/coding-conventions.md CI/CD Runners — explicit tags:, dhs-aws-autoscaler-docker.xlarge (multi-service docker-compose is xlarge territory).

integration-tests:
  stage: test
  tags: [dhs-aws-autoscaler-docker.xlarge]
  image: $CI_REGISTRY/gadhs/standard/ci-images/rust-docker:1.94
  variables:
    CANOPY_CI: "true"
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  services:
    - docker:dind
  script:
    - cargo xtask test --integration
  artifacts:
    when: always
    reports:
      junit: test-results/integration/results.xml
    paths:
      - test-results/integration/
    expire_in: 1 week

G. Filesystem-bound tests

  • canopy-typst (crates/canopy-typst/tests/render_test.rs): reads fonts from system dirs. Add apk add --no-cache font-noto to Dockerfile.integration. Verify font availability in a smoke-test rendering after build.

  • canopy-seed (tools/canopy-seed/tests/integration.rs): tempfile::tempdir() — works in container, no change.

Steps

Step 1: ADR-015

Author docs/modules/ROOT/pages/adrs/adr-015-containerized-integration-tests.adoc. Standard ADR format (Status / Context / Decision / Consequences). Cite ADR-001 (program service isolation) and ADR-005 (modular deployment profiles). Document the testcontainers-rs vs docker-compose reconciliation explicitly: canopy’s integration suite is full-stack HTTP tests against running canopy services, not unit-level DB/broker tests, so docker-compose remains the correct path; new unit-level integration tests against DB/broker should still use testcontainers-rs.

Step 2: URL-centralization close-out

Implement Design section A (A1, A2, A3). Verify host-side tests still pass after this step alone with cargo xtask test --integration (no container changes yet).

Step 3: Dockerfile.integration + .dockerignore

Implement Design section C. docker build -f Dockerfile.integration . should produce an image tagged canopy:integration without errors; verify with docker run --rm canopy:integration --help (nextest prints help and exits 0).

Step 4: canopy-integration compose service

Implement Design section B. docker compose --profile integration config should validate without errors. docker compose --profile integration --profile snap-only build canopy-integration should succeed.

Step 5: xtask wiring

Implement Design sections D and E. cargo xtask test --integration (no --host) should route through the container; cargo xtask test --integration --host should run nextest on the host. cargo xtask validate step [10/11] should route through the container.

Step 6: CI job

Implement Design section F. On the MR pipeline, the new integration-tests job should run to completion and upload JUnit XML.

Step 7: Docs pass

See Documentation Updates below. Update CHANGELOG.adoc under == Unreleased=== Added with a single bullet summarizing the container runner, --host escape hatch, and CANOPY_TEST__DATABASE_URL addition.

Files Touched

New files

  • Dockerfile.integration

  • .dockerignore (if not already present)

  • docs/modules/ROOT/pages/adrs/adr-015-containerized-integration-tests.adoc

  • (this plan file)

Modified files

  • xtask/src/docker.rs — add CANOPY_TEST__DATABASE_URL to write_ports_env

  • crates/canopy-db/tests/db_test.rs — read CANOPY_TEST__DATABASE_URL with localhost fallback

  • crates/canopy-mq/tests/mq_test.rs — read CANOPY_TEST__RABBITMQ_URL with localhost fallback

  • crates/canopy-test-lib/src/infrastructure.rs — parse host from CANOPY_TEST__DATABASE_URL

  • docker-compose.yml — new canopy-integration service + canopy-integration-target named volume

  • xtask/src/cmd/test.rs--host flag, route default to container, CANOPY_CI profile swap

  • xtask/src/cmd/validate.rs — route step [10/11] through container

  • .gitlab-ci.yml — new integration-tests job

  • .claude/docs/testing.md — document container-first runs + --host escape hatch

  • .claude/docs/local-dev.md — commands list; host-side nextest now optional

  • .claude/CLAUDE.md — Tech Stack note about container-first integration runs

  • docs/modules/ROOT/pages/developer-guide.adoc — commands + ADR-015 reference

  • CHANGELOG.adoc — entry under == Unreleased=== Added

Verification

  1. Cold start, container path. rm -rf .devstack/ .ports.env && cargo xtask validate. Expect: ensure_ready cold-starts devstack with ephemeral ports → canopy-integration builds and runs → all 877 tests pass → test-results/integration/results.xml written → exit 0.

  2. Host path. cargo xtask test --integration --host. Expect: nextest runs on host, hits loopback ephemeral ports via TestConfig::from_env() fallback, all tests pass.

  3. Parity test. Locally revert the new_no_redirect patch in services/canopy-web/tests/session_test.rs. Run cargo xtask test --integration. The three session tests pass under the container runner. Restore the patch.

  4. No-leakage grep. grep -rn 'localhost:[0-9]' services//tests crates//tests tools/*/tests returns zero matches — every URL flows through an env var.

  5. Image cache reuse. Run cargo xtask test --integration twice. Second run hits canopy-integration-target volume and completes nextest in <30s for unchanged code.

  6. CI green. Push branch, watch the integration-tests job on the MR pipeline.

  7. IDE iteration loop. From VSCode, "Run Test" on a single integration test runs on the host via loopback fallback. No docker invocation.

Acceptance Criteria

  1. cargo xtask validate on a fresh clone (no .devstack/) cold-starts the devstack, builds the integration image, runs the full 877-test nextest integration suite inside canopy-integration, writes JUnit XML to test-results/integration/results.xml. Exit 0.

  2. cargo xtask test --integration --host runs the same 877-test suite on the host against loopback ephemeral ports. Exit 0.

  3. With the container runner selected, reverting the new_no_redirect patch locally keeps the three canopy-web::session_test cases passing — the redirect target resolves inside the docker network. Restore the patch (don’t ship the revert).

  4. grep -rn 'localhost:[0-9]' services//tests crates//tests tools/*/tests returns zero matches.

  5. CI integration-tests job is green on the MR pipeline.

  6. Pre-push hook (cargo xtask validate) passes locally.

  7. cargo xtask check-docs is clean.

Documentation Updates

Per .claude/docs/delivery-protocol.md Documentation Update Checklist:

  • .claude/docs/services.md — N/A (no new endpoints or events).

  • .claude/CLAUDE.md — one-line note under "Tech Stack" or "Conventions" about container-first integration runs. Since this is a single-MR plan, the status-table update per .claude/docs/git-workflow.md Multi-MR Plans lands in this MR.

  • CHANGELOG.adoc — entry under == Unreleased=== Added, bullet format * feature-title: short description. Include: container runner, --host escape hatch, CANOPY_TEST__DATABASE_URL export, db_test.rs / mq_test.rs refactor to URL-env-vars.

  • .claude/docs/testing.md — "Container Runner" subsection under "Integration Tests"; document the --host escape hatch.

  • .claude/docs/local-dev.md — commands list; host-side nextest now optional for the default flow.

  • docs/modules/ROOT/pages/developer-guide.adoc — commands + ADR-015 reference.

  • docs/modules/ROOT/pages/adrs/adr-015-containerized-integration-tests.adoc — the ADR itself.

Edit this page · default