Local Development

On this page
NOTE

Full setup steps, devstack/container management, the staleness guard, seed data, the port map, and CI-vs-local differences live in the Developer Guide. This page is a bounded bootstrap quick-reference — keep it short.

Host dependencies

  • Rust toolchain pinned by rust-toolchain.toml (channel 1.96 + the wasm32-unknown-unknown target for the canopy-portal Dioxus client).

  • Docker Engine + the Docker Compose v2 plugin (daemon running, your user in the docker group). Stock-distro compose (2.40.x, e.g. Ubuntu’s docker-compose-v2) is supported: xtask activates dependency profiles explicitly on compose run (#1349), the documented-correct invocation on every compose version.

  • glab CLI (used by `cargo xtask validate’s public-visibility check).

  • cargo install cargo-nextest --locked (required by test / validate).

  • cargo install cargo-deny --locked (required by validate; warn-only if missing).

  • Not needed on the host: Node.js / Playwright (run inside the canopy-e2e container) and psql (the seed loads via docker exec).

Bootstrap sequence

git config core.hooksPath .githooks          # activate the vendored git hooks
git config commit.gpgsign true               # commit signing is required
cargo xtask secrets init                      # one-time: generate dev age keypair, add to .sops.yaml
cargo xtask dev start                         # start devstack (sops decrypt + inject happens here)
cargo xtask seed --seed 42                    # deterministic test data (9 households)
cargo xtask test                              # full battery: fmt + clippy + nextest
cargo xtask validate                          # pre-push gate: + signing/docs/SPDX + docker build

Fresh-machine provisioning checklist (#1358)

Tribal-knowledge steps the Bootstrap sequence assumes, in the order that works (each cost real diagnosis time on the 2026-08-07 fresh-box benchmark):

  1. Toolchain + docker first (Host dependencies above), then everything below, then cargo xtask dev start (writes the secret floor), then the battery.

  2. SOPS age key at ~/.config/sops/age/keys.txt BEFORE the first battery. cargo xtask secrets init generates a fresh recipient — but decrypting the existing secrets/dev.yaml needs an EXISTING recipient’s key (copy it from a current dev machine, or have a current machine run secrets add-recipient for your new key first).

  3. .keys/ signing keys: copy the directory from an existing dev machine. Regenerating instead requires re-aligning with secrets/dev.yaml — a partial or independently generated set fails 3 envelope-JWS tests with no hint that keys are the cause.

  4. /etc/hosts (optional since #1609): 127.0.0.1 host.docker.internal. canopy-test-lib now falls back to 127.0.0.1 itself when the container-canonical hostname does not resolve (one tracing warning per process names the fallback), so host-side tests work without the line. Adding it is still fine — it pins the canonical name and silences the warning.

  5. Firewalled hosts (ufw active) + Docker ≥ 28: container→host-published-port hairpin traffic lands on the INPUT chain (pre-28 NAT bypassed the firewall, so long-lived boxes never noticed). Allow the docker address pool: sudo ufw allow from 172.16.0.0/12.

  6. Version matrix: stock Ubuntu 26 ships engine 29 + compose 2.40 — both supported (dependency profiles are activated explicitly on compose run, #1349; see Host dependencies). Newer compose 5.x behaves identically for the documented invocations.

The clamav sidecar (ADR-042, #1006)

The full profile includes the digest-pinned clamd sidecar (devstack/clamav/): the image ships a signature-DB snapshot (fast first boot, ~30-60s to healthy — plus a bounded freshclam pre-pass on every boot since #1522, usually seconds, so on any boot where it completes clamd cannot load definitions a concurrent update is about to replace), the canopy_clamav_db volume persists freshclam’s incremental updates, and canopy-applications scans uploads through it asynchronously. Nothing depends_on it — uploads quarantine pending while it’s down. After editing devstack/clamav/*, docker volume rm canopy_clamav_db re-initializes the baked files.

Devstack-aggressive worker cadences (#1367)

Fixed worker/poll intervals set the integration and e2e wall-time floor (the nextest stage historically idled at 20-27% CPU waiting on them). Every such interval is an env-tunable knob whose production default is byte-identical to the historical constant (pinned by config tests); docker-compose.yml sets the aggressive values for the devstack only — grep it for #1367 to see the full set (outbox drainer 250 ms → 25 ms on all MQ services; notices worker/dispatcher 2 s → 200 ms; appeals CB assessment 5 s → 500 ms; enrollment issuance settlement 5 s → 500 ms; eligibility bulk tick 1 s → 100 ms; applications scan poll 5 s → 1 s; security chain/archive ticks and first-tick delays; reporting run tick, which predates this sweep). Knob domains and boot posture (error, never clamp): Configuration Reference. Do NOT compensate a slow flow by widening a test tolerance — tune the cadence.

System tuning for battery-heavy boxes (#1390)

Battery workloads (image rebuilds, ~5k DB tests, checkpoint bursts) conflict with aggressive desktop-responsiveness defaults — the #1341 five-way diagnosis is the evidence base (wall-clock freezes mid-battery; validated fixed by absence across ~24 batteries, maintainer-ratified 2026-08-10). Check these only if your distro tunes for desktop — a stock Ubuntu box has none of these pathologies (the fresh benchmark box never exhibited #1341).

Knob Settled value Why / applicability

vm.swappiness

60

CachyOS-family ships 100–150 — battery memory pressure then swaps hot pages. Stock Ubuntu is already 60.

MGLRU min_ttl_ms

1000

Default 100 lets the multi-gen LRU evict a working set the battery re-touches within the second.

vm.dirty_bytes / vm.dirty_background_bytes

2147483648 / 536870912

On big-RAM boxes the default ratio yields a ~256 MB global ceiling that conscripts every writer into D-state during checkpoint bursts; ~2 GB/512 MB decouples them.

ananicy-cpp rules for node / headless_shell

nice 0, sched other, ioclass none

CachyOS-family only: the default BG_CPUIO demotion SCHED_IDLEs the whole Playwright tree mid-e2e.

Persistence, as deployed on the reference box (bash- and fish-neutral files): /etc/sysctl.d/99-canopy-workstation.conf (the three vm.* lines), /etc/tmpfiles.d/mglru-min-ttl.conf (w! /sys/kernel/mm/lru_gen/min_ttl_ms - - - - 1000), and /etc/ananicy.d/99-local.rules (one JSON rule per binary name). Verification metrics from the #1341 window: chromium VmSwap ≈ 0 on green runs, no N-flags, no mid-e2e checkpoint storms.

CAUTION
the sysctl.d line is NOT sufficient for swappiness on CachyOS-family boxes — /usr/lib/udev/rules.d/30-zram.rules re-asserts vm.swappiness=150 when zram0 initializes, which fires AFTER systemd-sysctl on every boot (the reference box was found silently back at 150 on 2026-08-26, after a cachyos-settings update + reboot post-dated the #1341 window). Override by copying the rule to /etc/udev/rules.d/30-zram.rules (same filename wins) with the SYSCTL clause set to 60 — and re-verify sysctl vm.swappiness after any cachyos-settings package update.

Standing devstack requirements for pushing (#1386)

The pre-push battery’s E2E stage (a bare cargo xtask e2e) requires the devstack to be on the full compose profile (all program services) with the test-clock feature build — the journey lane is a blocking battery lane and needs both. Set up once (both are sticky across refreshes):

CANOPY_CARGO_FEATURES=canopy-api/test-clock cargo xtask dev start --profile full

A snap-only or production-shaped stack fails the push loudly (test-clock is probed before the suite; a silently-skipped journey lane fails afterwards). --profile snap-only remains fine for targeted iteration — just not for the battery.

Most-used devstack commands

  • cargo xtask dev start --profile snap-only --shared-db — the SNAP UAT subset on a single postgres (targeted iteration only — the pre-push battery needs --profile full + test-clock, above).

  • cargo xtask dev refresh — auto-detect changes, minimum rebuild; dev status shows staleness.

    • Source-change rebuilds are incremental: the in-image workspace build rides BuildKit cache mounts (#1365), recompiling only what changed instead of the whole workspace. The wall floor is now the portal image’s deliberately-cold dx build (#732 — correctness over warm speed) plus restarts. The builder cache grows over time — pruning is owned by the #1189 policy, and docker builder prune resets to a from-zero (but still correct) build.

  • cargo xtask dev reimport-realm — required after any edit to devstack/keycloak/canopy-realm.json (fixture users, roles, primary_programs attributes, protocol mappers). Keycloak’s --import-realm imports only when the realm is absent, so dev refresh and dev reload leave an edited realm file entirely unapplied — the stack comes up looking healthy while serving the old claims. The command force-recreates only the stateless keycloak container (fresh H2 ⇒ re-import; no data volume is touched) and clears tests/e2e/auth/*.json, whose cached tokens would otherwise still carry the pre-edit claims. Re-run cargo xtask e2e to re-authenticate the fixtures.

    • Since ADR-044 a worker fixture with no primary_programs attribute cannot sign in at all, so a stale realm shows up as an admission-rejection banner on the sign-in page rather than as a subtle scope difference. jane.unscoped is that state on purpose (the rejection-path E2E fixture); every other worker fixture carries an explicit claim.

  • cargo xtask dev logs [service] — follow logs; dev clean --confirm wipes all volumes.

  • cargo xtask target-report — target-dir retention diagnostic (#1384): per-subtree sizes, the debug/deps link surface (executable count/bytes — the workspace links ~250 test binaries and hash-renamed generations accrete; 200+ GiB observed), and 30/90-day staleness buckets. Reclaim with the explicit --prune <subtree> (whole separable subtrees only — llvm-cov-target, doc, doc-check, tmp, nextest, or a full debug/release profile as a deliberate cold-build trade). There is deliberately no auto-clean: an automatic clean converts every next run into a cold build.

  • Never run docker compose directly — always go through cargo xtask dev (raw compose breaks restart ordering / JWKS state). See Developer Guide for the rationale.

Edit this page · default