ADR-040: Build-Once, Gate-Complete Artifact Promotion

On this page

Status

Accepted (2026-07-14)

Realized by #1007 (subsumes #1025). Chronic gate failures that currently keep promotion parked are tracked separately: #1067 (cargo-test runner disk), #1068 (integration-tests image access).

Context

The production promotion pipeline had accreted five structural defects (#1007, verified against the live pipeline + registry on 2026-07-14):

  1. Tag pipelines never existed. workflow.rules matched only MRs and branches, so the tag-only docker-promote and sbom rules were dead code — a pushed release tag created no pipeline at all.

  2. Promotion was gate-incomplete. docker-promote declared an explicit needs: list of six jobs and ignored every other blocking test-stage job — the ADR-011/ADR-031 policy audits, typed-id-path-audit (#1025), quality-budgets, cargo-doctest, integration-tests, the GitLab security scanners, cargo-deny, cargo-machete among them. Promotion could become runnable, and push latest, while a blocking gate was still running or failing.

  3. The artifact-input map was untested and incomplete. Promotion triggered only on /*.rs, /Cargo.toml, Cargo.lock, Dockerfile — but the images also consume rulesets/, migrations and static assets under services/, tools/, the portal Dockerfile, and .dockerignore. A ruleset-only or migration-only merge left the registry artifact silently stale.

  4. Promotion rebuilt from source. The promote job ran docker build and pushed the result — the published image was not necessarily the artifact any gate had seen, and latest moved in the same breath with no ordering guard.

  5. The applicant portal had no production artifact. The root Dockerfile excludes canopy-portal by design (the Dioxus dx pipeline builds the WASM client + server together, ADR-008), and no CI job built services/canopy-portal/Dockerfile at all.

The observable consequence: the registry’s latest dated to 2026-03-30 — three and a half months of merged main never reached the registry — and no ADR covered image promotion, registry tagging, or container supply-chain.

Decision

  1. Tag pipelines exist. workflow.rules gains - if: $CI_COMMIT_TAG; the gates whose rules were branch/MR-scoped (cargo-audit, cargo-machete, secrets-yaml-lint) run unconditionally on tags, so a tag pipeline is as gate-complete as main.

  2. Two deployable images, built once, under immutable staging refs. build-service-image (root Dockerfile) and build-portal-image (services/canopy-portal/Dockerfile) run in the test stage on every artifact-affecting main/tag commit and push content-addressed staging refs $CI_REGISTRY_IMAGE/build:$CI_COMMIT_SHA and $CI_REGISTRY_IMAGE/build/portal:$CI_COMMIT_SHA (plus mutable main-cache layer-cache pointers). A pipeline whose commit already has a staging ref reuses it (build-once across pipelines: a release tag on a main-built commit promotes the exact digest main tested). Staging refs are pipeline-internal; the production repositories are $CI_REGISTRY_IMAGE and $CI_REGISTRY_IMAGE/portal.

  3. Promotion is a complete barrier + a registry-side retag. docker-promote declares no needs: — a promote-stage job without needs waits for the entire test stage, so every blocking gate (including both image builds) gates every production-registry mutation; advisory (allow_failure) and manual jobs do not block, which is their documented contract. Promotion resolves the staging digests and retags them with docker buildx imagetools create — never docker build. Every promoted commit gets an immutable :<short-sha> production ref; tag pipelines additionally get :<tag>.

  4. latest is guarded and serialized. The promote job moves latest only when $CI_COMMIT_SHA equals the current refs/heads/main (checked via git ls-remote at promote time), so an older pipeline finishing late can never move latest backwards; resource_group: registry-promote serializes concurrent promotions.

  5. Every production digest has a retained SBOM. Each build job generates a CycloneDX SBOM from its pushed ref with a syft that is version-pinned (SYFT_VERSION) and checksum-verified (SYFT_SHA256, sha256sum -c before the binary runs in a job holding registry push credentials), retained as a never-expiring artifact; the tags-only sbom job keeps producing the source-level cargo SBOM for releases.

  6. The invariants are statically gated. cargo xtask ci-config-lint — a CI job and a pre-push validate static gate (#896 subsumption) — parses both Dockerfiles' COPY/ADD sources (any case) and fails if the YAML-anchored artifact-input map misses any build input, and asserts: the tag workflow rule exists; docker-promote sits in stage: promote with no needs:, never invokes docker build/docker buildx build, and carries a resource_group; every :latest reference in the file sits inside the promote job after the main-head guard line; build jobs and promote name the same immutable $CI_COMMIT_SHA staging-ref strings (a one-sided rename cannot land); the build-job base and promote both reference the *artifact-inputs anchor (trigger paths cannot drift); and the syft pin + checksum are present. Each assertion has a negative test, and the gate’s own test suite runs it against the real repo config.

Consequences

  • Promotion latency equals the slowest blocking gate (integration tests) — the price of gate-completeness. Force-merging past red CI still works for merges (the local battery is the merge gate, see Contributor Workflow Conventions); it no longer lets an untested image reach the registry.

  • A chronically red blocking gate now visibly parks promotion instead of being silently skipped past — as of acceptance, #1067 and #1068 do exactly that and must be fixed for the first promoted image under this ADR.

  • MR pipelines do not build images; a Dockerfile-breaking change surfaces on the main pipeline. Accepted: image builds are ~20–30 min each, and main breakage is visible + cheap to revert pre-1.0.

  • Build-once extends to test consumption (#1073, 2026-07-15): the CI integration stack pulls the staging refs instead of compiling the workspace in-daemon, so integration tests exercise the exact digests promotion retags — and integration-tests consequently runs only where the staging refs exist (tag pipelines, and main commits touching the artifact-input map), mirroring the build jobs' rules. See the ADR-015 amendment for the mechanism.

  • Image signing/attestation (cosign, SLSA provenance) is out of scope here and remains open; the immutable digest + retained SBOM chain is the foundation it would build on.

  • Deploy tooling can now rely on: :<short-sha> (immutable, every promoted main commit), :<tag> (releases), :latest (the newest gate-complete main head), for both the service image and …/portal.

References

  • #1007 (defect + acceptance criteria), #1025 (subsumed), #1066/#1067/#1068 (chronic CI failures found during delivery)

  • ADR-015 (the DinD integration job that now gates promotion)

  • ADR-017 (CI secrets)

  • ADR-033 §5 (the test-clock feature must never reach a production image — the build jobs pass no CARGO_FEATURES, keeping the Dockerfile’s empty default)

  • ADR-008 (why the portal is a separate image)

Edit this page · default