Plan: test-results/validate-report.json — a trustworthy, self-describing validate report (#1253)

On this page
NOTE

Honest scope: validate only. This is not a whole-git push report — auth (pre-hook), commit signature, hook-level gates (cargo deny / check-docs / cargo doc / perf / LFS) and xtask-compile are outside cargo xtask validate and are tracked in #1254 (hook-owned manifest, with the template-owned half escalated upstream to claude-quickstart). git push stays git push; .env.local export (auth) and git ls-remote (remote landed) remain process facts this report cannot and does not replace. The in-process-gate full-capture follow-up is #1255.

Status

Step Description Status

0

File the v1 issue (#1253) + follow-ups (#1254 hook-owned manifest, #1255 in-process capture) + the upstream claude-quickstart macro-feedback escalation; commit this plan + nav.

Done (2026-07-27) — #1253 (this MR)

1

Implement the report harness — validate_report.rs (schema + atomic writer + Redactor + report.validate() + const inventory + spawn_and_tee + CaptureError + Harness), main.rs module wiring, the validate.rs run()/run_inner() refactor + run_gate/run_value_stage + tee helpers, junit.rs required-parse + per-test bodies, the [profile.validate] nextest profile + fs2 lock, validate-in-network host-ownership, the CI junit repoint, docs + CHANGELOG.

Done (2026-07-28) — this MR (see As-built deviations)

2

Local battery (shipped the manual way — the command isn’t on main yet) + J1–J8 fresh-subagent review + R1–R5; MR (Closes #1253); force-merge; close.

Done (2026-07-28) — !1008 (merge 1c678a0a, impl 28a49a21)

Issue: #1253 (high)
Follow-ups: #1254 (hook-owned manifest), #1255 (in-process-gate full-capture)
Branch: feature/validate-report

Context

cargo xtask validate (the pre-push battery core, .githooks/pre-push) is fail-fast: its gate stages run before the tests, so an earlier gate failure exits nonzero while test-results/integration/results.xml still holds the previous run’s green JUnit — "push failed, test-results green" — forcing a terminal log-grep. This v1 makes validate emit one always-present, atomically-written, self-describing report so that after git push you read one file (test-results/validate-report.json) to know exactly which stage failed and why.

Blocker: single-run ownership of the JUnit (resolved — both mechanisms)

validate and cargo xtask test --integration both run --profile integration and therefore both write test-results/integration/results.xml (.config/nextest.toml, validate.rs, test.rs), with no lock today (docker::acquire_lock exists but only perf/e2e/dev use it). Verified: cargo-nextest 0.9.133 has no CLI/env override for the JUnit path (its --config is Cargo’s, not nextest’s), and relocating the path would move the whole target/nextest/<profile>/ store, breaking the deliberate clippy↔nextest build-cache share. So resolve with both:

  1. Ownership — a dedicated nextest profile. Add [profile.validate] to .config/nextest.toml (mirror [profile.integration] — fail-fast=false, slow-timeout=120s, test-threads=4, the 3 test-group overrides, store-success/failure-output=true) with a [profile.validate.junit] path of ../../../test-results/validate/results.xml. validate switches to --profile validate and reads that path; ensure_test_results_dirs adds test-results/validate. Now validate never touches integration/results.xml — test --integration owns it exclusively, so validate’s embedded JUnit is unambiguous. (.config/nextest.toml is an active sync-override — sync-overrides.toml, immutable-hashed, drift held advisory exit-3 — so adding a profile is already sanctioned and fits the override’s "canopy’s junit paths feed its CI artifact layout" rationale.) CI consequence (must-fix, material): the CI validate-in-network job runs validate (not test --integration) and collects junit: test-results/integration/results.xml + paths: test-results/integration/ (.gitlab-ci.yml). After the profile switch validate writes test-results/validate/, so that job’s report/artifact would go permanently empty — repoint its junit:/paths: to test-results/validate/ in the same MR.

  2. Concurrency — an fs2 lock. Reuse the docker::acquire_lock RAII pattern (docker.rs, try_lock_exclusive + poll-timeout + drop-unlock; fs2 already an xtask dep) at the top of both validate::run and test::run, held to end of run — so two validates, and validate + test --integration, serialize. Under the lock the JUnit read is guaranteed this run’s, so embedding the parsed JUnit by value is race-free (no post-run copy needed). The lock file must be workspace-relative (e.g. test-results/.validate.lock), NOT docker::acquire_lock’s `env::temp_dir() default — the in-network container’s /tmp differs from the host’s, so a temp-dir lock wouldn’t contend across the bind mount; a workspace-relative file shares one inode through the mount so host and in-network runs contend. Documented residual: some container runtimes don’t honor cross-boundary flock; validate-in-network is opt-in (#339) and rarely concurrent with a host validate — stated as a known limit, not silently assumed. Version note (not a defect): the "nextest has no JUnit-path override" fact is pinned to cargo-nextest 0.9.133.

The artifact — schema

test-results/validate-report.json, atomically written, replaces the removed validate-summary.json.

ValidateReport {
  schema_version: 1,
  run_id: uuid-v7,  runner: "host" | "in-network",           // provenance
  commit_sha?, branch?, worktree_dirty?: bool,               // provenance, BEST-EFFORT (a git hiccup must not abort init)
  command, args,  started_at, ended_at?: RFC-3339,
  state: running | pass | fail | interrupted,                // only run_inner finalizes pass/fail
  current_stage?: String,  failed_stage?: String,
  stages: [ StageResult ],                                   // FULL predeclared inventory, in order
  test_report: JUnitEmbed,
}
StageResult {
  stable_id, display_name, index,
  execution: pass | fail | skip | not_run | running,
  policy: blocking | advisory,
  capture: full-tee | error-chain-only | na,
  started_at?, ended_at?, duration_ms?,                      // optional while running/not_run
  skip_reason?: String(redacted),
  error_chain?: String(redacted),
  diagnostics?: SubprocessDiagnostics,                       // same richness for advisory + blocking failures
}
SubprocessDiagnostics {
  program, argv: [String](redacted), env_removed: [String],
  spawn: spawned | spawn_failed{error},
  exit?: { code: i32 } | { signal: i32, name },
  stdout: BoundedTail, stderr: BoundedTail,
}
BoundedTail { text: String(redacted, <=64KiB), truncated: bool, total_bytes: u64 }
JUnitEmbed {
  path, status: ok | missing | malformed | unreadable | io_error,
  summary?: { total, failures, errors, skipped },
  failed_tests?: [ { suite, name, message(redacted), stdout: BoundedTail, stderr: BoundedTail } ], // per-test bodies
}

Lifecycle + state machine

  • run() → thin wrapper + run_inner() → Result<()>. The wrapper: acquire the fs2 lock; resolve workspace root; ensure_test_results_dirs + clear only validate-owned stale files (test-results/validate/results.xml + the old validate-summary.json; never touch unit/ / e2e/ / integration/); collect provenance; write the report state:running with the full predeclared inventory (all not_run) — before the first fallible gate (moved ahead of reconcile/preflight/visibility/signing/docs/SPDX, which today precede the in-body dir creation). Then call run_inner; on Ok finalize pass; on Err finalize fail + failed_stage. This init (lock → root discovery → ensure_test_results_dirs hoisted here → clear owned stale files → best-effort provenance → first atomic write) is a pre-inventory bootstrap phase, not itself an inventory stage (the inventory’s index-0 stage is sysmon-reset). Guarantee is therefore "present after successful initialization" — bootstrap can fail (root discovery, mkdir, first write) and leave no report, which is honest; provenance is best-effort so a git hiccup never aborts bootstrap.

  • Checkpoint BEFORE each stage. Flip that stage to running{started_at} + set current_stage and atomically rewrite the report before invoking it, so an interrupt mid-clippy is distinguishable from never-started.

  • Interrupt discrimination is by-artifact: state:running left on disk = killed by signal (no cleanup ran); a graceful unwind (anyhow ? / panic caught in main) rewrites running → fail (with failed_stage) or → interrupted for an interrupt sentinel. A Drop guard is best-effort only (marks interrupted if run_inner never finalized) — not the persistence path.

  • Invariants (pinned in a report.validate() used by tests): ended_at/duration_ms optional, forbidden while running; pass implies no BLOCKING stage in {fail, running, not_run} — an advisory-fail is permitted under pass (that’s the whole point of advisory, e.g. a red cargo deny with a green battery); fail implies failed_stage names a blocking-fail stage. Precedence: a stage’s verdict is authoritative for the process exit (never mask a red battery); a report-write failure surfaces separately (distinct stderr + nonzero) — ordering stage-fail > reporting-fail (JUnit) > persistence-error.

  • xtask-compile failure is out of reach (stated): if xtask won’t build, validate never runs and the prior report persists — but its run_id/commit_sha/worktree_dirty/started_at let a consumer detect staleness. Proving a later attempt happened needs the outer hook (#1254), not this binary.

Typed model + run_stage

ExecutionResult {Pass, Fail, Skip} × EnforcementPolicy {Blocking, Advisory} × diagnostics × a returned value:

fn run_stage<T: Default>(report, stable_id, policy, body: impl FnOnce() -> Result<StageBody<T>>) -> Result<T>
// StageBody<T> = Produced(T) | Skipped{reason}

Produced(v) → pass, return Ok(v). Skipped → skip, Ok(default). Err → capture full diagnostics, then: Blocking records fail + failed_stage + returns Err; Advisory records fail (identical diagnostics) and returns Ok(default) — the run continues. This fixes cargo deny (today catches + println! + swallows to Ok) and types the 4 value-returning setup stages (run_stage<T>): reconcile-ports-env (ReconcileResult), heal-realm-drift (the load-bearing env: Vec<(String,String)>), secrets-decrypt (advisory → Vec empty on Err, reproducing today’s literal-fallback while capturing sops stderr), devstack-project-name (String). The ~40 gate stages use the T=() form.

Capture — two chokepoints + explicit error-chain-only

One capture runner owns tee + SubprocessDiagnostics. run_cmd_with_env (today .status() only — "can tee" is the target) gains capture and an env-remove hook (so cargo machete’s `.env_remove("CARGO_PKG_NAME") stops being the hand-rolled exception). The same combinator wraps the second chokepoint docker::compose_cmd_with_env. This gives full-tee for all RC stages (deny, fmt, clippy, the 10 #896 gates, nextest, doctest, docker-build) + machete + compose/observability, and .output()-already stages (preflight, visibility glab, signing git config) route their captured buffers in.

Honestly error-chain-only (deep multi-subprocess flows, tagged capture:"error-chain-only" in the report, not silent): reconcile-ports-env, heal-realm-drift, devstack-ready (devstack_guard.rs), secrets-decrypt (devtools shell-out), image-cleanup. Closing this gap is #1255.

Diagnostics carrier (was unspecified): on failure the chokepoint attaches the SubprocessDiagnostics to the returned anyhow::Error as a typed context payload; run_stage downcasts it off the bubbling error and records it on the failing stage (no thread-local, no out-param).

Buried chokepoint calls: compose_cmd_with_env is reused deep inside devstack_guard::ensure_ready and the reconcile/heal paths — so when one of those buried calls fails, its diagnostics ride the same error up to the outer error-chain-only stage, which then records error-chain plus the failing subprocess’s diagnostics (a bonus, not a contradiction). Only a failure that never went through a chokepoint is error-chain-only.

Tee policy: 64 KiB per-stream byte cap (ring/tail, not lines); reader threads keep draining on terminal BrokenPipe (buffer-only) so the child never blocks; child.wait() (reap) on every path incl. reader-thread create/join failure; tails stored as bytes, from_utf8_lossy at serialize; signal exit → ProcExit::Signal and a stage fail.

Redaction, atomic write, JUnit verdict

  • Redaction. A Redactor seeded from an explicit sensitive-value set = the decrypted secret values (decrypt_to_pairs(), before env.extend) + the .env floor (docker.rs) — because ports and secrets are merged into one env vec, sensitivity must be handed in, not inferred. Redact every free-form persisted field (error_chain, skip_reason, argv, both BoundedTail.text, and JUnit message/bodies) before truncation. Over-redaction guard: only mask values len >= 8 and not in a small allowlist (true, canopy, localhost, numeric ports). Documented blind spots: (a) ambiently-exported shell secrets not in the injected vec can’t be value-matched — the schema doc states the report redacts injected secrets only; (b) an injected secret shorter than the 8-char guard is not masked in the 0644 file (realistic keys are >=8, but stated).

  • Atomic write. Linux-only (rust-toolchain.toml/musl): temp .<final>.<pid>.<nanos>.tmp in the dest dir, fsync file + parent dir, rename(2); unlink temp + propagate on error. Mode 0644, not 0600 — redaction (not the inode) makes the file non-secret, which also keeps it host-readable across the in-network boundary.

  • JUnit verdict (owned by a blocking test-report stage). Rename the inventory’s summarize-digest to a blocking test-report stage so the reporting-failure verdict can legally set failed_stage (the fail invariant requires a blocking-fail owner). On the required post-nextest path, nextest exit 0 but a missing/malformed/unreadable report makes test-report a blocking fail → state:fail (an apparently-successful run with no trustworthy artifact is untrustworthy). Status in {ok, missing, malformed, unreadable, io_error}; parse via a single File::open classified by ErrorKind (no exists()-then-read TOCTOU). Fix junit.rs which currently treats a missing file as an empty successful summary — that leniency stays only for optional/not-yet-run profile paths.

  • Per-test failure bodies (meets the no-grep goal). store-failure-output=true means the JUnit carries each failed test’s captured stdout/stderr; embed them as bounded, redacted BoundedTail`s per `FailedTest. This is what lets the report replace the log for the racing-test case (the 40P01 panic text lives in the failed test’s body).

validate-in-network + workspace-root scope

  • in-network ownership. validate_in_network.rs runs validate as root in a workspace-bind-mounted container → fresh-checkout root-owned artifacts. Not --user — running the container as the host uid would break the image’s root-owned cargo registry/target (build fails), and libc::getuid isn’t even a dep. Instead: the host-side wrapper reads its uid/gid via id -u / id -g (no new dep), passes them as -e HOST_UID -e HOST_GID, keeps the container root (cargo works), and the in-network command chown -R $HOST_UID:$HOST_GID the workspace test-results/ before exiting so every artifact is host-owned. Set runner:"in-network" provenance. With 0644 the report is host-readable regardless. Update validate-in-network.adoc.

  • Workspace-root: narrow. Resolve only the 3 report paths (report json, validate’s JUnit, cleared files) through docker::workspace_root(). Do not chdir the process — SPDX traversal, Dockerfile build context, sysmon output, api-docs snapshots are deliberately CWD-relative and correct from root (the hook runs from root); chdir would silently redefine the SPDX scan root.

The predeclared stage inventory (exact, ordered)

The report seeds all stages as not_run from a const table; duplicate/unknown stable ids are rejected at startup. stable_id`s are freshly assigned consts (reuse the `timings label where one exists, but ~11 stages have no timings.push — sysmon-reset, reconcile-ports-env, heal-realm-drift, secrets-decrypt, secret-floor-write, ensure-test-results-dirs, test-report, image-cleanup — the single devstack timings label maps to 3 ids, and the label is docker not docker-build; assign those fresh).

Conditional stages: secret-floor-write (runs only in the decrypt-Ok arm) and image-cleanup (only if docker-build ran; skipped under --skip-docker) land not_run when their guard/predecessor didn’t fire (reserve skip{reason} for a stage that ran its own decision and chose to skip, e.g. visibility-no-glab).

Full inventory (stable_id · capture · enforcement), in run() order:

sysmon-reset(na,na) · reconcile-ports-env(err-chain,block,SRV) · heal-realm-drift(err-chain,block,SRV) · secrets-decrypt(err-chain,advisory,SRV) · secret-floor-write(err-chain,advisory) · preflight(err-chain→routed,block) · visibility(err-chain→routed,block,skip) · signing(err-chain→routed,block) · docs(err-chain,block,skip) · spdx(err-chain,block) · ensure-test-results-dirs(na,block) · plan-lint · rules-lint-inputs · audit-literals · audit-unwraps · typed-ids-path-uuid · http-clients-client-new · route-authz · outbox-migrations · mq-topology (all err-chain,block) · deny(full-tee,advisory) · fmt(full-tee,block) · machete(routed-via-env-remove,block) · rules · vendor-check · capabilities · api-docs(err-chain,block; api-docs skip-capable) · clippy(full-tee,block) · quality-budgets(err-chain,block) · secrets-yaml-lint · migrations-lint · ci-config-lint · data-tenancy · policy-audit · policy-audit-federal · policy-drift · action-coverage · input-coverage · scenarios-audit (all full-tee,block) · devstack-project-name(na,block,SRV) · devstack-ready(err-chain,block) · devstack-observability(routed-via-compose,block) · nextest(full-tee,block) · test-report(embeds+verifies the JUnit; blocking — owns the reporting-failure verdict) · doctest(full-tee,block) · docker-build(full-tee,block,skip) · image-cleanup(err-chain,advisory).

(SRV = a value-returning setup stage using run_stage<T>.)

NOTE
Erratum (#1267, post-archive): api-docs was subsequently moved AFTER devstack-observability (immediately before nextest) — the gate diffs each LIVE service’s OpenAPI against the branch snapshots, so it must run after devstack-ready has synced the running stack to the tree. The inventory above records the order as shipped by #1253; validate_report::STAGES is canonical.

Files touched

File Change

xtask/src/validate_report.rs (new)

schema types (ValidateReport/StageResult/SubprocessDiagnostics/BoundedTail/JUnitEmbed) + serde; atomic temp+fsync+rename writer (0644); Redactor; report.validate() invariants; the const stage inventory.

xtask/src/main.rs

mod validate_report; (top-level, by mod junit;) + top-level unwind → fail/interrupted finalization for the report.

xtask/src/cmd/validate.rs

run() → wrapper (workspace-relative lock, init running first, finalize) + run_inner(); generic run_stage<T> + the () form; hoist ensure_test_results_dirs into bootstrap; tee+env-remove in run_cmd_with_env + typed-error diagnostics carrier; switch to --profile validate + read test-results/validate/results.xml + fix the setup println! string; the test-report blocking stage (rename summarize-digest); route preflight/visibility/signing captured buffers in; remove the validate-summary.json writer.

xtask/src/docker.rs

tee capture in compose_cmd_with_env via the shared combinator + the typed-error diagnostics carrier; the dedicated workspace-relative integration-junit lock helper.

xtask/src/cmd/test.rs

acquire the same lock at top of run().

xtask/src/cmd/validate_in_network.rs

keep the container root; pass -e HOST_UID/HOST_GID (host id -u/id -g, no new dep) + chown -R test-results/ before exit; runner:"in-network" provenance.

.gitlab-ci.yml

repoint the validate-in-network job’s junit: + paths: from test-results/integration/ to test-results/validate/ — else CI’s report goes empty after the profile switch.

xtask/src/junit.rs

required-parse path (open+classify, 5 states, no TOCTOU); embed per-test bounded bodies; delete now-unused write_summary_json (+ print_failures_digest unless the flow still prints a digest); fix the doc + the missing=empty-success leniency (required path only).

.config/nextest.toml

new [profile.validate] + [profile.validate.junit] (sanctioned sync-override edit).

docs/…​/testing.adoc (canonical) + short xrefs from tooling/xtask-catalog.adoc + contributor-workflow.adoc + validate-in-network.adoc

schema, states, validate-only scope, "does not prove remote delivery", the in-process error-chain-only limitation.

synced docs/modules/standards/pages/testing.adoc

its profile→JUnit-path table omits [profile.validate] after this lands; it’s a synced file (check-docs would flag a local edit as drift) → escalated to claude-quickstart (#1254) rather than edited locally; note the expected advisory drift.

CHANGELOG.adoc

== Unreleased entry (Closes #1253).

memory (post-merge)

update push-and-battery-ops: "read test-results/validate-report.json`"; the racing-test rule is retargeted (read the failed test’s embedded body, not the log) now that per-test bodies exist — keep the `.env.local-export + ls-remote rules (the report proves neither).

Decisions surfaced (resolved at sign-off)

  • Lock scope: a dedicated {project}-integration-junit.lock (serializes only validate + test --integration), not the shared {project}-xtask.lock (which would also block dev up/e2e/perf). Precise blast radius.

  • Lock timeout: 60s fail-fast with a clear message ("another validate/integration run holds the lock") — don’t queue behind a multi-minute battery.

  • Per-test bodies + memory: v1 does embed bounded per-test failure bodies, so the report genuinely replaces the log for test failures; the racing-test memory rule is retargeted to the report (not deleted, not left pointing at the log).

Verification

cargo build/clippy -p xtask clean, and cargo nextest run -p xtask covering: early-validate (SPDX) failure → fail`failed_stage`, not stale-green; a subprocess gate (clippy) failure carries the tee tail; an in-process gate carries the error chain; an advisory failure (deny) records `fail`advisory with full diagnostics, the run continues, and the final state:pass passes report.validate() (advisory-fail permitted under pass); setup run_stage<T> returns the value; nextest exit-0 with absent/unreadable/malformed JUnit → test-report blocking fail (5 states, failed_stage="test-report"); per-test failure body embedded + redacted; a buried compose_cmd_with_env failure attributes its diagnostics to the outer error-chain-only stage; secret value in captured output is redacted before truncation; report.validate() invariants; atomic write + simulated interrupt leaves valid JSON in running/interrupted (never corrupt); concurrent validate + validate/test --integration serialize on the lock; same-HEAD dirty-worktree provenance; a manual clippy defect outside xtask (so the reporting binary still compiles + initializes); alternate-CWD resolves the 3 report paths; exact predeclared inventory present with post-failure stages not_run; conditional secret-floor-write/image-cleanup land not_run on the not-taken path; fresh-checkout validate-in-network produces host-owned artifacts (chown-back) and the CI validate-in-network job collects a populated test-results/validate/results.xml; local Antora build renders the new pages + plan nav (not just check-docs).

Documentation updates

  • testing.adoc (schema/states/validate-only scope) + short xrefs from tooling/xtask-catalog.adoc, contributor-workflow.adoc, validate-in-network.adoc.

  • CHANGELOG.adoc == Unreleased (Closes #1253).

  • Follow-up issues filed (#1254 hook-owned manifest + upstream claude-quickstart escalation; #1255 in-process capture) + /relate.

  • Plan → Archive on completion (final MR of the stream).

As-built deviations (R2 — living spec)

The design shipped as specified except for these bounded, deliberate deviations:

  • Inventory is 46 stages, not 47. ensure-test-results-dirs runs in the pre-inventory bootstrap (the report can’t be written until its dir exists), so it is not a recorded stage. Index-0 is sysmon-reset, exactly as the plan’s "index-0 stage" note anticipated.

  • Typed model shape. run_stage<T> split into run_value_stage<T> (blocking, value-returning: reconcile / heal / devstack-project-name) + run_gate (unit T=(), honors blocking/advisory) + a bespoke secrets-decrypt arm (advisory, seeds the redactor, writes the floor). StageBody/in-stage self-skip was dropped: call-site skips use Harness::skip_stage (docker-build under --skip-docker / no Dockerfile), and internally-skipping gates (visibility/docs/api-docs) record pass in v1 (they return Ok). A skip execution is thus reserved for call-site skips; wiring the internal-skip stages to a skip{reason} is a minor future refinement.

  • Compose capture is a new opt-in variant. Rather than wrap the shared docker::compose_cmd_with_env (used by interactive dev/e2e), a dedicated compose_cmd_with_env_captured tees only validate’s devstack-observability call — so dev/e2e terminal I/O is untouched. Consequently devstack-ready (and the buried devstack_guard compose calls) stay purely error-chain-only; the "buried diagnostics ride up" bonus lands with #1255, not v1.

  • preflight/visibility/signing are error-chain-only in v1 (their bail! messages already embed the pertinent stderr). "Routing their captured .output() buffers in" is folded into #1255.

  • Unwind finalization is the Harness Drop guard, not a main catch_unwind. main.rs only adds mod validate_report;. A blocking-stage failure finalizes fail directly; a ? that bypassed a recorder is finalized by the wrapper (finalize_uncaught); a panic/early-return trips Drop → interrupted; a signal kill leaves running on disk.

  • Bootstrap order is root → lock → dirs (the workspace-relative lock path needs the root first), a trivial reorder of the plan’s "lock → root".

  • Per-stage started_at/ended_at are populated via chrono (a cheap in-process clock, already an xtask dep) rather than omitted — the schema’s optional timing fields are filled.

None of these change the artifact’s contract (schema, states, invariants, single-run ownership, redaction, host-ownership) or the user-facing guarantee.

Delivery + lifecycle

This is a meaty single MR (report harness + capture + lock + profile + in-network + docs); it stays one coherent MR because a partial report is worse than none. Implement on feature/validate-report; full battery shipped the current manual way (the command isn’t on main yet). Closes #1253.

Edit this page · default