Plan: chain-v2 substrate — schema, KAT vectors, restricted roles, empty-genesis install (#1246, epic &73)

#1278 — anchor authority (critical, w5): the async ExternalAnchorAuthority trait + canopy-store impl with append-only hardening (conditional create, receipt/version capture — today’s put discards the object_store PutResult, crates/canopy-store/src/store.rs:93-95 —, version-aware reads, split credentials, configuration attestation, outage/replay tests); production anchor-signing key identity + RFC 7638 JWK-thumbprint kid (chain-purpose/source namespace); strict anchor-JWS verification wiring; emission cadence/SLO; the final authority selection RATIFIED by a further ADR-014 amendment. #1279 depends on it.

#1279 — coordinated-downtime cutover (critical, w8): v1 truncate (guard-GUC, recorded ADR-016 exception) → drop → v2 rename + _v2 index rename/rebuild + re-creating the family-named append functions against renamed tables + #1197 EXPLAIN retarget; runtime-URL swap + explicit per-object v1-surface grants (never ALTER DEFAULT PRIVILEGES); BYTEA reader adaptation before reopen; old-owner handoff; credential activation (NOLOGIN→LOGIN); genesis + external notarization + epoch activation; binding shard-count selection; the executable go/no-go gate + recovery-matrix dry-run.

#1280 — epoch closure/rollover (high, w5): the C4/C5 crash-resumable cross-database transition executor (the substrate ships the fenced state column, the append-side fence, and the epoch→head lock-order protocol ONLY).

#1281 — serde_jcs conformance, repo-wide (high, w3): migrate the seven legacy canonicalization call sites off the nonconforming serde_jcs 0.1 (they produce RFC-compatible bytes only by accident of ASCII snake_case keys + in-bounds integers). Not in #1246’s scope.

No KAT precedent exists. Insta snapshots deliberately REDACT hashes (services/canopy-security/tests/security_test.rs:350-352); only single-primitive well-known digests exist (e.g. crates/canopy-policy/src/source.rs:194-200). This plan defines the repo pattern.

The canopy login is simultaneously devstack cluster superuser (POSTGRES_USER: canopy, docker-compose.yml:43,156-224), boot-time migration runner (crates/canopy-api/src/bootstrap.rs:117-126 — and the app pool is created at bootstrap.rs:103-111 BEFORE the migrator runs), and runtime identity (docker-compose.yml:845,960,1012). Zero role/GRANT DDL exists in any migration. The #624 append-only guard is trigger+GUC precisely because the owner bypasses GRANT/REVOKE (services/canopy-security/migrations/20260603120000_audit_events_append_only_guard.sql:9-11); C8’s mandate is to remove that constraint structurally.

v1 DDL: nullable TEXT hex hashes, no UNIQUEs on chain columns, LIKE … INCLUDING ALL archives (20260326000000_create_security_tables.sql, 20260402000001_add_hash_chain.sql; tanf 20260325000001 + 20260425000000; medicaid copies). No version column (Amendment 1’s hash_version dropped by 20260624130000). Current migration max: 20260904000000.

Signing: ECDSA P-256 detached JWS over JCS bytes, kid + dual-key rotation (crates/canopy-signing/src/{signer,envelope,verifier}.rs); typ hardcoded canopy-determination+jwt (signer.rs:55); the verifier validates no header fields strictly (verifier.rs:36-47); p256 is RFC 6979 deterministic (signature KATs are reproducible).

Object store: Garage devstack / S3 production via canopy-store (crates/canopy-store/src/lib.rs:5-9); the current API is overwrite-capable and discards put results — hence #1278’s hardening ACs.

JCS today: serde_jcs = 0.1 at seven call sites incl. the audit chain (services/canopy-security/src/store/mod.rs:48-72) with PG-JSONB normalization SELECT $1::jsonb (:112-121).

EphemeralSchema: per-test test_<hex> schemas created/owned by canopy, options=-c search_path=<schema>,public (crates/canopy-test-lib/src/db.rs:260-274); the sqlx migrator holds a per-database advisory lock (same-DB runs serialize; cross-DB runs on a shared cluster do not).

xtask has no PG driver by design (xtask/src/psql.rs:3-5 shells out); its dispatcher is synchronous. (D7 deliberately reverses the no-driver stance for the genesis installer / migration job — the first xtask commands that must own transactions.)

Research verified serde_jcs 0.1 is nonconforming beyond the external review’s claim: it sorts SERIALIZED key bytes (quotes + escapes included), which flips ordering even on plain-ASCII keys ("a" vs "a b" — space 0x20 < closing-quote 0x22; the escape class \b vs !), plus the non-BMP UTF-16 divergence and verbatim i64 emission (serde_jcs-0.1.0/src/entry.rs:5, ser.rs:348-358,221-226), and a todo!() panic path on arbitrary_precision numbers (ser.rs:166-171).

serde_json_canonicalizer 0.3.2 (MIT, 2026-02 release, ~2.5M recent downloads, three-crate footprint, ryu-js 1.0.1) is conforming on BOTH axes — raw-key UTF-16 code-unit sort, ES6 number emission — and bundles the official cyberphone/json-canonicalization test corpus (tests/resources/testdata/), which with RFC 8785 §3.2.3 + Appendix B seeds canopy’s KAT corpus INDEPENDENTLY of our implementation (killing the KAT-circularity finding). Alternatives rejected: serde_jcs 0.2 (conforming but ryu-js 0.2, no bundled vectors, thin adoption), json-canon (numbers still nonconforming, dormant), json-syntax (code-point sort), vendoring (audit surface without benefit).

The validation layer is mandatory regardless of crate: conforming implementations SILENTLY ROUND integers beyond ±(2^53−1) through f64 — two distinct i64s canonicalize to identical bytes, a semantic collision an audit chain must refuse. canopy-chain therefore recursively validates the ENTIRE value tree (payload, manifest, every integer — not just chain_seq) before hashing: reject non-finite numbers and INTEGERS with |n| > 2^53−1 with a typed ChainError. Non-integer (float) values PASS — arbitrary event payloads legitimately carry them, and the conforming serializer emits ES6 shortest-round-trip floats which a strict verifier reproduces; only the integer-collision class is refused. Keys need no restriction.

multiple-versions = "warn" in deny.toml tolerates the transient ryu-js 0.2.2+1.0.1 dual; #1281 retires 0.2.2.

Validated newtypes (invalid states unrepresentable): ShardCount (1..=32767 — the SMALLINT-lossless cap), ShardId (< count), Epoch (>=0), EventSeq (1..=2^53−1), HeadSeq (0..=2^53−1), RoutingVersion, ChainInstanceId (UUIDv7: separate mint vs fallible decode paths), HashFormulaVersion (const 2; fallible DB/wire decode), EventHash([u8;32]) (hex only at edges), ChainFamily (Audit/Fti/EleGrant), ChainSource (canopy-security/canopy-tanf/canopy-medicaid — the C1 service/database identity, see D2), ChainError (the thiserror enum — every canonicalization/validation/genesis fn is fallible). Carve-out stated up front: the payload traffics in serde_json::Value (wildcard event payloads — the AuditChainInputs.metadata precedent, store/mod.rs:48-62), annotated per the house rule.

Typed per-family payload builders with CLOSED field sets (the field-coverage tables below) — the only production way to construct a canonical_event_payload; excluded columns are structurally unplaceable. The SQL side re-validates exact keys/types (D6).

Exact ChainEnvelope preimage JSON (JCS-ordered; formula-bearing, pinned):

{"chain_epoch": 0, "chain_family": "audit|fti|ele_grant",
 "chain_instance_id": "<uuid-lowercase>", "chain_seq": 1,
 "chain_source": "canopy-security|canopy-tanf|canopy-medicaid",
 "domain_tag": "canopy-chain-v2", "hash_formula_version": 2,
 "payload": {}, "previous_hash": "<64-hex-lowercase>", "shard_id": 0}

Independent protocol versions (a vector change bumps ITS OWN version, never the others): event_hash_formula_version=2, routing_version=1, genesis_version=1, anchor_manifest_version=1, anchor_signing_version=1.

shard_for(routing_id: Uuid, shard_count: ShardCount) → ShardId — first 8 bytes of SHA-256("canopy-chain-v2/routing/v1" ‖ 16 raw UUID bytes) as u64 BE, mod count (NonZero-backed — no arithmetic_side_effects hazard). Routing id = the immutable event id: the bus envelope id for #-queue events; for direct ingest the RECOMMENDATION is the server-minted envelope event_id (api/mod.rs:240,265 already mints it; the row PK is append-minted and excluded) — the binding pin is #1207’s per C3, and Amendment 6 corrects the ADR’s "row id" wording. Shard placement is NOT re-derived in-database (no pgcrypto dependency): the SCRUB recomputes and validates placement fail-closed (#1205/#1206), and #1207 tests the write side.

Genesis rules (C3 "all specified"): the empty-head hash = SHA-256 of the JCS of {"chain_epoch": <int>, "chain_family": …, "chain_instance_id": …, "domain_tag": "canopy-chain-v2/genesis", "genesis_version": 1, "shard_id": <int>}; a first row’s previous_hash = that hash — never NULL, never a "GENESIS" sentinel. Epoch N>0 adds "previous_epoch_anchor_hash": "<64-hex>" (C4 linkage). Epoch-0 root = the topology row’s instance id (no prior anchor; the zero sentinel appears only in the genesis anchor’s previous_anchor_hash).

GenesisPlan::compute(instance_id, minted_at, family, source, shard_count) → GenesisPlan — PURE (identity and clock INJECTED; xtask mints them), returns every row to insert + the genesis manifest bytes; verify_genesis_state(fetched_rows) → Result<(), ChainError> — the pure post-install check (D7).

Roles (per cluster; created by migrations; NOLOGIN until cutover credential activation — passwordless-LOGIN is NOT dormant, peer/trust/cert auth could still bind): service-qualified owners canopy_chain_owner_security / _tanf / _medicaid (ADR-001: one cluster-global owner would give members ownership power across co-located service DBs in shared-cluster mode); canopy_chain_verify (privilege bundle, never a connection identity); canopy_security_verify (the LOGIN principal for ALL verifier pools, member of the bundle, created on all three clusters); canopy_chain_maintenance (EXECUTE on archive/purge fns; granted to nothing until #1208/#1247); canopy_chain_incident_admin (EXECUTE on the incident-RESOLUTION fn only — background verification carries NO resolution authority; wired to the admin surface at #1205/#1206); the _app runtime logins (canopy_security_app etc.).

Grant matrix (pinned): _app — EXECUTE its family’s append fns + EXECUTE chain_head_lock (part of the append flow — the only path to last_hash for Rust-side hashing) + SELECT own family tables + SELECT instances/topology/epochs/sources (epoch + shard-count discovery; no chained data) + SELECT chain_status_v; NO DML on any chain table, no chain-role membership. canopy_chain_verify — program DBs: SELECT projection views (live + archive) + instances/topology/epochs/heads/sources, strictly read-only; canopy_security additionally: EXECUTE the checkpoint lease/fence CAS fn, INSERT on runs, EXECUTE the incident-latch fn and the anchor append/transition fns (NO table UPDATE grants anywhere — every state change goes through a guarded fn). PUBLIC EXECUTE is REVOKED on every function at creation (PostgreSQL defaults it on — without the revoke, any login could call the SECURITY DEFINER appends). Explicit CONNECT + schema USAGE for every real principal; tests assert PUBLIC/app/verify lack schema CREATE.

Provisioning: in MIGRATIONS (init scripts run once per volume; the program containers have no init mounts, compose:169-201; production never sees them). Idempotent DO-blocks catch duplicate_object OR unique_violation (the concurrent loser gets 23505 off pg_authid; the real race is cross-database — tanf+medicaid on one shared cluster; same-DB runs are serialized by sqlx’s per-DB migrator lock). On exists: reconcile, fail closed — assert exact LOGIN/SUPERUSER/CREATEROLE/REPLICATION/BYPASSRLS/password-null/membership state against pg_roles/pg_auth_members (role squatting or stale config refuses the migration).

Ownership mechanism: create-then-transfer with the enabling grants. No SET ROLE (it persists → sqlx’s sqlx_migrations bookkeeping would run as the owner role and abort). Between the role block and the transfers: GRANT <owner_role> TO current_user (PG16+ CREATEROLE grants the creator ADMIN OPTION, not membership — membership is required for ALTER OWNER; records a membership row, harmless for superusers) and GRANT USAGE, CREATE ON SCHEMA <current_schema()> TO <owner_role> (ALTER OWNER requires the new owner hold schema CREATE; SECURITY DEFINER bodies need USAGE — public masks USAGE via PUBLIC’s default, EphemeralSchema test<hex> schemas do not). Both statements are dynamic SQL in DO blocks (GRANT takes no expression). Transfers cover tables, functions, VIEWS, and composite types. Runbook: the production migration credential needs CREATEROLE + schema ownership (or CREATE WITH GRANT OPTION); the rotation wrinkle is documented (a rotated CREATEROLE credential lacks ADMIN on roles it didn’t create — the reconciliation step surfaces this rather than half-applying).

The migration JOB (the runtime process must never hold the privileged credential): migrations for the three chain services run via cargo xtask migrate apply --service <svc> (nested under the EXISTING migrate command — xtask/src/cmd/migrate.rs carries snapshot/rollback today) as a deploy-time step — devstack: a compose one-shot service per chain service (dormant until #1279 activates the split); production: a deploy-job in the runbook. Bootstrap keeps its embedded migrator for ALL services until cutover; at cutover the three chain services set {PREFIX}SKIP_MIGRATIONS (boolean env, default false — listed in configuration-reference.adoc, tested in MR-3): true means bootstrap runs NO migrator (the job owns migrations from then on; refused in CANOPY_ENV=development unless the compose one-shot is configured — realized as a second documented boolean, {PREFIX}MIGRATIONS_JOB_CONFIGURED, set only by deployments that wire the job (the devstack chain-migration-split profile) — so a dev cannot silently strand a schema), the app pool connects as _app, and NO privileged URL exists in the runtime environment — the compromise-resistant form an in-process migration-URL design could not give (a closed pool does not un-know a credential). The second bootstrap fix landing in MR-3, benefiting every service: the ordering fix — migrate FIRST over a max-one short-lived pool, closed on every path, THEN build the app pool (today the app pool is created at bootstrap.rs:103-111 before the migrator runs at :117-126); an injected settings seam for tests (no env mutation). The xtask migrate apply job carries the full guard posture on its URL (validate_database_name + DbPool::connect_with TLS — never bare PgPool::connect).

Functions (SECURITY DEFINER, owner-role-owned, created via DO … EXECUTE format(…, current_schema()) with a per-function interpolated SET search_path = <schema>, pg_temp):

Projection views (live _v2 + archive _v2 per family): the hashed business fields as typed columns + position/hash columns + canonical_event_payload; NEVER an excluded column. The minimality test asserts the exact information_schema column list. The verify bundle’s duties (D3 layer 4, archive-awareness, orphan detection via instances/topology) map onto its SELECT set above.

Rerun predicate (three arms, ALWAYS validating before any refusal): no topology → fresh install; topology + genesis anchor absent → RESUME Phase B (after proving heads untouched: all last_seq=0, hashes = the genesis values); topology + anchor present → run the FULL Phase-C validation and only then report "already installed" (a crash after B can still complete C); any supplied param differing from installed state → hard error, never silently ignored.

Phase A (one tx, TARGET DB only): insert the chain_instances row + the topology pointer + the chain_sources row(s) binding this DB’s source(s) to the new instance (without this, every post-cutover append refuses against an empty registry — and no dormant-phase test would catch it) + the epoch-0 row (state='installing' — the chain is NOT appendable until the #1279-gated chain_epoch_activate after external confirmation) + all heads (last_seq=0, KAT’d empty-head hashes). Identity + clock are injected into the pure GenesisPlan.

Phase B (idempotent, ANCHOR DB = canopy_security): the genesis manifest (anchor_kind='genesis', anchor_seq=1, previous_anchor_hash = the 64-zero-hex sentinel) is a pure function of installed Phase-A state — rebuilt from re-fetched rows on any rerun; inserted via chain_anchor_append (same key + identical bytes → no-op; different bytes → hard error). Precondition checked with an actionable error: canopy_security’s substrate migration applied.

Phase C: re-fetch everything, independently REBUILD the expected manifest and byte-compare (never merely re-hash stored bytes), and run verify_genesis_state: instances/topology/chain_sources rows present and mutually consistent (exact source, family, instance binding); exact shards 0..N−1; last_seq=0; archived_through=0; routing/version/state values; a UUIDv7 check on the instance id; anchor kind/seq/sentinel/pending; ZERO rows in the live and archive tables. Any mismatch fails the command. (Post-activation reruns hit the "already installed" arm’s validation against genesis-shaped state and fail corruption-shaped — correct: chain-genesis is never legitimate against an activated chain; the message names chain_epoch_activate as the boundary.) Fault-injection tests: kill after A, after B, concurrent installers, corrupt state, target/anchor outage, same-key/different-bytes.

Files: crates/canopy-chain/tests/vectors/ — canonicalizer_rfc8785/ (the official cyberphone testdata + RFC 8785 §3.2.3 + Appendix B samples — INDEPENDENT provenance, documented per file), event_hash_audit.json, event_hash_fti.json, event_hash_ele_grant.json (the closed enum is fully covered NOW; the family’s tables are deferred to #1248 — these vectors pin the ENVELOPE level via a TEST-ONLY raw-payload constructor with a designated placeholder payload; the closed-builder rule governs production construction and #1248 adds the real builder), normalization.json, genesis.json, routing.json, anchor_manifest.json; canopy-signing carries anchor_signature.json (TEST-ONLY key). Chain-specific vectors are seeded by an independent derivation (documented manual/second-implementation computation for a seed subset), then extended by the generator — provenance recorded in each file header.

Adversarial cases pinned: non-BMP + escaped keys; numeric edges (±2^53−1, −0, rejection cases: non-finite/oversize integers); malformed values; UUID version/case; max shard count; strict-JWS negatives (wrong alg/typ/kid); rotation; test-key exclusion.

Generator: a cargo example (--example generate_vectors) that refuses overwrite (no #[ignore] test). Freeze discipline: changing a vector = bumping THAT surface’s protocol version + an ADR-014 amendment. The VECTOR_CORPUS_SHA256 const forces a visible source diff on regeneration — review discipline, not claimed as a mechanical gate.

Property tests (fixed seeds, bounded counts, explicit tolerances; enum coverage via exhaustive match helpers): canonicalization determinism/idempotence over arbitrary payload trees (validation-layer-filtered), hash shape, routing stability + bounded-count full-range reachability, manifest roundtrip byte-stability.

MR-1: cargo build -p canopy-chain, cargo clippy -p canopy-chain --all-targets — -D warnings, cargo nextest run -p canopy-chain; the RFC 8785 official vectors green through the D-CANON stack; the full KAT + coverage + property suites; the canopy-signing vector tests.

MR-2 (EphemeralSchema + devstack; suites live in the three services' tests/ with canopy-chain dev-deps): append happy path; refusals (linkage incl. whole-batch interior, contiguity, epoch not-active, routing-version, watermark, payload exact-key, batch-max, source-registry); rollback consistency (no partial advance); the concurrency category — two parallel same-shard txs serialize with contiguous seq + a single head advance, and different-shard txs proceed in parallel (no global serialization); the append-vs-epoch-close race; the CAS graphs (illegal transitions refused; confirmed immutable; resolution denied to the verify principal); real-fixed-role assertions — exact pg_roles/pg_auth_members/owners/ACLs; owner-bypass (direct DML refused for _app-shaped + real roles; appends via the fns succeed); PUBLIC-EXECUTE negatives; projection minimality (live + archive) against the pinned lists; the production-shaped NON-superuser-migrator scenario + rotated-migrator + concurrent tanf/medicaid migrations on one cluster (shared-db); reconciliation fail-closed (a pre-created conflicting role refuses).

MR-3: the two-phase genesis incl. all three rerun arms, param-mismatch, the fault-injection points, installer-lock contention, both-guard URL validation; the bootstrap ordering test via the injected seam (migration pool first, closed on all paths); SKIP_MIGRATIONS semantics; the full battery per MR (LOCAL cargo xtask validate before each first push); the Antora build renders this plan + ADR Amendment 6.

This MR (plan-commit): this plan + nav.adoc + the parent plan + ADR-014 Amendment 6 + CHANGELOG.adoc.

MR-1: crates/canopy-chain/ (new) + workspace members + Cargo.toml (serde_json_canonicalizer into [workspace.dependencies]); crates/canopy-signing/ (dev-dep + vectors + the strict-header verify function).

MR-2: the three services/*/migrations/<post-20260904>_chain_v2_substrate.sql; crates/canopy-test-lib/src/db.rs (touch); the services' tests/ + their `Cargo.toml`s (canopy-chain dev-deps).

MR-3: xtask/src/cmd/chain_genesis.rs + the apply arm in xtask/src/cmd/migrate.rs + registration + xtask/Cargo.toml (sqlx, canopy-chain, canopy-db); crates/canopy-api/src/bootstrap.rs; docker-compose.yml (one-shot migration services, dormant); docs — configuration-reference.adoc (the new env vars), the xtask catalog, data-models/canopy-{security,tanf,medicaid}.adoc, shared-crates.adoc, security.adoc/security-operations.adoc (runbook incl. the migration job + rotation wrinkle), testing.adoc (the KAT pattern), the services.adoc pointer, CHANGELOG.adoc.