Plan: chain-v2 external anchor authority — enumerable transparency frontier, signed manifests, verifier confirmation (#1278, epic &73)

On this page
IMPORTANT

DEFERRED (2026-08-02, #1294) — superseded by ADR-014 Amendment 11. This v2 design was REJECTED at a second external review: its "independently enumerable transparency frontier" cannot prove completeness or currentness — a delete-then-recreate truncation is invisible to signatures
conditional-create + last_modified alone — and it assumed object_store capabilities the pinned 0.13.2 API does not expose. The corrected architecture is the WORM capability-tier trust model (Amendment 11); the byte-level replacement is chain-v2 anchor authority — WORM-tier. This file is retained UNCHANGED below only as the historical record Amendment 10 references — do NOT implement against it.

NOTE

Child of the chain-v2 rollout plan (#1236); successor to the substrate (#1246), append transport (#1207), and verifiers (#1205/#1206). The contract is ADR-014 Amendment 5 C5 (anchors) as revised through Amendment 9, with the authority/credential selection RATIFIED by Amendment 10 (this plan’s 0b commit — Amendment 6 delegated it explicitly). A contextless implementer reads Amendments 5–10 first; this plan owns the byte-level design the ADR delegates.

Sequencing. #1278 is the last code-side blocker for the #1279 cutover (go/no-go: "authority + production key provisioned") and blocks #1280 (epoch closure). Everything here lands dormant — the emitter deployable and the confirmer sub-step do nothing until their flags + the #1279 credential activation. #1280 (epoch-closure anchors) and #1208/#1247 (archive/purge boundary anchors) REUSE this machinery: submission and storage are kind-agnostic, but the confirm chain-check arm is pinned to genesis+periodic_tip here — the other kinds define their check arms in their own issues (confirm DEFERS an unsupported kind, never latches on it).

Provenance. v1 of this plan (author draft + two internal contextless review rounds, 13+4 findings folded) was REJECTED at external review 2026-08-02 (~1 critical, ~20 high, ~6 material findings — the external authority rooted its trust in mutable local DB state; recovery overwrote evidence; confirm was ID-only/unfenced; receipts unrepresentable; role/LOGIN, Object Lock, IAM, and production delivery under-specified). This v2 changes the architecture: the authority is an INDEPENDENTLY ENUMERABLE transparency frontier, evidence is append-only, confirmation is ordered/linked/fenced/ version-pinned, and the emitter is a separate deployable. Three forks were ratified by the maintainer: (1) separate emitter deployable (not a single-process residual); (2) production account/bucket provisioning is a separate deployment concern, OUT of #1278 (which delivers code + config
behavioral attestation + a provisioning-requirements runbook only); (3) Object Lock retention is a per-jurisdiction, per-family ruleset value (ADR-006/003/011), floor-bounded — not a hardcoded constant. The full finding-disposition appendix lives in the review record; the design below is the post-disposition state.

Status

Step Description Status

0a

Tracker reshape: reconcile #1278 ACs to this design; file the JWKS-widening follow-up (chain-anchor slug not JWKS-servable today); tracker notes on #1279 (carrier-LOGIN invariant, AWS-validation go/no-go item, the one-shot confirm_genesis cutover step before chain_epoch_activate), #1280/#1208/#1247 (their confirm-check arms are out of #1278 scope; #1208/#1247 reuse the retention ruleset key); epic &73 note; claim #1278.

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

0b

Plan-commit (docs-only, direct to main after battery): this plan + nav (Active) + ADR-014 Amendment 10 + CHANGELOG. The mandated independent contextless reviews ran against this exact artifact pre-commit — four rounds (10 → 11 → 4 → 2 findings, all folded; round 4 converged with only trivial text fixes). The maintainer’s confirmatory pass (0c) runs against the committed artifact before implementation.

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

1

MR-1 store hardening + devstack (feature/1278-store-hardening, Relates to #1278): canopy-store put_create/get_version/list_prefix/PutReceipt/bounded-streaming read/checksum header/.with_endpoint none-fix + typed errors; xtask Garage bucket/split-key provisioning; the empirical Garage v2.2.0 conditional-write + versioning + list + Object-Lock probes; devstack persistence (named Garage volume, chain-anchor key wiring, recreate-and-boot E2E).

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

2

MR-2 crates (feature/1278-anchor-crates, Relates to #1278): RFC 7638 thumbprint + anchor_kid + ANCHOR_SIGNING_VERSION→2 + vector regeneration + expanded KATs (canopy-signing); AnchorObjectKey + the anchor-envelope codec (size-capped, proptest, KAT) in charter-pure canopy-chain (NO async); the NEW canopy-anchor-authority crate — ExternalAnchorAuthority trait + AuthorityError + StoreAnchorAuthority impl + the moved DTOs + the shared env-parameterized ChainAnchorConfig (+ parse test).

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

3

MR-3 substrate (feature/1278-anchor-substrate, Relates to #1278): the three migrations (attempts table + recovery journal + recovery_token + split receipt columns + emit reshape + confirm CAS reshape + anchor_integrity family-global + chain_anchor_recovery_rotate + emitter grants incl. the security-DB read set + safe key registration + the registered_at-based kid window) + the per-family retention ruleset key; VerifyReject::AnchorIntegrity. Schema only — dormant. (ChainAnchorConfig itself is MR-2, in the authority crate; its capability-matrix VALIDATION is consumed when the deployables are built, MR-4/MR-5.)

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

4

MR-4 emitter deployable (feature/1278-anchor-emitter, Relates to #1278): the canopy-anchor-emitter service (writer authority, _probe/ attestation, key load + boot-barrier registration, the per-family emit/submission loop, attempt recording, adopt/divergent classifier, recovery-rotate wiring, bounded retry) + its Dockerfile/CI/compose wiring, dormant.

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

5

MR-5 confirmer + status + amendment-realization (feature/1278-anchor-confirm, Closes #1278): the confirmer sub-step in canopy-security’s verifier (enumerate-frontier, ordered/linked/fenced/version-pinned confirm, rollback detection, failed-anchor evidence path) + the anchor census + the coverage-lag status input + manifest_coverage_lagging + max-age tightening + the family-run ok external gate; the attestation harness + security-operations provisioning-requirements runbook + rotation/recovery runbooks; docs.

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

6

Post-merge: close #1278 with the closing comment; parent + epic bookkeeping; move this plan to Archive + fix nav/xrefs; #1279 go/no-go item recorded.

Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294)

Epic: &73
Issue: #1278 (critical, w5) — blocked by #1246 (Done); blocks #1279, #1280
Branches: this docs MR, then feature/1278-store-hardening-anchor-crates-anchor-substrate-anchor-emitter-anchor-confirm
Discipline: local cargo xtask validate before each MR’s first push; every MR dormant-safe; Relates to #1278 on MR-1..4, Closes #1278 on MR-5 (the >1-MR justification is the five independently-reviewable subsystems: store primitives, pure crates, substrate, the new deployable, the confirmer).

Context (recon facts — verified file:line)

  • The substrate is landed and dormant. chain_anchors + chain_anchor_heads exist in canopy_security ONLY, all families (services/canopy-security/migrations/20260910000000_chain_v2_substrate.sql:253-287): anchor_kind ∈ {genesis, periodic_tip, epoch_closure, archive_watermark, purge_boundary}; notarization_state ∈ {pending, submitted, confirmed, failed} default pending; UNIQUE (instance, family, anchor_seq); CHECK (state='pending' OR (jws NOT NULL AND kid NOT NULL)); a single nullable external_version column; verifier_confirmed_at; created_at. chain_anchor_append (:623-688) locks the head FOR UPDATE, enforces seq-1-with-zero-sentinel genesis, contiguity, and previous-hash linkage, and is same-bytes idempotent.

  • The emit/confirm arm split is landed + grant-negative tested. Hardening (20261010000000_chain_verification_hardening.sql:596-654): chain_anchor_transition_emit(id, to, jws, kid, external_ref, external_version) covers pending→submitted|failed, submitted→failed, failed→submitted, refuses confirmed, COALESCE-sets columns; chain_anchor_transition_confirm(id) allows only submitted→confirmed and touches no caller columns. canopy_chain_anchor_emitter (NOLOGIN) has EXECUTE on append+emit + SELECT on chain_anchors/chain_anchor_heads (:707-713); canopy_chain_verify has EXECUTE on confirm (:711). Negatives pinned at chain_v2_substrate_test.rs:1236-1266, chain_verifier_host_test.rs grant matrix.

  • The emitter role reconcile has NO carrier allowlist AND raises on LOGIN (hardening.sql:47-55 raises if the role has rolcanlogin; :66-75 raises on any member NOT LIKE 'test\_%'). It re-runs on every EphemeralSchema replay (roles are cluster-global). Contrast the verify side, which allowlists canopy_security_verify as a member (substrate.sql:104) and flips it LOGIN operationally at #1279 (never in the migration set). D9 resolves the emitter model against this precedent.

  • chain_incident_resolve is immutable and epoch-loose (hardening.sql:543-591): it requires a MANUAL ok run of the incident’s detected_loop_kind, same instance/family/shard, finished after detection — but does NOT check epoch. chain_incident_latch dedups on (instance, family, epoch, shard, kind, detected_loop) with NULLS NOT DISTINCT, so a NULL-epoch family-global incident dedups correctly. D6/D8 use both facts.

  • canopy-chain has the manifest + the unconsumed DTOs. AnchorManifest (RFC 8785, 13-key preimage, from_canonical_bytes re-encode byte-equality, src/anchor.rs:85-207), AnchorSubmission{manifest_bytes, jws} + AnchorReceipt{authority_id, external_ref, version_or_etag, stored_at} (:320-342, ZERO consumers), DOMAIN_TAG_ANCHOR/ANCHOR_MANIFEST_VERSION=1/ANCHOR_SIGNING_VERSION=1 (src/versions.rs). KATs for all five kinds (tests/vectors/anchor_manifest.json).

  • canopy-signing has the strict JWS surface, unwired. ANCHOR_JWS_TYP="canopy-chain-anchor+jws"; sign_anchor_detached; verify_anchor_jws(vk, expected_kid, payload, jws) (strict alg/typ/kid, deny_unknown_fields, RFC 6979 deterministic, src/anchor.rs). Vectors declared "provisional_until": "#1278…" (tests/vectors/anchor_signature.json, test kid TEST-ONLY-anchor-key). The determination kid scheme (derive_kid = canopy-{program}-{16 hex of SHA-256(PEM text)}, src/signer.rs:96-104) is UNSUITABLE for anchors; JWK x/y helpers exist (src/jwk.rs); no RFC 7638 code anywhere.

  • canopy-store is overwrite-capable and drops receipts. Store::put is #[deprecated] and discards PutResult (src/store.rs:92-97); object_store 0.13.2 supports PutMode::Create (AWS→If-None-Match) + PutResult{e_tag, version} + GetOptions; from_config unconditionally calls .with_endpoint(&config.s3_endpoint) even when empty (:57) and sets no checksum header; get buffers the whole body (:142); enforce_backend fails closed to s3 outside CANOPY_ENV=development. Consumers: canopy-applications, canopy-notices. Garage devstack dxflrs/garage:v2.2.0, region garage, API :3900 / admin :3903 (token canopy-admin-token), ports in .ports.env; NO bucket/key provisioning automation; NO named Garage data volume; canopy-security mounts /tmp as tmpfs.

  • The verifier host is family-leased. services/canopy-security/src/chain_verify/host.rs: fetch_topology → FAMILY lease → halt gate → jobs → family_manifest_step (:438-475, calls engine manifest_check which fetches the LATEST CONFIRMED anchor and never verifies JWS or touches external storage) + census → per-shard tail/scrub under a global visit budget. verifier_pass returns idle on TopologyStatus::NotActive (:104) — fetch_topology’s `NotActive arm carries no instance/epoch/shard_count/source (crates/canopy-common/src/chain_append.rs:306-371). No production code calls append/emit/confirm — only tests + xtask chain-genesis (which appends the pending seq-1 genesis anchor). Per-family verifier tasks spawn on flag+URL (main.rs:181-247).

  • Retention is policy data, per ADR-006. Federal floors are jurisdiction-overridable inputs in rulesets/{jurisdiction}/jurisdiction.toml (+ a sibling citations.toml per ADR-011); services load scalar policy values from the ruleset at boot (e.g. services/canopy-tanf/src/params.rs, medicaid [shared.timing]). canopy-security reads no jurisdiction config today. No retention key exists in any ruleset yet. C7 (#1208/#1247) already mandates "per-jurisdiction ruleset values, per family, legal-hold aware, Pub 1075 §4 floor."

D0 — the load-bearing principle: an EXTERNAL, ENUMERABLE transparency frontier

The purpose of an external authority is to be a root of trust the local-DB attacker cannot roll back. The rejected v1 failed this: every decision (which object to read, which anchor is newest, what time it is) came from mutable local rows, so a coherent local-snapshot rollback made the verifier fetch an old, valid, Object-Locked object and never look for the newer ones. This plan makes the authority itself the high-water mark, discovered by enumeration, never inferred from local state.

  1. Reader ENUMERATES the authority. The confirmer and census list {prefix}/{instance}/{family}/ (prefix-scoped ListBucket) to learn the true set of anchor objects that exist externally — the maximum recovery_token namespace(s) and the maximum anchor_seq present — independent of any local row. Enumeration is a first-class reader capability (and it distinguishes 404 from 403: without ListBucket an absent object under a deny-by-default policy returns 403, making "missing vs denied" undecidable).

  2. Rollback is a breach. If the external frontier is AHEAD of local confirmed state (objects exist for (token, seq) the local DB has no confirmed — or no any — anchor row for), the local DB was rolled back or an anchor suppressed. That latches anchor_integrity (family-global). The store is append-only (conditional create + deny delete/overwrite + Object Lock), so "external ahead of local" is unforgeable.

  3. The anti-rollback defense is the confirmer’s enumeration + an independent auditor — NOT read-time status. Status is served from LOCAL state (a pure derive_status over precomputed inputs), so NO read-time local signal can resist a full privileged-DB attacker: they can SET any freshness stamp to now() by plain DML AND freeze the confirmer, keeping status green. Be honest about this. The genuine, unforgeable guarantees are EVENTUAL and require an honest observer: (i) any honest confirmer pass re-enumerates the authority (list) and latches anchor_integrity on external-ahead-of-local (D0.2) or on a live-vs-stored authority_time discrepancy; (ii) an independent auditor with only the bucket + the public key reconstructs and verifies the whole anchor chain out-of-band, regardless of what canopy’s local status says (the Kerckhoffs root of trust). Against a fully-compromised DB whose confirmer is permanently frozen, local status is by construction untrustworthy — which is exactly why the external authority + the external audit, not the status endpoint, ARE the trust anchor. Read-time status freshness is therefore BEST-EFFORT liveness only: the confirmer stamps a live-observed authority time each pass (D6/D8), status ages off it, and a frozen frontier or a dead confirmer ages to stale in the honest case — but it is not claimed to defeat the privileged rollback (that is (i)/(ii)'s job). The emitter being a separate deployable means a privileged-DB attacker holds no writer S3 credential and cannot mint a fresh object version (and a new version over a recorded one is itself a version-pinned breach, D6) — so the confirmer’s live enumeration reads the true aging frontier.

  4. The object carries the order it asserts. The envelope embeds the full manifest (binding anchor_seq, previous_anchor_hash, the C1 identity, and per-shard tips), so an independent auditor holding only the bucket + the public key reconstructs and verifies the entire anchor chain from genesis with zero database access. Local state is a cache of this, never the source.

Every D-section below serves D0.

D1 — canopy-store append-only hardening (MR-1)

New primitives on Store (crates/canopy-store/src/store.rs); the deprecated put stays for the #435 upload seam — put_create is a DISTINCT integrity seam for machine-generated canonical evidence (codec-validated; magic-byte sniffing would reject JSON), documented as such.

  • pub struct PutReceipt { pub version_id: Option<String>, pub etag: Option<String>, pub authority_time: Option<DateTime<Utc>> } — the three receipt facts SPLIT (the DTO version_or_etag collapse was a rejection finding). authority_time from the store’s returned/last-modified metadata.

  • put_create(path, data) → Result<PutReceipt, StoreError> via put_opts(PutMode::Create) (AWS→If-None-Match: *; LocalFileSystem native), setting the SDK checksum header Object-Lock uploads require. Maps object_store::Error::AlreadyExists → StoreError::AlreadyExists{path}.

  • get_version(path, version_id) → Result<(Bytes, PutReceipt), StoreError> via GetOptions{version, ..} — version-pinned read (fetch the RECORDED version, not "current"), so a new current version or a delete marker over the recorded version is DETECTABLE even when bytes match.

  • head(path) → Result<ObjectMeta, StoreError> and get_capped(path, max_bytes) → Result<(Bytes, PutReceipt), StoreError> — metadata-first size check + a streaming read hard-capped at max_bytes (the current get buffers the whole body; a preplanted large object must not exhaust memory).

  • list_prefix(prefix) → impl Stream<Item = Result<ObjectMeta, StoreError>> — prefix-scoped enumeration (D0.1); reuses the #1215 list_keys_stream shape.

  • StoreError gains AlreadyExists{path}, PermissionDenied{path}, NotFound{path}, TooLarge{path, size, cap}, VersionGone{path, version} (a new current version/delete-marker over a recorded version); From<StoreError> for ApiError arms (409/403/404/413/409).

  • from_config (:46-77): pass s3_endpoint as None when empty (native AWS must not receive .with_endpoint("")); the anchor authority constructs ObjectStoreConfig values programmatically (its own namespace, D11) — the two trust domains never share env or bucket.

Devstack (MR-1): a named Garage data volume (survives container recreation — today Postgres persists but Garage does not, so a recreate can keep confirmed rows while deleting their objects); chain-anchor key generation + mount wiring for the emitter (the manual generator rejects that name today, xtask/src/cmd/gen_signing_keys.rs; the auto-generator omits it); xtask Garage provisioning (idempotent, via the admin API :3903: bucket canopy-chain-anchors, writer/reader/enumerator key pairs); a recreate-and-boot E2E proving the authority survives.

Empirical gates (MR-1, documented in-test + local-dev.adoc): Garage v2.2.0 (a) conditional-write (put_create twice → AlreadyExists?), (b) versioning, (c) list, (d) whether object_store 0.13.2 exposes per-object Object-Lock retain-until on put_opts. Outcomes pin D10’s enforcement mechanism and whether the devstack anchor store is Garage-s3 or the Local backend (native PutMode::Create) with Garage reachability-only. No fake attestation either way.

D2 — pure primitives (canopy-chain) + the async authority crate (MR-2)

Crate placement (a rejection finding — canopy-chain is charter-pure "bytes-only, no async, no db, no signing", crates/canopy-chain/src/lib.rs:3-14). The split:

  • canopy-chain (pure, no new deps): AnchorObjectKey (a value type + its object_path string derivation) and the envelope codec — no I/O, no async.

  • canopy-anchor-authority (NEW async crate; deps canopy-chain + canopy-store + async-trait + chrono
    serde):
    the ExternalAnchorAuthority trait, AuthorityError, the I/O value types (StoredAnchor, AnchorObjectMeta, the reshaped AnchorReceipt, AnchorSubmission — moved here from canopy-chain), the concrete StoreAnchorAuthority impl over canopy_store::Store (writer + reader constructions), and the shared env-parameterized ChainAnchorConfig derivation (D11). It needs NO sqlx/p256 (no DB pool, no key parsing) — AnchorKeyStore + ConfirmDeps are CONFIRMER-only and live in canopy-security (which already has sqlx + canopy-signing), so the authority crate stays minimal. BOTH deployables (the emitter, MR-4; canopy-security’s confirmer, MR-5) depend on THIS crate — never on each other — so the process arm split holds with no shared service-lib dependency.

// canopy-chain (pure):
pub struct AnchorObjectKey { pub instance: ChainInstanceId, pub family: ChainFamily,
    pub recovery_token: Uuid /* UNPREDICTABLE namespace, D4 */, pub anchor_seq: i64 }
impl AnchorObjectKey {
    /// {prefix}/{instance}/{family}/{recovery_token}/{anchor_seq:020}.json
    /// 20-digit zero-pad ⇒ lexicographic == numeric across the full 2^53-1 domain.
    pub fn object_path(&self, prefix: &str) -> String { /* … */ }
}

// canopy-anchor-authority (async):
#[async_trait]
pub trait ExternalAnchorAuthority: Send + Sync {
    fn authority_id(&self) -> &str;
    async fn store(&self, key: &AnchorObjectKey, sub: &AnchorSubmission) -> Result<AnchorReceipt, AuthorityError>;
    async fn get(&self, key: &AnchorObjectKey, version_id: Option<&str>) -> Result<StoredAnchor, AuthorityError>;
    async fn list(&self, instance: ChainInstanceId, family: ChainFamily) -> Result<Vec<AnchorObjectMeta>, AuthorityError>;
}
pub struct StoredAnchor { pub bytes: Vec<u8>, pub receipt: AnchorReceipt }
pub struct AnchorObjectMeta { pub recovery_token: Uuid, pub anchor_seq: i64, pub version_id: Option<String>, pub authority_time: Option<DateTime<Utc>>, pub size: u64 }
pub enum AuthorityError { AlreadyExists, NotFound, PermissionDenied(String), Transient(String), VersionGone, Invalid(String) }
pub struct StoreAnchorAuthority { /* canopy_store::Store + prefix + authority_id */ }  // the concrete impl

The concrete StoreAnchorAuthority maps AnchorObjectKey::object_path → Store::put_create/get_version/ list_prefix, PutReceipt → AnchorReceipt, StoreError → AuthorityError (incl. VersionGone → VersionGone); store()/get() set the metadata-first size cap (D1). AnchorReceipt = { authority_id, external_ref, version_id, etag, authority_time } (the version_or_etag/stored_at collapse removed). The DTOs AnchorSubmission/AnchorReceipt MOVE from canopy-chain to canopy-anchor-authority (they are I/O contracts, not pure bytes); the substrate plan’s "DTOs in canopy-chain" note is superseded (recorded in the CHANGELOG Changed). The trait is &dyn-safe so #1280 reuses it.

Envelope codec (canopy-chain src/anchor.rs, ANCHOR_ENVELOPE_VERSION=1): encode_anchor_envelope(manifest_bytes, jws) → Result<Vec<u8>> builds {"anchor_envelope_version":1,"jws":…,"manifest":<value>}, canonical_bytes, and enforces the post-condition that re-extracting + re-canonicalizing the manifest reproduces the input bytes; decode_anchor_envelope(bytes, max_len) → Result<(Vec<u8>, String)> strict 3-key decode + version pin
full re-encode byte-equality + size cap. Proptest (parser/serializer rule) + a frozen KAT.

D3 — anchor-key identity (MR-2, canopy-signing) + safe registration/retirement (MR-3)

  • jwk.rs: p256_jwk_thumbprint(public_key_pem) → Result<String, SigningError> — RFC 7638 §3 exactly ({"crv":"P-256","kty":"EC","x":…,"y":…}, lexicographic members, no whitespace, b64url-unpadded coords via public_pem_to_xy), SHA-256, base64url-unpadded. anchor.rs: ANCHOR_KID_PREFIX="canopy-chain-anchor:", anchor_kid(pem) = prefix + thumbprint.

  • Header freeze + version bump. Freezing the provisional vectors is a change to frozen vectors under Amendment 6’s rule, so ANCHOR_SIGNING_VERSION2; vectors regenerate at v2 under anchor_kid(TEST_ONLY pubkey) (delete + regenerate via the no-overwrite generator — the deliberate act recorded in the MR); drop provisional_until. `verify_anchor_jws’s exact-kid check is unchanged. KATs additionally cover coordinate encoding, leading-zero x/y, PEM normalization, wrong-curve/malformed-key refusal, and the retired-key-vs-new-anchor rule. Amendment 10 pins whether the protected-header BYTES are normative (member order/encoding) or only the semantic fields.

  • Retention in signing_key_history (MR-3). Registration becomes SELECT-and-compare, not ON CONFLICT DO NOTHING: an existing kid whose stored program or normalized public key disagrees with the recomputed RFC 7638 kid is a hard error, never silently accepted. Retirement is a TIME window over the EXISTING signing_key_history.registered_at, checked against the anchor object’s authority LastModified — NO new column, no per-seq formula (a seq window is wrong here: one anchor key signs ALL THREE families whose anchor_seq spaces each restart at 1, and a GREATEST(head+1,…) formula can never cover genesis seq 1 — the round-2 seq approach had both bugs). One anchor key is current at a time (the emitter holds one CHAIN_ANCHOR_SIGNING_KEY); rotation registers a successor. resolve_kid_for_object_time(kid, object_authority_time) (the AnchorKeyStore method) accepts kid iff it is a registered chain-anchor key AND the object’s authority LastModified falls in [kid.registered_at, successor.registered_at) (current kid: to infinity), tie-broken on (registered_at, kid). This is (i) genesis-safe (no seq), (ii) cross-family-correct (registration time is global), and (iii) non-forgeable by the privileged-DB attacker: the object’s LastModified is authority-set (Object Lock + no writer credential for the DB attacker), so a leaked RETIRED key cannot be used to confirm a NEW anchor — its object would carry a post-retirement LastModified outside the key’s window. Old keys stay VERIFIABLE for their own era (history). Residual, documented: forging a confirmable anchor at all still requires the WRITER credential (a separate deployable, Object-Locked), so a retired-key compromise adds no attack surface beyond a writer-credential compromise, which the recovery runbook contains by rotating the writer credential first. The reserved program slug is chain-anchor (JWKS-servable-slug widening is a filed follow-up — the Program enum does not admit it today).

  • The boot barrier is emitter-side and split from the confirmer (a rejection finding). Registration is NOT the existing fire-and-forget HTTP path (signing_registration.rs, which is best-effort, Program-enum-gated, and re-derives the determination kid). Instead the EMITTER deployable, at boot, performs a BLOCKING direct signing_key_history SELECT-and-compare + INSERT of its RFC 7638 anchor key (it holds the INSERT grant, D4; emitter tasks do not start until it succeeds). The CONFIRMER (canopy-security) has no signing key and cannot INSERT — its "barrier" is a SELECT-present check: if the anchor key is absent it cannot verify JWS, so confirmation stays dormant (anchors remain submitted → age to stale), never a crash. The INSERT grant survives #1279’s identity restriction.

D4 — schema (MR-3): one canopy-security migration + tanf/medicaid twins

DDL follows the substrate discipline (DO/EXECUTE format() with SET search_path = <schema>, pg_temp; create-then-transfer to canopy_chain_owner_security; REVOKE PUBLIC; overload rule = DROP before any signature change). Timestamp AFTER 20261015000000 (security) / AFTER 20261010000000 (tanf/medicaid — their fti_preimage_id postdates the substrate); crates/canopy-test-lib/src/db.rs is re-touched so sqlx::migrate! re-embeds.

Append-only attempts (the recovery-overwrites-evidence fix). Mutable per-anchor generation/ref columns would let failed(token_g0) → rotate → submitted(g1) erase the planted g0 object before the verifier sees it. Instead:

CREATE TABLE chain_anchor_attempts (
    id                UUID PRIMARY KEY,
    anchor_id         UUID NOT NULL REFERENCES chain_anchors(id),
    anchor_seq        BIGINT NOT NULL,        -- DENORMALIZED from chain_anchors; the emit fn (which
                                              -- locks the anchor row) validates it equals the anchor's
                                              -- seq. A cross-table CHECK cannot express this in PG.
    recovery_token    UUID NOT NULL,
    external_ref      TEXT NOT NULL,          -- the canonical object path this attempt used
    external_version_id TEXT,                 -- non-null required in production (enforced in the fn)
    external_etag     TEXT,
    external_authority TEXT NOT NULL,
    authority_time    TIMESTAMPTZ,            -- the store's LiveModified as observed AT STORE TIME
                                              -- (recorded for audit; freshness reads the LIVE value, D0.3)
    submitted_kid     TEXT NOT NULL,
    submitted_jws     TEXT NOT NULL,
    outcome           TEXT NOT NULL CHECK (outcome IN ('stored','adopted','divergent','error')),
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE (recovery_token, anchor_seq)       -- one attempt per external key (token+seq fixes the object)
);
-- append-only guard (statement-level trigger, the signing_key_history precedent)

chain_anchors gains current_attempt_id UUID REFERENCES chain_anchor_attempts(id) (the confirmed/latest attempt pointer). The confirmed/failed ⇒ outcome and production ⇒ version_id invariants are enforced IN the SECURITY DEFINER chain_anchor_transition_emit/_confirm bodies (which already lock and read both rows) — NOT as table CHECKs, because a CHECK cannot reference another table and "production" is not row data (a rejection finding). The single-table CHECK that DOES hold: `outcome’s closed set (above). No nullable/COALESCE "maybe-stamped" columns — every attempt is a full row.

chain_anchor_heads gains recovery_token UUID NOT NULL DEFAULT gen_random_uuid() — the current key namespace (unpredictable; a leaked writer cannot preplant the next integer key).

No new checkpoint columns. Read-time status-age keeps using the LANDED path — chain_anchor_trusted_vchain_anchors.created_at (read_trusted_manifest, status.rs:441-459), unchanged — and is honestly labelled BEST-EFFORT liveness (D0.3/D8): a live-restamped authority_observed_at on the family checkpoint row was considered and REJECTED because the family row is CHECK-forbidden from carrying cursors/stamps (Amendment 9’s family-shape invariant) and the verify role cannot write checkpoints except via chain_checkpoint_advance — adding a stamp there would fight the substrate for a signal that is best-effort by construction (no read-time local signal defeats the privileged rollback; the enumeration latch does). The anti-rollback guarantee is entirely the confirmer’s external-ahead-of-local enumeration
the independent auditor (D0).

Recovery rotation replaces the naive "generation bump":

CREATE FUNCTION chain_anchor_recovery_rotate(
    p_instance uuid, p_family text, p_expected_token uuid, p_operation_id uuid,
    p_reason text, p_incident_id uuid
) RETURNS uuid  -- the new token (or the already-applied one, replayed)
-- locks the head (the head row is guaranteed present — genesis pre-created it); the p_operation_id
-- lookup runs UNDER the head lock; if already journaled → replay its result (idempotent);
-- refuse if p_expected_token <> current (concurrent/stale-retry fence); else mint a fresh
-- random token, journal (from/to token, operation_id, actor=session_user, reason, incident_id,
-- bumped_at) into chain_anchor_recovery_journal, advance the head token.
-- EXECUTE → canopy_chain_incident_admin (a recovery action under incident authority).

chain_anchor_recovery_journal carries UNIQUE (operation_id) — replay is schema-enforced, not merely lock-ordering discipline (a concurrency test asserts two same-operation_id rotations produce exactly one journal row).

A stale old-token store result arriving after rotation cannot mutate the anchor row: its attempt carries the old token; the head’s current token is the new one; the burned (old_token, seq) object is still discoverable via its attempt row (the evidence survives). Precondition (runbook, not machine-checkable): the compromised writer credential is rotated FIRST — a token rotation alone does not contain a leaked writer.

Emit reshapechain_anchor_transition_emit grows p_recovery_token, p_external_authority, p_external_version_id, p_external_etag, p_authority_time, p_attempt_outcome, p_anchor_seq (the denormalized value, validated against the locked anchor row), and p_require_version boolean (the emitter derives it from CANOPY_ENV — a SECURITY DEFINER fn cannot read the env, and "production" is not row data). It INSERTs a new chain_anchor_attempts row + advances current_attempt_id on EVERY store-attempt transition — → failed (divergence) included — enforcing IN THE BODY: p_attempt_outcome IN ('stored','adopted') AND p_require_version ⇒ p_external_version_id NOT NULL (rejects a version-less production store); the confirmed/failed ⇒ outcome half is genuine row data. Per-attempt exact assignment, no COALESCE.

Confirm CAS reshapechain_anchor_transition_confirm grows (p_lease_token, p_expected_predecessor_id, p_expected_predecessor_hash, p_checked_fingerprint). TWO fence arms (the confirmed-immutable rule holds in both):

  • Steady (periodic_tip): commits only if the FAMILY LEASE token (p_lease_token, validated the landed chain_run_record way — PERFORM 1 FROM chain_verification_checkpoints WHERE …shard_id=0 AND loop_kind='family' AND lease_token=p_lease_token) still holds AND the predecessor + the checked fingerprint (manifest_hash, external_ref, external_authority, version_id, kid, recovery_token) still match — a stale verifier that lost its lease during the (non-transactional, bounded) S3 I/O cannot confirm.

  • Genesis (a rejection finding — no family pass/lease exists during installing): for anchor_kind='genesis' (seq 1) the fn accepts p_lease_token = NULL and instead STRUCTURALLY fences on the epoch row being installing (a state reachable only pre-activation, and the #1279 cutover is a single-actor coordinated operation — there is no concurrency to fence). It still requires the checked fingerprint match. confirm_genesis (D6) calls this arm; a test pins that it succeeds during installing and is refused once the epoch is active (steady genesis re-confirm is impossible).

Incident kindanchor_integrity (23rd) joins the chain_incidents kind CHECK; latched family-global (epoch NULL, shard NULL — dedups via NULLS NOT DISTINCT), so a per-epoch family run cannot clear it. VerifyReject::AnchorIntegrity{anchor_seq, recovery_token, external_ref, version_id, kid, phase, expected, got} (canopy-common); evidence carries anchor_seq (NEVER top-level seq — the _app projection reads evidence→>'seq' as an event position; position_seq stays NULL for this kind).

Grants — scoped PER DATABASE (a rejection finding: signing_key_history, chain_anchor_attempts, and the recovery journal exist ONLY in canopy-security; anchors are security-only; a literal all-three twin would fail relation does not exist). These are TABLE privileges, not role memberships, so none collide with the hardening membership reconcile.

  • All three DBs (security + tanf + medicaid — the fti families' fetch_topology + capture_heads reads): canopy_chain_anchor_emitter gains CONNECT + schema USAGE + SELECT on chain_topology, chain_epochs, chain_sources, chain_heads. The tanf/medicaid substrates do NOT create canopy_chain_anchor_emitter (only the security hardening migration did), so each twin migration first CREATEs it NOLOGIN under the substrate reconcile discipline (idempotent dual-SQLSTATE, fail-closed on privileged attributes / non-test_% members), then grants.

  • canopy-security only: canopy_chain_anchor_emitter gains INSERT + SELECT on signing_key_history (the boot-barrier registration, D3/D5, and the adopt-classifier’s old-kid JWS verify); canopy_chain_verify gains SELECT on signing_key_history + chain_anchor_attempts + chain_anchor_recovery_journal.

Grant-negative (PUBLIC-probe) tests both directions in each DB.

D5 — the emitter deployable canopy-anchor-emitter (MR-4)

A NEW service (services/canopy-anchor-emitter/, its own binary/Dockerfile/CI/compose entry) — the process-level arm split (a compromise of this process holds the signing key + writer creds but CANNOT confirm; a compromise of canopy-security’s confirmer holds reader creds but no signing key/writer creds). It loads rulesets/{jurisdiction}/jurisdiction.toml at boot (D10 retention), registers the anchor public key (boot barrier, D3), and runs one task per family (audit via the security DB; fti-tanf/fti-medicaid via their DBs — X10 per-family dormancy on flag+URL). All functions ≤40 lines (decompose like host.rs).

run_emitter_loop(target, cfg, security_pool, chain_pool, writer, key, retention):
  sleep(first_tick_delay); attested=false; backoff=250ms
  loop:
    if !attested:
      attest(writer, reader, CANOPY_ENV)?          # D12 probes; Err → error!, backoff, continue
      attested=true
    emitter_pass(...)  → Ok: backoff=250ms | Err: warn!, backoff=min(2*backoff, emit_interval)
    sleep(tick_ms)

emitter_pass(target, cfg, security_pool, chain_pool, writer, key, retention):
  # 1. topology (installing OK — reads chain_topology/chain_sources DIRECTLY, not fetch_topology,
  #    so genesis SUBMITS while the epoch is `installing`; genesis CONFIRMATION is the cutover's
  #    one-shot confirm_genesis step, D6)
  topo = read_topology_any_state(chain_pool, target.family)?  ; if absent → Idle
  if target.fti_source? != topo.source → Err(SlotMismatch)   # swapped URL fails loudly
  scope = VerifyScope::from(topo)

  # 2. EMISSION (cadence; single-writer via advisory lock + in-tx recheck)
  tx = security_pool.begin(); SET LOCAL statement_timeout = store_timeout
  if pg_try_advisory_xact_lock(key(instance, family)):
    if newest_anchor_authority_age(instance, family) >= emit_interval AND topo.state == active:
      heads = capture_heads(chain_pool, scope); m = AnchorManifest{PeriodicTip, seq=last+1, prev=last_hash, heads…}
      chain_anchor_append(instance, family, epoch, last+1, 'periodic_tip', m.bytes, m.hash, last_hash)
    tx.commit else tx.rollback   # loser skips; append contiguity RAISE is the benign backstop

  # 3. SUBMISSION (oldest-first; the xtask genesis PENDING row rides this path unmodified)
  for row in select pending|failed order by anchor_seq limit submit_batch:
    verify local manifest_hash == sha256(row.manifest_bytes)  else → CRITICAL, Halt
    jws = sign_anchor_detached(key, row.manifest_bytes)?      # sign err → stays pending (CHECK), Halt
    okey = AnchorObjectKey{instance, family, head.recovery_token, row.anchor_seq}
    match timeout(store_timeout, writer.store(okey, {envelope, jws}, retention_for(family))):
      Ok(rcpt)          → emit(row, submitted, jws, key.kid, okey.path, rcpt…, outcome='stored')
      AlreadyExists     → stored = writer.get(okey, None)?    # explicit fetch; SEMANTIC classify:
                          if decode(stored).manifest_bytes == row.manifest_bytes
                             AND verify_anchor_jws(resolve(stored.kid), stored.kid, row.manifest_bytes, stored.jws):
                               emit(row, submitted, stored.jws, stored.kid, okey.path, stored.rcpt…, 'adopted')  # crash-recovery
                          else emit(row, failed, jws, key.kid, okey.path, rcpt…, 'divergent'); CRITICAL; Halt
      Transient/timeout → warn!; Continue    # unchanged; retry rides emit_interval, not the tick
      PermissionDenied  → error!; attested=false; Halt   # IAM drift → re-probe

Divergence ALWAYS reaches the verifier: the failed row’s current_attempt records the canonical key, and D6’s failed-anchor evidence path fetches it and latches. Failed rows retry oldest-first on the EMIT INTERVAL (capped backoff), not the tick — one bounded probe per interval, not log spam.

D6 — the confirmer (MR-5, in canopy-security’s verifier)

Runs in family_confirm_step, inserted in the family pass BEFORE family_manifest_step (a just-confirmed anchor becomes the trusted ref the same pass). Holds ONLY the reader authority (get/list) + the verify pools — no signing key, no writer creds. ConfirmDeps = None (reader store unconfigured) ⇒ dormant, but an explicit CHAIN_ANCHOR_CONFIRM_ENABLED=true with no reader creds is a STARTUP error (no silent disable).

family_confirm_step(pools, cfg, db, scope, family, family_lease, reader, keys):
  # (A) ROLLBACK DETECTION via enumeration (D0) — the real anti-rollback defense
  ext = reader.list(scope.instance, family)?              # the external frontier, prefix-scoped
  if ext.max_seq > local_max_any_anchor_seq(pools.security, scope, family):
    latch AnchorIntegrity{phase:'frontier', expected:local_max, got:ext.max_seq}; Halt
  # Integrity (not freshness): compare the live LastModified of the newest confirmed anchor's object
  # to its stored attempt.authority_time; a beyond-tolerance discrepancy is a local-rewrite signal.
  if abs(ext.newest.authority_time - stored_attempt_authority_time(...)) > tolerance:
    latch AnchorIntegrity{phase:'authority-time'}; Halt
  # (read-time status-age uses the landed chain_anchors.created_at path — best-effort liveness, D8;
  #  NO checkpoint stamp is written here — the family row is cursor/stamp-free, Amendment 9.)
  # (B) ORDERED, LINKED, FENCED confirm walk (stop at the oldest unresolved anchor)
  prev = local_last_confirmed(pools.security, scope, family)   # id + manifest_hash + tip snapshot
  for row in select submitted order by anchor_seq:            # STRICT order; a gap halts the walk
    if row.anchor_seq != prev.anchor_seq + 1
       or row.previous_anchor_hash != prev.manifest_hash: latch AnchorIntegrity{phase:'linkage'}; Halt
    if kind(row) not in {genesis, periodic_tip}: log defer; Halt   # #1280/#1208/#1247 arms elsewhere
    # (1) continuity, not just tip ordering: chain hash at prev tip seq still == prev tip hash (per shard, same epoch)
    check_continuity(pools.chain, scope, prev.tips)?           else latch{phase:'continuity'}; Halt
    # (2) full anchor-vs-chain check on THIS anchor (engine check_anchor_by_id: strict decode, 7 cols, topology, tip prefix)
    check_anchor_by_id(pools.security, pools.chain, scope, family, row.id)?  else latch (existing manifest kinds); Halt
    # (3) external evidence at the CANONICAL key (row.recovery_token from its attempt), version-pinned
    okey = AnchorObjectKey{scope.instance, family, row.attempt.recovery_token, row.anchor_seq}
    if row.attempt.external_ref != okey.path or row.attempt.external_authority != cfg.authority_id:
        latch{phase:'noncanonical'}; Halt
    match reader.get(okey, row.attempt.external_version_id):
      Transient → warn!; return          # stays submitted; NO latch; age → stale is the SLO
      NotFound|VersionGone → latch{phase:'evidence'}; Halt      # submitted arm: missing/rolled = breach
      Ok(stored) →
        if decode(stored).manifest_bytes != row.manifest_bytes: latch{phase:'bytes'}; Halt
        # kid retirement is a registered_at window checked against the OBJECT's authority time (D3) —
        # cross-family-correct + genesis-safe + non-forgeable by the DB attacker (object time is authority-set)
        vk = keys.resolve_kid_for_object_time(row.kid, stored.receipt.authority_time) else latch{phase:'kid'}; Halt
        verify_anchor_jws(vk, row.kid, row.manifest_bytes, decode(stored).jws)? else latch{phase:'jws'}; Halt
        if row.kid != keys.current_kid(): warn!(rotation visibility)
    # (4) CAS confirm under the family fence (no tx across S3; lease heartbeats during I/O)
    chain_anchor_transition_confirm(row.id, family_lease.token, prev.id, prev.manifest_hash, fingerprint(row))
    prev = row
  # (C) FAILED-ANCHOR evidence path (divergence detection independent of the emitter)
  #     failed ⇒ outcome ∈ {divergent, error} (D4). Both may have a stored object at the attempt key;
  #     an 'error' attempt that never stored yields NotFound (benign — the emitter retries).
  for row in select failed where current_attempt.outcome in ('divergent','error') order by anchor_seq:
    okey = AnchorObjectKey{scope.instance, family, row.attempt.recovery_token, row.anchor_seq}
    match reader.get(okey, row.attempt.external_version_id):
      NotFound → return          # emitter retries; a genuinely absent object is not (yet) evidence
      Ok(stored) → if not semantically_equal(stored, row): latch AnchorIntegrity{phase:'failed-evidence'}; Halt

Engine seam (a rejection finding — the named seam did not match the check). Today fetch_and_check_anchor is PRIVATE, takes only the security pool, and does steps 1–3 (row metadata / topology / constants); the tip-prefix check is check_tip_prefixes (needs the chain pool), composed only by manifest_check. So MR-5 adds a PUBLIC check_anchor_by_id(security, chain, scope, family, id) = fetch_and_check_anchor (id-selected, expected_state='submitted') plus check_tip_prefixes — the both-pools, tip-prefix-inclusive check the confirm walk calls; manifest_check keeps its LatestConfirmed fetch BYTE-IDENTICAL (the 24-test engine suite stays pinned to it), refactored to share the selector AnchorSelector{ LatestConfirmed | ById(id, expected_state) }. check_continuity(chain, scope, prev.tips) is EXPLICIT new MR-5 engine work (reuse cursor_hash_at — assert the current chain hash at each prev tip seq equals prev.tips[shard].last_hash, same epoch/topology), NOT part of check_anchor_by_id. The confirm walk composes: linkage → check_continuitycheck_anchor_by_id → external evidence → CAS.

Confirm wiring + the missing types (a rejection finding). VerifierPools (chain_verify/mod.rs, today {chain, security}) gains confirm: Option<ConfirmDeps>; spawn_family_task’s signature and `verifier_pass → leased_pass → scheduled_pass thread it to family_confirm_step. None ⇒ confirm dormant (anchors age to stale, never a crash); CHAIN_ANCHOR_CONFIRM_ENABLED=true with no reader creds is a STARTUP error (D11). The types ConfirmDeps + AnchorKeyStore are CONFIRMER-only and live in canopy-security (it holds sqlx + canopy-signing; reader: Arc<dyn ExternalAnchorAuthority> is the only authority-crate type they reference) — the authority crate stays DB/key-free:

pub struct ConfirmDeps {
    pub reader: Arc<dyn ExternalAnchorAuthority>,   // reader creds + ListBucket only
    pub keys: AnchorKeyStore,
    pub confirm_batch: usize,
    pub authority_id: String,
    pub key_prefix: String,
}
/// DB-backed over signing_key_history (program 'chain-anchor'); small in-memory cache.
pub struct AnchorKeyStore { /* pool + cache */ }
impl AnchorKeyStore {
    /// The kid valid at an object's authority time: `kid` is a registered chain-anchor key AND
    /// `object_authority_time ∈ [kid.registered_at, successor.registered_at)` (current kid: to ∞),
    /// tie-broken (registered_at, kid) (D3 — uses the EXISTING registered_at; no seq column; genesis-safe,
    /// cross-family-correct, non-forgeable via the authority-set object time). None ⇒ unregistered/out-of-window.
    pub async fn resolve_kid_for_object_time(&self, kid: &str, object_authority_time: DateTime<Utc>) -> Option<VerifyingKey>;
    pub async fn current_kid(&self) -> Option<String>;
}

Genesis confirmation is a one-shot CUTOVER step, not the steady family pass (a rejection finding). The steady family_confirm_step runs inside the leased family pass, which verifier_pass reaches only for an active topology (host.rs:104 returns idle on NotActive, and fetch_topology maps installing → NotActive with no scope). But Amendment 6 requires the genesis anchor VERIFIER-CONFIRMED before chain_epoch_activate, i.e. while the epoch is still installing. Reworking the whole verifier host to run its family pass during installing is invasive and wrong; instead genesis confirmation is a dedicated confirm_genesis(security, chain, reader, keys, scope) entry point that reads chain_topology/ chain_sources DIRECTLY (like the emitter’s read_topology_any_state — a new crates/canopy-common/src/chain_append.rs helper, added to Files Touched), runs the SAME confirm ladder (linkage/continuity/check_anchor_by_id/external-evidence) against the installing-epoch genesis anchor, and calls chain_anchor_transition_confirm’s genesis fence arm (D4: `p_lease_token=NULL, structurally fenced on the epoch being installing) — NOT the family-lease arm, because no family pass/lease/checkpoint row exists during installing (the earlier "runs the SAME confirm and calls confirm" glossed this; the fence source is now explicit). It is invoked as an explicit step of the #1279 cutover sequence (which is already a coordinated operation: quiesce → genesis → notarize → CONFIRM GENESIS → activate → reopen) — genesis confirmation belongs there, before activation. This is not a "manual verify" in the deprecated sense; it is a bounded cutover step. The earlier "genesis-confirm now in-process, no manual path" framing is corrected: steady-state periodic_tip confirmation is in-process (the family pass); genesis confirmation is the cutover’s own step.

D7 — the anchor census (MR-5)

Appended to family_census_step (same cadence stamp). Combines local integrity + the external frontier:

  • No cursor — a bounded full walk each cadence. Anchors accrue ~1 per emit interval (~9k/year/family, so ~88k/decade, at hourly cadence), so even a decade of history is a cheap full walk (hash + JWS re-verify) needing no resume state — which is fortunate, because the family checkpoint row is CHECK-forbidden from carrying a cursor (Amendment 9’s family-shape invariant) and the verify role cannot write checkpoints outside chain_checkpoint_advance (a round-3 finding killed the "durable cursor on the family checkpoint" idea). If a family ever exceeds ~100k anchors (~a decade out), a resume cursor would be a SEPARATE dedicated table + guarded writer, filed then — not the family row.

  • Local: seq contiguity, manifest_hash == sha256(manifest_bytes), previous_anchor_hash linkage, strict decode + seq/kind/epoch cross-check, JWS re-verify (non-pending, kid valid at the object time), head consistency — detects owner-level local rewrites.

  • External: re-list + spot-get (version-pinned, capped) a bounded sample per cadence to detect external-side rewrite/rollback the confirm walk hasn’t re-touched; TooLarge on any object rejects it metadata-first.

  • Epoch-spanning (walks all local anchors, all epochs) — so a manual family ok run genuinely re-verifies old-epoch anchors, which is what makes D8’s resolution gate sound.

D8 — status: coverage freshness, the resolution gate, the max-age tightening (MR-5)

  • Coverage-lag input. A compromised emitter could emit fresh anchors with FROZEN tips forever; age-based staleness alone would stay green while everything above the frozen tips sits outside manifest protection. Status computes max over shards (head.last_seq − trusted-manifest tip.last_seq) (from data status already loads) and degrades to stale via a NEW additive reason manifest_coverage_lagging when it exceeds CHAIN_MANIFEST_MAX_LAG. Reason vocabulary 15→16 — additive wire change (domain enum chain_verify/status.rs, wire enum chain.rs, serde pin, CLI parse pin, service bijection); no path or schema-shape change. An idle chain (equal tips, heads not advancing) does NOT trip it.

  • Age from the LANDED pathread_trusted_manifestchain_anchor_trusted_vchain_anchors.created_at (status.rs:441-459), UNCHANGED. This is BEST-EFFORT liveness (D0.3): in the honest case a frozen frontier / dead confirmer ages to stale; a full privileged-DB attacker who rewrites created_at AND freezes the confirmer is NOT defeated by read-time age — the privileged-rollback defense is the confirmer’s external-ahead-of-local enumeration + the authority-time discrepancy latch (eventual, D6 step A) and the independent auditor, NOT this read-time age. (A live-restamped authority_observed_at was rejected — it fights the family-row-no-cursor CHECK for a best-effort signal.)

  • CHAIN_MANIFEST_MAX_AGE_SECS default 604800 → 21600 (6h), with the relationship ≥ 3 × CHAIN_ANCHOR_EMIT_INTERVAL_SECS (startup error).

  • Resolution gate (review Q). anchor_integrity is family-global; the immutable chain_incident_resolve requires a manual ok family run of the detected loop newer than detection. That is sound ONLY if a family run’s ok outcome is GATED on the external arm: a family run CANNOT report ok while any submitted/failed anchor or latched anchor_integrity exists unless the confirmer’s external checks (confirm walk + census) actually ran and passed this run. A dormant confirmer therefore cannot produce an ok that clears an external breach.

D9 — the emitter role / LOGIN model (MR-3), resolved against the verify precedent

The landed hardening reconcile raises on BOTH a non-test_% member of canopy_chain_anchor_emitter (:66-75) AND the role itself having LOGIN (:47-55), and re-runs every replay — so neither a member-carrier nor a migration-flipped LOGIN works. The verify side proves the only viable model: canopy_security_verify is flipped LOGIN operationally at #1279 (never in the migration set), so every replay runs with it NOLOGIN and the reconcile passes. The emitter follows this exactly: canopy_chain_anchor_emitter is the login identity, flipped LOGIN+password by #1279’s operational credential-activation step, and stays NOLOGIN in all migration runs. Tested (review S’s "test with the role already LOGIN"): apply the full migration set (role NOLOGIN → reconcile green), then operationally ALTER ROLE … LOGIN, then prove the emitter connects + emits + is refused confirm; and document the cutover invariant — the substrate/hardening migrations are not re-applied against a cluster whose carriers are LOGIN (uniform for verify/incident-admin/emitter; a #1279-wide property, recorded on #1279, tested here). Grants added in D4.

D10 — retention: a per-jurisdiction, per-family ruleset value (MR-3)

Per ADR-006, the federal floor is a jurisdiction-overridable input, not a source constant; per C7, retention is per-jurisdiction, per-family. #1278 INTRODUCES the chain-retention ruleset key (which #1208/#1247 reuse — one source of truth): a [chain.retention] table in rulesets/{jurisdiction}/jurisdiction.toml with per-family *_retention_years values + a sibling citations.toml entry (ADR-011), bounded below by the family’s federal floor (Pub 1075 §4 5y for FTI; the applicable floor for audit) — the emitter’s boot load rejects a value below the floor.

Enforcement mechanism is pinned by MR-1’s empirical gate (D1): if object_store 0.13.2 exposes per-object Object-Lock retain-until on put_opts, the emitter sets each object’s retain-until from the ruleset value at store time (application-controlled); otherwise the retain-until is a BUCKET DEFAULT set at provisioning from the same ruleset value (documented in the provisioning-requirements runbook, D13) — the ruleset is the single source either way, and the object time ≥ retention is the immutability window that makes "external ahead of local" durable evidence. Object Lock ENABLEMENT (bucket-level) is provisioning (D13); the retention VALUE is ruleset policy (this section).

D11 — config: the capability matrix (ChainAnchorConfig type + parse: MR-2 in the authority crate; capability-matrix validation consumed MR-4/MR-5)

The parsed chain_anchor namespace + its ObjectStoreConfig derivation live in the shared canopy-anchor-authority crate (D2), env-prefix PARAMETERIZED — so the emitter deployable and canopy-security’s confirmer each construct their own view without either depending on the other’s service lib (a rejection finding: "one struct at SecurityConfig reused by the emitter" would make the emitter compile-depend on canopy-security, undoing the arm split). Validated to capability-specific matrices — every violation a STARTUP error, never a clamp:

  • Emitter capability (emit enabled): writer creds + reader creds (AlreadyExists GET + probes) + signing key + emitter DB URL(s) + bucket + authority_id + jurisdiction (retention).

  • Confirmer capability (CHAIN_ANCHOR_CONFIRM_ENABLED): reader creds + ListBucket + verify pools; NO writer creds, NO signing key. Confirm-on-without-reader-creds fails at startup.

Knobs (all CANOPY_ANCHOR_EMITTER on the emitter deployable / CANOPY_SECURITY for the confirmer side): CHAIN_ANCHOR_EMIT_ENABLED (false) · CHAIN_ANCHOR_CONFIRM_ENABLED (false) · TICK_MS 15000 (250..=60000) · FIRST_TICK_DELAY_SECS 30 (0..=3600) · EMIT_INTERVAL_SECS 3600 (60..=604800) · SUBMIT_BATCH 4 · CONFIRM_BATCH 4 · STORE_TIMEOUT_MS 10000 (1000..=120000) · ANCHOR_DATABASE_URL (+TANF/MEDICAID per-family, RedactedUrl, lazy) · STORE_{BACKEND s3|local(dev-only via CANOPY_ENV), BUCKET, S3_ENDPOINT, S3_REGION, WRITER_ACCESS_KEY/SECRET, READER_ACCESS_KEY/SECRET, ENUMERATOR_*, ALLOW_HTTP (dev-only), KEY_PREFIX (grammar/length), MAX_OBJECT_BYTES} · AUTHORITY_ID (nonempty ≤128) · EXPECTED_AWS_ACCOUNT_OWNER · SIGNING_KEY (canopy-secrets; .keys/chain-anchor-private.pem dev fallback) · CHAIN_MANIFEST_MAX_LAG 1000000 (≥1, documented SLO derivation) · existing CHAIN_MANIFEST_MAX_AGE_SECS → 21600 with the ≥ 3× emit interval rule.

Access keys/secrets use secret wrappers with redacted Debug/serialize (coding-conventions:304). bucket/prefix/authority_id are IMMUTABLE once anchors exist — a persistent authority-registry row (in canopy_security) rejects a silent trust-domain swap at boot.

D12 — attestation (MR-4): behavioral probes, honestly scoped

At the emitter task’s first pass when CANOPY_ENV != development (fail-closed), under the reserved disjoint {prefix}/_probe/ namespace (never a burnable anchor key; stable canary key, lifecycle-bounded):

  1. conditional-create enforced (put_create on the canary → Ok first-ever or AlreadyExists thereafter; a second put_create MUST be AlreadyExists);

  2. unconditional overwrite DENIED — an explicit unconditional PUT MUST be rejected and the original version MUST remain current (the "create twice" check alone proves only create semantics, not overwrite-denial);

  3. writer delete DENIED; reader write DENIED; reader read + list work.

Probe failure ⇒ the emitter goes dormant on capped backoff + error! (anchoring fails closed) WITHOUT killing the process. Control-plane facts (Object Lock retention/legal-hold, versioning, the bucket policy, split IAM) are NOT reachable through object_store’s data plane — they are provisioning-time operator attestation (D13). No fake attestation.

D13 — production provisioning is a SEPARATE deployment concern (OUT of #1278)

Ratified: #1278 does NOT ship production IaC or perform account/bucket/credential provisioning. #1278 delivers: the code (works against any S3-compatible authority), the config surface (D11), the behavioral attestation harness (D12, runnable against a real account as the #1279 go/no-go evidence), the Garage devstack, and a security-operations.adoc provisioning-requirements runbook enumerating what production MUST provide — bucket + versioning + Object Lock compliance mode + the retention default (from D10’s ruleset value) + the bucket policy (require s3:if-none-match; deny DeleteObject/ DeleteObjectVersion/unconditional overwrite; grant reader GetObject+GetObjectVersion+prefix-scoped ListBucket) + split writer/reader/enumerator IAM + the production signing-key generation ceremony. The actual account/credential provisioning is recorded as a deployment concern on the tracker (a separate ops issue linked from #1278), not silently deferred into #1279.

D14 — ADR-014 Amendment 10 (0b commit)

Amendment 10 ratifies (discharging Amendment 6’s delegation): the authority selection (S3-compatible object store over canopy-store/object_store; production = AWS S3, versioning, Object Lock compliance
retention, split IAM, the conditional-write-required + delete-denied policy; Garage devstack functional-not-adversarial); the ENUMERABLE transparency frontier + authority-derived time as the rollback-defeating root of trust; the RFC 7638 kid identity + ANCHOR_SIGNING_VERSION→2 + kid retirement as a signing_key_history.registered_at window checked against the object’s authority time; the unpredictable-recovery-token key layout + envelope v1; monotonic replay protection (enumeration + conditional create + version-pinned reads + confirm-time byte equality); the ordered/linked/fenced/version-pinned confirm contract + failed-anchor evidence path; the anchor census; the anchor_integrity kind + family-global scope + the family-run ok resolution gate; the coverage-lag input + max-age tightening; per-jurisdiction ruleset retention (ADR-006); the separate-emitter-deployable arm split; and the honest attestation split (behavioral probes vs provisioning attestation). It also records that #1280/#1208/#1247 own their own confirm-check arms.

Failure-transition table (C5; the plan owns this byte-level table — Amendment 10 summarizes)

# Failure Actor Effect / recovery

1

manifest build/append error

emitter

warn + backoff; cadence gap → age (authority time) → stale

2

cross-replica emission race

emitter

advisory-lock loser skips; append RAISE benign backstop

3

sign failure

emitter

stays pending (CHECK needs jws+kid for non-pending); retried; persistent → stale

4

store outage/timeout at put_create

emitter

no state change; submission halts this tick; retry rides emit interval

5

AlreadyExists, semantically EQUAL

emitter

submitted, outcome='adopted' (crash-recovery; stored kid/jws/receipt persisted)

6

AlreadyExists, DIVERGENT

emitter

failed, outcome='divergent', family submission halts at seq; D6© latches anchor_integrity

7

PermissionDenied mid-run

emitter

unchanged; re-attestation loop (IAM drift fail-closed)

8

attestation probe failure at boot

emitter

dormant + capped retry + error; process alive; no anchors → stale

9

confirm: chain-check reject

confirmer

existing manifest kinds latch; family halts

10

confirm: external outage/timeout

confirmer

stays submitted; NO latch (outage ≠ integrity); age → stale

11

confirm: frontier ahead / missing / version-gone / bytes / jws / kid / non-canonical / linkage / continuity

confirmer

latch anchor_integrity (family-global); family halts → breached; D7 resolution

12

key rotation mid-flight

both

an anchor’s kid verifies iff the object’s authority time is in that kid’s [registered_at, successor.registered_at) window (D3, WARN on non-current); emitter signs with current

13

genesis at cutover

emitter + cutover

emitter SUBMITS via the installing-state read (rows 4–6 cover crash windows); the one-shot confirm_genesis step CONFIRMS it before chain_epoch_activate (a confirm-ladder failure there aborts the cutover, fail-closed)

14

census: local rewrite / linkage / head tamper / bad stored JWS / external rewrite

confirmer

latch anchor_integrity; family halts

15

burned canonical key (leaked writer preplant)

operator

submission wedges at seq (fail-closed) → rotate writer cred THEN chain_anchor_recovery_rotate (fresh random token) → resume; old attempts still census-verifiable

16

recovery rotate concurrent/retry

operator

idempotent by operation_id; fenced on expected_token; stale old-token store cannot mutate (attempt carries old token)

17

confirmed anchor mutation attempt

anyone

impossible via fns (immutable); out-of-band owner rewrite caught by census (row 14)

Test inventory (~80 named; grouped by MR)

MR-1 (canopy-store): put_create_returns_split_receipt; put_create_conflict_is_already_exists; get_version_pins_the_recorded_version; get_version_detects_new_current_version_and_delete_marker; get_capped_rejects_oversized_metadata_first + …streaming_cap; list_prefix_enumerates; from_config_omits_empty_endpoint_for_native_aws; permission_denied_maps_typed; devstack: garage_conditional_create_probe, garage_versioning_probe, garage_list_probe, object_lock_retain_until_support_probe, provisioned_split_key_roundtrip_reader_write_denied, recreate_and_boot_preserves_the_authority.

MR-2 (crates): p256_jwk_thumbprint_matches_rfc7638_kat (+ leading-zero coords, PEM-norm, wrong-curve/malformed refusal); anchor_kid_is_prefixed_thumbprint; anchor_vectors_regenerate_at_v2; anchor_vectors_kid_is_the_real_derivation; verify_anchor_jws_strict_negatives; object_path_zero_pads_lexicographic (1 vs 2 vs 10 vs 2^53-1); envelope_round_trips_canonically; envelope_rejects_noncanonical/foreign-keys/unknown-version/oversized; envelope_preserves_manifest_bytes_exactly; proptest_envelope_decode_encode_identity; canopy-anchor-authority: store_authority_maps_store_errors_to_authority_errors, store_authority_get_by_version_and_version_gone, store_authority_list_enumerates, chain_anchor_config_parses_under_two_env_prefixes (no canopy-security dep).

MR-3 (substrate): emit_reshape_writes_an_attempt_row_including_on_failed; confirmed_requires_confirmed_attempt_and_prod_version; attempts_are_append_only; recovery_rotate_is_idempotent_by_operation_id; recovery_rotate_fences_on_expected_token; stale_old_token_store_cannot_mutate_after_rotate; recovery_journal_records_actor_reason_incident; anchor_integrity_is_family_global_and_dedups; anchor_integrity_evidence_has_anchor_seq_not_seq (position_seq NULL); confirm_cas_refuses_stale_lease_and_mismatched_fingerprint; emitter_reads_topology_epochs_sources_heads_in_all_three_dbs
emitter_cannot_read_preimage_views (ADR-004); emitter_and_verify_can_read_signing_key_history; signing_key_registration_select_and_compare_rejects_kid_mismatch; retired_kid_refuses_new_anchor_but_verifies_old (registered_at window vs object authority time); genesis_kid_resolves_for_the_genesis_object (first key covers the seq-1 genesis, cross-family); migration_set_replays_green_twice_with_emitter_nologin; emitter_role_login_connect_emit_refuse_confirm; retention_ruleset_key_below_floor_is_rejected.

MR-4 (emitter): emitter_dormant_by_default; emitter_config_capability_matrix_and_redaction; emitter_emits_periodic_tip_after_interval + …skips_recent + …skips_when_installing; cross_replica_emission_race_yields_one; emitter_submits_and_records_full_attempt; crash_replay_adopts_stored_evidence_semantically (kid-rotation-across-crash ADOPTS, no false breach); divergent_object_marks_failed_and_records_attempt; transient_outage_leaves_pending_retry_on_interval; store_timeout_bounded; permission_denied_reattests; sign_failure_leaves_pending; fti_slot_mismatch_refuses; genesis_pickup_via_real_xtask; attestation: conditional_create_enforced, unconditional_overwrite_denied_original_stays_current, writer_delete_denied, reader_write_denied, probe_failure_is_dormant_not_fatal; anchor_key_boot_barrier_blocks_task_start_until_registered.

MR-5 (confirmer/status): confirm_happy_path_advances_trusted_ref_same_pass; confirm_ordered_walk_stops_at_oldest_unresolved; confirm_linkage_and_continuity_enforced (old-prefix rewrite with advanced tip caught); confirm_cas_refuses_lost_lease; confirm_noncanonical_external_ref_or_authority_refuses; confirm_version_pinned_read_detects_new_version_delete_marker_identical_bytes; external_ahead_of_local_latches_rollback (the real anti-rollback defense); local_authority_time_rewrite_is_latched (the D6 step-A discrepancy latch); status_age_from_created_at_is_best_effort_liveness (honest — not claimed to defeat a frozen-confirmer attacker); confirm_genesis_during_installing_before_activation (the one-shot cutover step, genesis fence arm); kid_window_by_registered_at_vs_object_time_refuses_out_of_window; failed_anchor_divergent_object_latches_without_emitter; failed_anchor_error_outcome_missing_object_does_not_latch; unknown_kind_confirm_defers_without_latch; census_detects_local_rewrite/linkage/head_tamper/bad_jws/external_rewrite; census_full_walk_is_bounded_by_anchor_count_no_cursor; census_rejects_oversized_object_metadata_first; coverage_lag_degrades_to_stale_reason_16_while_fresh_frozen_anchors_exist; idle_chain_equal_tips_confirms_clean; manifest_max_age_default_21600_and_relationship_enforced; family_run_ok_gated_on_external_arm_cannot_clear_dormant; anchor_integrity_resolves_only_via_manual_ok_family_run_with_external_checks; status_wire_shape_unchanged_except_additive_reason (insta snapshots green); engine_manifest_check_byte_identical_after_selector_refactor (24-test suite green); arm_split_confirmer_holds_no_signing_key_or_writer_creds; dormant_confirmer_no_activity.

Files Touched (by MR)

Area Change

crates/canopy-store/src/{store.rs,error.rs,lib.rs} + tests/store_test.rs

MR-1 hardening + probes

xtask/src/{devtools.rs,cmd/…,docker.rs}, docker-compose.yml, .ports.env, crates/canopy-test-lib

MR-1 Garage provisioning + persistence + chain-anchor key + recreate E2E

crates/canopy-signing/src/{jwk.rs,anchor.rs} + examples/generate_anchor_vectors.rs + tests/{anchor_vectors_test.rs,vectors/anchor_signature.json}

MR-2 kid + freeze + vectors

crates/canopy-chain/src/{anchor.rs,versions.rs,lib.rs}

MR-2 AnchorObjectKey (pure) + envelope codec (NO async trait — charter-pure)

crates/canopy-anchor-authority/** (NEW async crate; deps canopy-chain + canopy-store + async-trait + chrono + serde; NO sqlx/p256)

MR-2 ExternalAnchorAuthority trait + AuthorityError + StoreAnchorAuthority impl + AnchorReceipt/AnchorSubmission DTOs (moved from canopy-chain) + the shared env-parameterized ChainAnchorConfig + its parse test. (AnchorKeyStore/ConfirmDeps are NOT here — confirmer-only, in canopy-security.)

services/canopy-security/migrations/2026…_chain_anchor_authority.sql; services/canopy-{tanf,medicaid}/migrations/2026…_chain_anchor_emitter.sql (each CREATEs canopy_chain_anchor_emitter NOLOGIN + the all-three read-set grant only)

MR-3 schema + attempts + recovery journal + roles + retention key

crates/canopy-common/src/{chain_verify.rs,chain_append.rs} (+ status.rs), crates/canopy-contracts-security/src/chain.rs, tools/canopy-cli

MR-3/MR-5 AnchorIntegrity, AnchorSelector`check_anchor_by_id`check_continuity, read_topology_any_state/confirm_genesis helper, reason 16

rulesets/{jurisdiction}/jurisdiction.toml + citations.toml

MR-3 the [chain.retention] per-family key

services/canopy-anchor-emitter/** (new crate/binary/Dockerfile) + docker-compose.yml + CI

MR-4 the emitter deployable (depends on canopy-anchor-authority; loads the ruleset for retention)

services/canopy-security/src/chain_verify/{host.rs,mod.rs} (+ AnchorKeyStore/ConfirmDeps home here, confirmer-only), src/main.rs, src/config.rs (constructs ConfirmDeps from the shared ChainAnchorConfig; no anchor config struct defined here)

MR-5 confirmer + census + status

docs/modules/ROOT/pages/adrs/adr-014-fti-audit-hash-chain.adoc

0b Amendment 10

docs/modules/ROOT/pages/{configuration-reference,security-operations,services,data-models/canopy-security,data-models/canopy-tanf}.adoc, runbooks/*, CHANGELOG.adoc, nav.adoc

0b + per-MR docs

Scope — explicitly OUT

  • Production account/bucket/IAM/credential PROVISIONING (D13 — a separate deployment concern; ops issue).

  • The epoch_closure (#1280) and archive_watermark/purge_boundary (#1208/#1247) confirm-check arms — this plan pins genesis+periodic_tip; those kinds submit+store here but confirm DEFERS them.

  • Widening the JWKS endpoint’s Program enum to serve the chain-anchor slug (filed follow-up).

  • The #1279 operational LOGIN flip + the repo-wide serde_jcs migration (#1281).

Verification

  1. cargo xtask validate (fmt, clippy -D warnings, no unwrap/expect/panic in prod paths) before each push.

  2. cargo nextest run --workspace — all suites incl. the ~80 above; Garage-dependent tests skip when infrastructure_available() is false.

  3. cargo xtask dev restart (schema) between MR-3 and later local runs; touch crates/canopy-test-lib/src/db.rs.

  4. Every MR proves DORMANCY (flags off ⇒ zero emitter/confirmer activity); the attestation harness runs green against Garage (and is runnable against a real S3 account for the #1279 go/no-go).

  5. cargo xtask plan-lint green; the doc gate (check-docs) green.

Documentation Updates

  • ADR-014 Amendment 10 (0b).

  • configuration-reference.adoc — the CHAIN_ANCHOR_* matrix + the retention ruleset key.

  • security-operations.adoc — the provisioning-requirements runbook (D13) + rotation + recovery-token runbooks.

  • data-models/canopy-security.adoc (+ canopy-tanf.adoc role notes) — attempts table, recovery journal, recovery_token, split receipt columns, emitter grants.

  • services.adoc — the new canopy-anchor-emitter deployable pointer.

  • CHANGELOG.adoc == Unreleased — Added (anchor authority, emitter deployable) + Changed (ANCHOR_SIGNING_VERSION→2, additive reason 16, CHAIN_MANIFEST_MAX_AGE_SECS default, the AnchorSubmission/AnchorReceipt relocation from canopy-chain to canopy-anchor-authority superseding the substrate DTO-home note).

  • nav.adoc — this plan under Active (0b), moved to Archive at Step 6.

Edit this page · default