Plan: chain-v2 external anchor authority — enumerable transparency frontier, signed manifests, verifier confirmation (#1278, epic &73)
On this page
- Status
- Context (recon facts — verified
file:line) - D0 — the load-bearing principle: an EXTERNAL, ENUMERABLE transparency frontier
- D1 — canopy-store append-only hardening (MR-1)
- D2 — pure primitives (canopy-chain) + the async authority crate (MR-2)
- D3 — anchor-key identity (MR-2, canopy-signing) + safe registration/retirement (MR-3)
- D4 — schema (MR-3): one canopy-security migration + tanf/medicaid twins
- D5 — the emitter deployable
canopy-anchor-emitter(MR-4) - D6 — the confirmer (MR-5, in canopy-security’s verifier)
- D7 — the anchor census (MR-5)
- D8 — status: coverage freshness, the resolution gate, the max-age tightening (MR-5)
- D9 — the emitter role / LOGIN model (MR-3), resolved against the verify precedent
- D10 — retention: a per-jurisdiction, per-family ruleset value (MR-3)
- D11 — config: the capability matrix (
ChainAnchorConfigtype + parse: MR-2 in the authority crate; capability-matrix validation consumed MR-4/MR-5) - D12 — attestation (MR-4): behavioral probes, honestly scoped
- D13 — production provisioning is a SEPARATE deployment concern (OUT of #1278)
- D14 — ADR-014 Amendment 10 (0b commit)
- Failure-transition table (C5; the plan owns this byte-level table — Amendment 10 summarizes)
- Test inventory (~80 named; grouped by MR)
- Files Touched (by MR)
- Scope — explicitly OUT
- Verification
- Documentation Updates
DEFERRED (2026-08-02, #1294) — superseded by
ADR-014 Amendment 11. This v2 design was
REJECTED at a second external review: its "independently enumerable transparency frontier" cannot prove
completeness or currentness — a delete-then-recreate truncation is invisible to signatures
conditional-create + last_modified alone — and it assumed object_store capabilities the pinned
0.13.2 API does not expose. The corrected architecture is the WORM capability-tier trust model
(Amendment 11); the byte-level replacement is
chain-v2 anchor authority — WORM-tier. This file is retained UNCHANGED
below only as the historical record Amendment 10 references — do NOT implement against it.
Child of the chain-v2 rollout plan (#1236); successor to the substrate (#1246), append transport (#1207), and verifiers (#1205/#1206). The contract is ADR-014 Amendment 5 C5 (anchors) as revised through Amendment 9, with the authority/credential selection RATIFIED by Amendment 10 (this plan’s 0b commit — Amendment 6 delegated it explicitly). A contextless implementer reads Amendments 5–10 first; this plan owns the byte-level design the ADR delegates.
Sequencing. #1278 is the last code-side blocker for the #1279 cutover (go/no-go:
"authority + production key provisioned") and blocks #1280 (epoch closure). Everything here lands
dormant — the emitter deployable and the confirmer sub-step do nothing until their flags + the
#1279 credential activation. #1280 (epoch-closure anchors) and #1208/#1247 (archive/purge boundary
anchors) REUSE this machinery: submission and storage are kind-agnostic, but the confirm chain-check
arm is pinned to genesis+periodic_tip here — the other kinds define their check arms in their own
issues (confirm DEFERS an unsupported kind, never latches on it).
Provenance. v1 of this plan (author draft + two internal contextless review rounds, 13+4 findings
folded) was REJECTED at external review 2026-08-02 (~1 critical, ~20 high, ~6 material findings —
the external authority rooted its trust in mutable local DB state; recovery overwrote evidence;
confirm was ID-only/unfenced; receipts unrepresentable; role/LOGIN, Object Lock, IAM, and production
delivery under-specified). This v2 changes the architecture: the authority is an INDEPENDENTLY
ENUMERABLE transparency frontier, evidence is append-only, confirmation is ordered/linked/fenced/
version-pinned, and the emitter is a separate deployable. Three forks were ratified by the maintainer:
(1) separate emitter deployable (not a single-process residual); (2) production account/bucket
provisioning is a separate deployment concern, OUT of #1278 (which delivers code + config
behavioral attestation + a provisioning-requirements runbook only); (3) Object Lock retention is a
per-jurisdiction, per-family ruleset value (ADR-006/003/011), floor-bounded — not a hardcoded
constant. The full finding-disposition appendix lives in the review record; the design below is the
post-disposition state.
Status
| Step | Description | Status |
|---|---|---|
0a |
Tracker reshape: reconcile #1278 ACs to this design; file the JWKS-widening follow-up ( |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
0b |
Plan-commit (docs-only, direct to main after battery): this plan + nav (Active) + ADR-014 Amendment 10 + CHANGELOG. The mandated independent contextless reviews ran against this exact artifact pre-commit — four rounds (10 → 11 → 4 → 2 findings, all folded; round 4 converged with only trivial text fixes). The maintainer’s confirmatory pass (0c) runs against the committed artifact before implementation. |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
1 |
MR-1 store hardening + devstack ( |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
2 |
MR-2 crates ( |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
3 |
MR-3 substrate ( |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
4 |
MR-4 emitter deployable ( |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
5 |
MR-5 confirmer + status + amendment-realization ( |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
6 |
Post-merge: close #1278 with the closing comment; parent + epic bookkeeping; move this plan to Archive + fix nav/xrefs; #1279 go/no-go item recorded. |
Deferred (2026-08-02) — superseded by ADR-014 Amendment 11 (#1294) |
Epic: &73
Issue: #1278 (critical, w5) — blocked by #1246 (Done); blocks #1279, #1280
Branches: this docs MR, then feature/1278-store-hardening → -anchor-crates → -anchor-substrate → -anchor-emitter → -anchor-confirm
Discipline: local cargo xtask validate before each MR’s first push; every MR dormant-safe; Relates to #1278 on MR-1..4, Closes #1278 on MR-5 (the >1-MR justification is the five independently-reviewable subsystems: store primitives, pure crates, substrate, the new deployable, the confirmer).
Context (recon facts — verified file:line)
-
The substrate is landed and dormant.
chain_anchors+chain_anchor_headsexist in canopy_security ONLY, all families (services/canopy-security/migrations/20260910000000_chain_v2_substrate.sql:253-287):anchor_kind ∈ {genesis, periodic_tip, epoch_closure, archive_watermark, purge_boundary};notarization_state ∈ {pending, submitted, confirmed, failed}defaultpending;UNIQUE (instance, family, anchor_seq);CHECK (state='pending' OR (jws NOT NULL AND kid NOT NULL)); a single nullableexternal_versioncolumn;verifier_confirmed_at;created_at.chain_anchor_append(:623-688) locks the head FOR UPDATE, enforces seq-1-with-zero-sentinel genesis, contiguity, and previous-hash linkage, and is same-bytes idempotent. -
The emit/confirm arm split is landed + grant-negative tested. Hardening (
20261010000000_chain_verification_hardening.sql:596-654):chain_anchor_transition_emit(id, to, jws, kid, external_ref, external_version)covers pending→submitted|failed, submitted→failed, failed→submitted, refusesconfirmed, COALESCE-sets columns;chain_anchor_transition_confirm(id)allows only submitted→confirmed and touches no caller columns.canopy_chain_anchor_emitter(NOLOGIN) has EXECUTE on append+emit + SELECT onchain_anchors/chain_anchor_heads(:707-713);canopy_chain_verifyhas EXECUTE on confirm (:711). Negatives pinned atchain_v2_substrate_test.rs:1236-1266,chain_verifier_host_test.rsgrant matrix. -
The emitter role reconcile has NO carrier allowlist AND raises on LOGIN (
hardening.sql:47-55raises if the role hasrolcanlogin;:66-75raises on any memberNOT LIKE 'test\_%'). It re-runs on every EphemeralSchema replay (roles are cluster-global). Contrast the verify side, which allowlistscanopy_security_verifyas a member (substrate.sql:104) and flips it LOGIN operationally at #1279 (never in the migration set). D9 resolves the emitter model against this precedent. -
chain_incident_resolveis immutable and epoch-loose (hardening.sql:543-591): it requires a MANUALokrun of the incident’sdetected_loop_kind, same instance/family/shard, finished after detection — but does NOT check epoch.chain_incident_latchdedups on(instance, family, epoch, shard, kind, detected_loop)withNULLS NOT DISTINCT, so a NULL-epoch family-global incident dedups correctly. D6/D8 use both facts. -
canopy-chain has the manifest + the unconsumed DTOs.
AnchorManifest(RFC 8785, 13-key preimage,from_canonical_bytesre-encode byte-equality,src/anchor.rs:85-207),AnchorSubmission{manifest_bytes, jws}+AnchorReceipt{authority_id, external_ref, version_or_etag, stored_at}(:320-342, ZERO consumers),DOMAIN_TAG_ANCHOR/ANCHOR_MANIFEST_VERSION=1/ANCHOR_SIGNING_VERSION=1(src/versions.rs). KATs for all five kinds (tests/vectors/anchor_manifest.json). -
canopy-signing has the strict JWS surface, unwired.
ANCHOR_JWS_TYP="canopy-chain-anchor+jws";sign_anchor_detached;verify_anchor_jws(vk, expected_kid, payload, jws)(strict alg/typ/kid, deny_unknown_fields, RFC 6979 deterministic,src/anchor.rs). Vectors declared"provisional_until": "#1278…"(tests/vectors/anchor_signature.json, test kidTEST-ONLY-anchor-key). The determination kid scheme (derive_kid=canopy-{program}-{16 hex of SHA-256(PEM text)},src/signer.rs:96-104) is UNSUITABLE for anchors; JWK x/y helpers exist (src/jwk.rs); no RFC 7638 code anywhere. -
canopy-store is overwrite-capable and drops receipts.
Store::putis#[deprecated]and discardsPutResult(src/store.rs:92-97); object_store 0.13.2 supportsPutMode::Create(AWS→If-None-Match) +PutResult{e_tag, version}+GetOptions;from_configunconditionally calls.with_endpoint(&config.s3_endpoint)even when empty (:57) and sets no checksum header;getbuffers the whole body (:142);enforce_backendfails closed to s3 outsideCANOPY_ENV=development. Consumers: canopy-applications, canopy-notices. Garage devstackdxflrs/garage:v2.2.0, regiongarage, API :3900 / admin :3903 (tokencanopy-admin-token), ports in.ports.env; NO bucket/key provisioning automation; NO named Garage data volume; canopy-security mounts/tmpas tmpfs. -
The verifier host is family-leased.
services/canopy-security/src/chain_verify/host.rs:fetch_topology→ FAMILY lease → halt gate → jobs →family_manifest_step(:438-475, calls enginemanifest_checkwhich fetches the LATEST CONFIRMED anchor and never verifies JWS or touches external storage) + census → per-shard tail/scrub under a global visit budget.verifier_passreturns idle onTopologyStatus::NotActive(:104) —fetch_topology’s `NotActivearm carries no instance/epoch/shard_count/source (crates/canopy-common/src/chain_append.rs:306-371). No production code calls append/emit/confirm — only tests +xtask chain-genesis(which appends the pending seq-1 genesis anchor). Per-family verifier tasks spawn on flag+URL (main.rs:181-247). -
Retention is policy data, per ADR-006. Federal floors are jurisdiction-overridable inputs in
rulesets/{jurisdiction}/jurisdiction.toml(+ a siblingcitations.tomlper ADR-011); services load scalar policy values from the ruleset at boot (e.g.services/canopy-tanf/src/params.rs, medicaid[shared.timing]). canopy-security reads no jurisdiction config today. Noretentionkey exists in any ruleset yet. C7 (#1208/#1247) already mandates "per-jurisdiction ruleset values, per family, legal-hold aware, Pub 1075 §4 floor."
D0 — the load-bearing principle: an EXTERNAL, ENUMERABLE transparency frontier
The purpose of an external authority is to be a root of trust the local-DB attacker cannot roll back. The rejected v1 failed this: every decision (which object to read, which anchor is newest, what time it is) came from mutable local rows, so a coherent local-snapshot rollback made the verifier fetch an old, valid, Object-Locked object and never look for the newer ones. This plan makes the authority itself the high-water mark, discovered by enumeration, never inferred from local state.
-
Reader ENUMERATES the authority. The confirmer and census
list{prefix}/{instance}/{family}/(prefix-scopedListBucket) to learn the true set of anchor objects that exist externally — the maximumrecovery_tokennamespace(s) and the maximumanchor_seqpresent — independent of any local row. Enumeration is a first-class reader capability (and it distinguishes 404 from 403: withoutListBucketan absent object under a deny-by-default policy returns 403, making "missing vs denied" undecidable). -
Rollback is a breach. If the external frontier is AHEAD of local confirmed state (objects exist for
(token, seq)the local DB has no confirmed — or no any — anchor row for), the local DB was rolled back or an anchor suppressed. That latchesanchor_integrity(family-global). The store is append-only (conditional create + deny delete/overwrite + Object Lock), so "external ahead of local" is unforgeable. -
The anti-rollback defense is the confirmer’s enumeration + an independent auditor — NOT read-time status. Status is served from LOCAL state (a pure
derive_statusover precomputed inputs), so NO read-time local signal can resist a full privileged-DB attacker: they canSETany freshness stamp tonow()by plain DML AND freeze the confirmer, keeping status green. Be honest about this. The genuine, unforgeable guarantees are EVENTUAL and require an honest observer: (i) any honest confirmer pass re-enumerates the authority (list) and latchesanchor_integrityon external-ahead-of-local (D0.2) or on a live-vs-storedauthority_timediscrepancy; (ii) an independent auditor with only the bucket + the public key reconstructs and verifies the whole anchor chain out-of-band, regardless of what canopy’s local status says (the Kerckhoffs root of trust). Against a fully-compromised DB whose confirmer is permanently frozen, local status is by construction untrustworthy — which is exactly why the external authority + the external audit, not the status endpoint, ARE the trust anchor. Read-time status freshness is therefore BEST-EFFORT liveness only: the confirmer stamps a live-observed authority time each pass (D6/D8), status ages off it, and a frozen frontier or a dead confirmer ages tostalein the honest case — but it is not claimed to defeat the privileged rollback (that is (i)/(ii)'s job). The emitter being a separate deployable means a privileged-DB attacker holds no writer S3 credential and cannot mint a fresh object version (and a new version over a recorded one is itself a version-pinned breach, D6) — so the confirmer’s live enumeration reads the true aging frontier. -
The object carries the order it asserts. The envelope embeds the full manifest (binding
anchor_seq,previous_anchor_hash, the C1 identity, and per-shard tips), so an independent auditor holding only the bucket + the public key reconstructs and verifies the entire anchor chain from genesis with zero database access. Local state is a cache of this, never the source.
Every D-section below serves D0.
D1 — canopy-store append-only hardening (MR-1)
New primitives on Store (crates/canopy-store/src/store.rs); the deprecated put stays for the
#435 upload seam — put_create is a DISTINCT integrity seam for machine-generated canonical evidence
(codec-validated; magic-byte sniffing would reject JSON), documented as such.
-
pub struct PutReceipt { pub version_id: Option<String>, pub etag: Option<String>, pub authority_time: Option<DateTime<Utc>> }— the three receipt facts SPLIT (the DTOversion_or_etagcollapse was a rejection finding).authority_timefrom the store’s returned/last-modified metadata. -
put_create(path, data) → Result<PutReceipt, StoreError>viaput_opts(PutMode::Create)(AWS→If-None-Match: *; LocalFileSystem native), setting the SDK checksum header Object-Lock uploads require. Mapsobject_store::Error::AlreadyExists → StoreError::AlreadyExists{path}. -
get_version(path, version_id) → Result<(Bytes, PutReceipt), StoreError>viaGetOptions{version, ..}— version-pinned read (fetch the RECORDED version, not "current"), so a new current version or a delete marker over the recorded version is DETECTABLE even when bytes match. -
head(path) → Result<ObjectMeta, StoreError>andget_capped(path, max_bytes) → Result<(Bytes, PutReceipt), StoreError>— metadata-first size check + a streaming read hard-capped atmax_bytes(the currentgetbuffers the whole body; a preplanted large object must not exhaust memory). -
list_prefix(prefix) → impl Stream<Item = Result<ObjectMeta, StoreError>>— prefix-scoped enumeration (D0.1); reuses the #1215list_keys_streamshape. -
StoreErrorgainsAlreadyExists{path},PermissionDenied{path},NotFound{path},TooLarge{path, size, cap},VersionGone{path, version}(a new current version/delete-marker over a recorded version);From<StoreError> for ApiErrorarms (409/403/404/413/409). -
from_config(:46-77): passs3_endpointasNonewhen empty (native AWS must not receive.with_endpoint("")); the anchor authority constructsObjectStoreConfigvalues programmatically (its own namespace, D11) — the two trust domains never share env or bucket.
Devstack (MR-1): a named Garage data volume (survives container recreation — today Postgres persists but Garage does not, so a recreate can keep confirmed rows while deleting their objects); chain-anchor key generation + mount wiring for the emitter (the manual generator rejects that name today, xtask/src/cmd/gen_signing_keys.rs; the auto-generator omits it); xtask Garage provisioning (idempotent, via the admin API :3903: bucket canopy-chain-anchors, writer/reader/enumerator key pairs); a recreate-and-boot E2E proving the authority survives.
Empirical gates (MR-1, documented in-test + local-dev.adoc): Garage v2.2.0 (a) conditional-write
(put_create twice → AlreadyExists?), (b) versioning, (c) list, (d) whether object_store 0.13.2
exposes per-object Object-Lock retain-until on put_opts. Outcomes pin D10’s enforcement mechanism and
whether the devstack anchor store is Garage-s3 or the Local backend (native PutMode::Create) with
Garage reachability-only. No fake attestation either way.
D2 — pure primitives (canopy-chain) + the async authority crate (MR-2)
Crate placement (a rejection finding — canopy-chain is charter-pure "bytes-only, no async, no db,
no signing", crates/canopy-chain/src/lib.rs:3-14). The split:
-
canopy-chain (pure, no new deps):
AnchorObjectKey(a value type + itsobject_pathstring derivation) and the envelope codec — no I/O, no async. -
canopy-anchor-authority (NEW async crate; deps canopy-chain + canopy-store + async-trait + chrono
serde): theExternalAnchorAuthoritytrait,AuthorityError, the I/O value types (StoredAnchor,AnchorObjectMeta, the reshapedAnchorReceipt,AnchorSubmission— moved here from canopy-chain), the concreteStoreAnchorAuthorityimpl overcanopy_store::Store(writer + reader constructions), and the shared env-parameterizedChainAnchorConfigderivation (D11). It needs NO sqlx/p256 (no DB pool, no key parsing) —AnchorKeyStore+ConfirmDepsare CONFIRMER-only and live in canopy-security (which already has sqlx + canopy-signing), so the authority crate stays minimal. BOTH deployables (the emitter, MR-4; canopy-security’s confirmer, MR-5) depend on THIS crate — never on each other — so the process arm split holds with no shared service-lib dependency.
// canopy-chain (pure):
pub struct AnchorObjectKey { pub instance: ChainInstanceId, pub family: ChainFamily,
pub recovery_token: Uuid /* UNPREDICTABLE namespace, D4 */, pub anchor_seq: i64 }
impl AnchorObjectKey {
/// {prefix}/{instance}/{family}/{recovery_token}/{anchor_seq:020}.json
/// 20-digit zero-pad ⇒ lexicographic == numeric across the full 2^53-1 domain.
pub fn object_path(&self, prefix: &str) -> String { /* … */ }
}
// canopy-anchor-authority (async):
#[async_trait]
pub trait ExternalAnchorAuthority: Send + Sync {
fn authority_id(&self) -> &str;
async fn store(&self, key: &AnchorObjectKey, sub: &AnchorSubmission) -> Result<AnchorReceipt, AuthorityError>;
async fn get(&self, key: &AnchorObjectKey, version_id: Option<&str>) -> Result<StoredAnchor, AuthorityError>;
async fn list(&self, instance: ChainInstanceId, family: ChainFamily) -> Result<Vec<AnchorObjectMeta>, AuthorityError>;
}
pub struct StoredAnchor { pub bytes: Vec<u8>, pub receipt: AnchorReceipt }
pub struct AnchorObjectMeta { pub recovery_token: Uuid, pub anchor_seq: i64, pub version_id: Option<String>, pub authority_time: Option<DateTime<Utc>>, pub size: u64 }
pub enum AuthorityError { AlreadyExists, NotFound, PermissionDenied(String), Transient(String), VersionGone, Invalid(String) }
pub struct StoreAnchorAuthority { /* canopy_store::Store + prefix + authority_id */ } // the concrete impl
The concrete StoreAnchorAuthority maps AnchorObjectKey::object_path → Store::put_create/get_version/
list_prefix, PutReceipt → AnchorReceipt, StoreError → AuthorityError (incl. VersionGone → VersionGone);
store()/get() set the metadata-first size cap (D1). AnchorReceipt = { authority_id, external_ref,
version_id, etag, authority_time } (the version_or_etag/stored_at collapse removed). The DTOs
AnchorSubmission/AnchorReceipt MOVE from canopy-chain to canopy-anchor-authority (they are I/O
contracts, not pure bytes); the substrate plan’s "DTOs in canopy-chain" note is superseded (recorded in
the CHANGELOG Changed). The trait is &dyn-safe so #1280 reuses it.
Envelope codec (canopy-chain src/anchor.rs, ANCHOR_ENVELOPE_VERSION=1):
encode_anchor_envelope(manifest_bytes, jws) → Result<Vec<u8>> builds
{"anchor_envelope_version":1,"jws":…,"manifest":<value>}, canonical_bytes, and enforces the
post-condition that re-extracting + re-canonicalizing the manifest reproduces the input bytes;
decode_anchor_envelope(bytes, max_len) → Result<(Vec<u8>, String)> strict 3-key decode + version pin
full re-encode byte-equality + size cap. Proptest (parser/serializer rule) + a frozen KAT.
D3 — anchor-key identity (MR-2, canopy-signing) + safe registration/retirement (MR-3)
-
jwk.rs:p256_jwk_thumbprint(public_key_pem) → Result<String, SigningError>— RFC 7638 §3 exactly ({"crv":"P-256","kty":"EC","x":…,"y":…}, lexicographic members, no whitespace, b64url-unpadded coords viapublic_pem_to_xy), SHA-256, base64url-unpadded.anchor.rs:ANCHOR_KID_PREFIX="canopy-chain-anchor:",anchor_kid(pem) = prefix + thumbprint. -
Header freeze + version bump. Freezing the provisional vectors is a change to frozen vectors under Amendment 6’s rule, so
ANCHOR_SIGNING_VERSION→ 2; vectors regenerate at v2 underanchor_kid(TEST_ONLY pubkey)(delete + regenerate via the no-overwrite generator — the deliberate act recorded in the MR); dropprovisional_until. `verify_anchor_jws’s exact-kid check is unchanged. KATs additionally cover coordinate encoding, leading-zero x/y, PEM normalization, wrong-curve/malformed-key refusal, and the retired-key-vs-new-anchor rule. Amendment 10 pins whether the protected-header BYTES are normative (member order/encoding) or only the semantic fields. -
Retention in
signing_key_history(MR-3). Registration becomes SELECT-and-compare, notON CONFLICT DO NOTHING: an existing kid whose stored program or normalized public key disagrees with the recomputed RFC 7638 kid is a hard error, never silently accepted. Retirement is a TIME window over the EXISTINGsigning_key_history.registered_at, checked against the anchor object’s authorityLastModified— NO new column, no per-seq formula (a seq window is wrong here: one anchor key signs ALL THREE families whoseanchor_seqspaces each restart at 1, and aGREATEST(head+1,…)formula can never cover genesis seq 1 — the round-2 seq approach had both bugs). One anchor key is current at a time (the emitter holds oneCHAIN_ANCHOR_SIGNING_KEY); rotation registers a successor.resolve_kid_for_object_time(kid, object_authority_time)(theAnchorKeyStoremethod) acceptskidiff it is a registeredchain-anchorkey AND the object’s authorityLastModifiedfalls in[kid.registered_at, successor.registered_at)(current kid: to infinity), tie-broken on(registered_at, kid). This is (i) genesis-safe (no seq), (ii) cross-family-correct (registration time is global), and (iii) non-forgeable by the privileged-DB attacker: the object’sLastModifiedis authority-set (Object Lock + no writer credential for the DB attacker), so a leaked RETIRED key cannot be used to confirm a NEW anchor — its object would carry a post-retirementLastModifiedoutside the key’s window. Old keys stay VERIFIABLE for their own era (history). Residual, documented: forging a confirmable anchor at all still requires the WRITER credential (a separate deployable, Object-Locked), so a retired-key compromise adds no attack surface beyond a writer-credential compromise, which the recovery runbook contains by rotating the writer credential first. The reserved program slug ischain-anchor(JWKS-servable-slug widening is a filed follow-up — theProgramenum does not admit it today). -
The boot barrier is emitter-side and split from the confirmer (a rejection finding). Registration is NOT the existing fire-and-forget HTTP path (
signing_registration.rs, which is best-effort,Program-enum-gated, and re-derives the determination kid). Instead the EMITTER deployable, at boot, performs a BLOCKING directsigning_key_historySELECT-and-compare + INSERT of its RFC 7638 anchor key (it holds the INSERT grant, D4; emitter tasks do not start until it succeeds). The CONFIRMER (canopy-security) has no signing key and cannot INSERT — its "barrier" is a SELECT-present check: if the anchor key is absent it cannot verify JWS, so confirmation stays dormant (anchors remainsubmitted→ age tostale), never a crash. The INSERT grant survives #1279’s identity restriction.
D4 — schema (MR-3): one canopy-security migration + tanf/medicaid twins
DDL follows the substrate discipline (DO/EXECUTE format() with SET search_path = <schema>, pg_temp;
create-then-transfer to canopy_chain_owner_security; REVOKE PUBLIC; overload rule = DROP before any
signature change). Timestamp AFTER 20261015000000 (security) / AFTER 20261010000000 (tanf/medicaid —
their fti_preimage_id postdates the substrate); crates/canopy-test-lib/src/db.rs is re-touched so
sqlx::migrate! re-embeds.
Append-only attempts (the recovery-overwrites-evidence fix). Mutable per-anchor generation/ref columns
would let failed(token_g0) → rotate → submitted(g1) erase the planted g0 object before the verifier
sees it. Instead:
CREATE TABLE chain_anchor_attempts (
id UUID PRIMARY KEY,
anchor_id UUID NOT NULL REFERENCES chain_anchors(id),
anchor_seq BIGINT NOT NULL, -- DENORMALIZED from chain_anchors; the emit fn (which
-- locks the anchor row) validates it equals the anchor's
-- seq. A cross-table CHECK cannot express this in PG.
recovery_token UUID NOT NULL,
external_ref TEXT NOT NULL, -- the canonical object path this attempt used
external_version_id TEXT, -- non-null required in production (enforced in the fn)
external_etag TEXT,
external_authority TEXT NOT NULL,
authority_time TIMESTAMPTZ, -- the store's LiveModified as observed AT STORE TIME
-- (recorded for audit; freshness reads the LIVE value, D0.3)
submitted_kid TEXT NOT NULL,
submitted_jws TEXT NOT NULL,
outcome TEXT NOT NULL CHECK (outcome IN ('stored','adopted','divergent','error')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (recovery_token, anchor_seq) -- one attempt per external key (token+seq fixes the object)
);
-- append-only guard (statement-level trigger, the signing_key_history precedent)
chain_anchors gains current_attempt_id UUID REFERENCES chain_anchor_attempts(id) (the confirmed/latest
attempt pointer). The confirmed/failed ⇒ outcome and production ⇒ version_id invariants are enforced IN
the SECURITY DEFINER chain_anchor_transition_emit/_confirm bodies (which already lock and read both
rows) — NOT as table CHECKs, because a CHECK cannot reference another table and "production" is not row
data (a rejection finding). The single-table CHECK that DOES hold: `outcome’s closed set (above). No
nullable/COALESCE "maybe-stamped" columns — every attempt is a full row.
chain_anchor_heads gains recovery_token UUID NOT NULL DEFAULT gen_random_uuid() — the current key
namespace (unpredictable; a leaked writer cannot preplant the next integer key).
No new checkpoint columns. Read-time status-age keeps using the LANDED path — chain_anchor_trusted_v
→ chain_anchors.created_at (read_trusted_manifest, status.rs:441-459), unchanged — and is honestly
labelled BEST-EFFORT liveness (D0.3/D8): a live-restamped authority_observed_at on the family checkpoint
row was considered and REJECTED because the family row is CHECK-forbidden from carrying cursors/stamps
(Amendment 9’s family-shape invariant) and the verify role cannot write checkpoints except via
chain_checkpoint_advance — adding a stamp there would fight the substrate for a signal that is
best-effort by construction (no read-time local signal defeats the privileged rollback; the enumeration
latch does). The anti-rollback guarantee is entirely the confirmer’s external-ahead-of-local enumeration
the independent auditor (D0).
Recovery rotation replaces the naive "generation bump":
CREATE FUNCTION chain_anchor_recovery_rotate(
p_instance uuid, p_family text, p_expected_token uuid, p_operation_id uuid,
p_reason text, p_incident_id uuid
) RETURNS uuid -- the new token (or the already-applied one, replayed)
-- locks the head (the head row is guaranteed present — genesis pre-created it); the p_operation_id
-- lookup runs UNDER the head lock; if already journaled → replay its result (idempotent);
-- refuse if p_expected_token <> current (concurrent/stale-retry fence); else mint a fresh
-- random token, journal (from/to token, operation_id, actor=session_user, reason, incident_id,
-- bumped_at) into chain_anchor_recovery_journal, advance the head token.
-- EXECUTE → canopy_chain_incident_admin (a recovery action under incident authority).
chain_anchor_recovery_journal carries UNIQUE (operation_id) — replay is schema-enforced, not merely
lock-ordering discipline (a concurrency test asserts two same-operation_id rotations produce exactly one
journal row).
A stale old-token store result arriving after rotation cannot mutate the anchor row: its attempt carries
the old token; the head’s current token is the new one; the burned (old_token, seq) object is still
discoverable via its attempt row (the evidence survives). Precondition (runbook, not machine-checkable):
the compromised writer credential is rotated FIRST — a token rotation alone does not contain a leaked
writer.
Emit reshape — chain_anchor_transition_emit grows p_recovery_token, p_external_authority,
p_external_version_id, p_external_etag, p_authority_time, p_attempt_outcome, p_anchor_seq (the
denormalized value, validated against the locked anchor row), and p_require_version boolean (the emitter
derives it from CANOPY_ENV — a SECURITY DEFINER fn cannot read the env, and "production" is not row
data). It INSERTs a new chain_anchor_attempts row + advances current_attempt_id on EVERY store-attempt
transition — → failed (divergence) included — enforcing IN THE BODY: p_attempt_outcome IN
('stored','adopted') AND p_require_version ⇒ p_external_version_id NOT NULL (rejects a version-less
production store); the confirmed/failed ⇒ outcome half is genuine row data. Per-attempt exact
assignment, no COALESCE.
Confirm CAS reshape — chain_anchor_transition_confirm grows (p_lease_token, p_expected_predecessor_id,
p_expected_predecessor_hash, p_checked_fingerprint). TWO fence arms (the confirmed-immutable rule holds
in both):
-
Steady (periodic_tip): commits only if the FAMILY LEASE token (
p_lease_token, validated the landedchain_run_recordway —PERFORM 1 FROM chain_verification_checkpoints WHERE …shard_id=0 AND loop_kind='family' AND lease_token=p_lease_token) still holds AND the predecessor + the checked fingerprint (manifest_hash, external_ref, external_authority, version_id, kid, recovery_token) still match — a stale verifier that lost its lease during the (non-transactional, bounded) S3 I/O cannot confirm. -
Genesis (a rejection finding — no family pass/lease exists during
installing): foranchor_kind='genesis'(seq 1) the fn acceptsp_lease_token = NULLand instead STRUCTURALLY fences on the epoch row beinginstalling(a state reachable only pre-activation, and the #1279 cutover is a single-actor coordinated operation — there is no concurrency to fence). It still requires the checked fingerprint match.confirm_genesis(D6) calls this arm; a test pins that it succeeds duringinstallingand is refused once the epoch isactive(steady genesis re-confirm is impossible).
Incident kind — anchor_integrity (23rd) joins the chain_incidents kind CHECK; latched
family-global (epoch NULL, shard NULL — dedups via NULLS NOT DISTINCT), so a per-epoch family run
cannot clear it. VerifyReject::AnchorIntegrity{anchor_seq, recovery_token, external_ref, version_id, kid,
phase, expected, got} (canopy-common); evidence carries anchor_seq (NEVER top-level seq — the _app
projection reads evidence→>'seq' as an event position; position_seq stays NULL for this kind).
Grants — scoped PER DATABASE (a rejection finding: signing_key_history, chain_anchor_attempts, and
the recovery journal exist ONLY in canopy-security; anchors are security-only; a literal all-three twin
would fail relation does not exist). These are TABLE privileges, not role memberships, so none collide
with the hardening membership reconcile.
-
All three DBs (security + tanf + medicaid — the fti families'
fetch_topology+capture_headsreads):canopy_chain_anchor_emittergains CONNECT + schema USAGE + SELECT onchain_topology, chain_epochs, chain_sources, chain_heads. The tanf/medicaid substrates do NOT createcanopy_chain_anchor_emitter(only the security hardening migration did), so each twin migration first CREATEs it NOLOGIN under the substrate reconcile discipline (idempotent dual-SQLSTATE, fail-closed on privileged attributes / non-test_%members), then grants. -
canopy-security only:
canopy_chain_anchor_emittergains INSERT + SELECT onsigning_key_history(the boot-barrier registration, D3/D5, and the adopt-classifier’s old-kid JWS verify);canopy_chain_verifygains SELECT onsigning_key_history+chain_anchor_attempts+chain_anchor_recovery_journal.
Grant-negative (PUBLIC-probe) tests both directions in each DB.
D5 — the emitter deployable canopy-anchor-emitter (MR-4)
A NEW service (services/canopy-anchor-emitter/, its own binary/Dockerfile/CI/compose entry) — the
process-level arm split (a compromise of this process holds the signing key + writer creds but CANNOT
confirm; a compromise of canopy-security’s confirmer holds reader creds but no signing key/writer creds).
It loads rulesets/{jurisdiction}/jurisdiction.toml at boot (D10 retention), registers the anchor public
key (boot barrier, D3), and runs one task per family (audit via the security DB; fti-tanf/fti-medicaid
via their DBs — X10 per-family dormancy on flag+URL). All functions ≤40 lines (decompose like host.rs).
run_emitter_loop(target, cfg, security_pool, chain_pool, writer, key, retention):
sleep(first_tick_delay); attested=false; backoff=250ms
loop:
if !attested:
attest(writer, reader, CANOPY_ENV)? # D12 probes; Err → error!, backoff, continue
attested=true
emitter_pass(...) → Ok: backoff=250ms | Err: warn!, backoff=min(2*backoff, emit_interval)
sleep(tick_ms)
emitter_pass(target, cfg, security_pool, chain_pool, writer, key, retention):
# 1. topology (installing OK — reads chain_topology/chain_sources DIRECTLY, not fetch_topology,
# so genesis SUBMITS while the epoch is `installing`; genesis CONFIRMATION is the cutover's
# one-shot confirm_genesis step, D6)
topo = read_topology_any_state(chain_pool, target.family)? ; if absent → Idle
if target.fti_source? != topo.source → Err(SlotMismatch) # swapped URL fails loudly
scope = VerifyScope::from(topo)
# 2. EMISSION (cadence; single-writer via advisory lock + in-tx recheck)
tx = security_pool.begin(); SET LOCAL statement_timeout = store_timeout
if pg_try_advisory_xact_lock(key(instance, family)):
if newest_anchor_authority_age(instance, family) >= emit_interval AND topo.state == active:
heads = capture_heads(chain_pool, scope); m = AnchorManifest{PeriodicTip, seq=last+1, prev=last_hash, heads…}
chain_anchor_append(instance, family, epoch, last+1, 'periodic_tip', m.bytes, m.hash, last_hash)
tx.commit else tx.rollback # loser skips; append contiguity RAISE is the benign backstop
# 3. SUBMISSION (oldest-first; the xtask genesis PENDING row rides this path unmodified)
for row in select pending|failed order by anchor_seq limit submit_batch:
verify local manifest_hash == sha256(row.manifest_bytes) else → CRITICAL, Halt
jws = sign_anchor_detached(key, row.manifest_bytes)? # sign err → stays pending (CHECK), Halt
okey = AnchorObjectKey{instance, family, head.recovery_token, row.anchor_seq}
match timeout(store_timeout, writer.store(okey, {envelope, jws}, retention_for(family))):
Ok(rcpt) → emit(row, submitted, jws, key.kid, okey.path, rcpt…, outcome='stored')
AlreadyExists → stored = writer.get(okey, None)? # explicit fetch; SEMANTIC classify:
if decode(stored).manifest_bytes == row.manifest_bytes
AND verify_anchor_jws(resolve(stored.kid), stored.kid, row.manifest_bytes, stored.jws):
emit(row, submitted, stored.jws, stored.kid, okey.path, stored.rcpt…, 'adopted') # crash-recovery
else emit(row, failed, jws, key.kid, okey.path, rcpt…, 'divergent'); CRITICAL; Halt
Transient/timeout → warn!; Continue # unchanged; retry rides emit_interval, not the tick
PermissionDenied → error!; attested=false; Halt # IAM drift → re-probe
Divergence ALWAYS reaches the verifier: the failed row’s current_attempt records the canonical key, and
D6’s failed-anchor evidence path fetches it and latches. Failed rows retry oldest-first on the EMIT
INTERVAL (capped backoff), not the tick — one bounded probe per interval, not log spam.
D6 — the confirmer (MR-5, in canopy-security’s verifier)
Runs in family_confirm_step, inserted in the family pass BEFORE family_manifest_step (a just-confirmed
anchor becomes the trusted ref the same pass). Holds ONLY the reader authority (get/list) + the verify
pools — no signing key, no writer creds. ConfirmDeps = None (reader store unconfigured) ⇒ dormant, but
an explicit CHAIN_ANCHOR_CONFIRM_ENABLED=true with no reader creds is a STARTUP error (no silent
disable).
family_confirm_step(pools, cfg, db, scope, family, family_lease, reader, keys):
# (A) ROLLBACK DETECTION via enumeration (D0) — the real anti-rollback defense
ext = reader.list(scope.instance, family)? # the external frontier, prefix-scoped
if ext.max_seq > local_max_any_anchor_seq(pools.security, scope, family):
latch AnchorIntegrity{phase:'frontier', expected:local_max, got:ext.max_seq}; Halt
# Integrity (not freshness): compare the live LastModified of the newest confirmed anchor's object
# to its stored attempt.authority_time; a beyond-tolerance discrepancy is a local-rewrite signal.
if abs(ext.newest.authority_time - stored_attempt_authority_time(...)) > tolerance:
latch AnchorIntegrity{phase:'authority-time'}; Halt
# (read-time status-age uses the landed chain_anchors.created_at path — best-effort liveness, D8;
# NO checkpoint stamp is written here — the family row is cursor/stamp-free, Amendment 9.)
# (B) ORDERED, LINKED, FENCED confirm walk (stop at the oldest unresolved anchor)
prev = local_last_confirmed(pools.security, scope, family) # id + manifest_hash + tip snapshot
for row in select submitted order by anchor_seq: # STRICT order; a gap halts the walk
if row.anchor_seq != prev.anchor_seq + 1
or row.previous_anchor_hash != prev.manifest_hash: latch AnchorIntegrity{phase:'linkage'}; Halt
if kind(row) not in {genesis, periodic_tip}: log defer; Halt # #1280/#1208/#1247 arms elsewhere
# (1) continuity, not just tip ordering: chain hash at prev tip seq still == prev tip hash (per shard, same epoch)
check_continuity(pools.chain, scope, prev.tips)? else latch{phase:'continuity'}; Halt
# (2) full anchor-vs-chain check on THIS anchor (engine check_anchor_by_id: strict decode, 7 cols, topology, tip prefix)
check_anchor_by_id(pools.security, pools.chain, scope, family, row.id)? else latch (existing manifest kinds); Halt
# (3) external evidence at the CANONICAL key (row.recovery_token from its attempt), version-pinned
okey = AnchorObjectKey{scope.instance, family, row.attempt.recovery_token, row.anchor_seq}
if row.attempt.external_ref != okey.path or row.attempt.external_authority != cfg.authority_id:
latch{phase:'noncanonical'}; Halt
match reader.get(okey, row.attempt.external_version_id):
Transient → warn!; return # stays submitted; NO latch; age → stale is the SLO
NotFound|VersionGone → latch{phase:'evidence'}; Halt # submitted arm: missing/rolled = breach
Ok(stored) →
if decode(stored).manifest_bytes != row.manifest_bytes: latch{phase:'bytes'}; Halt
# kid retirement is a registered_at window checked against the OBJECT's authority time (D3) —
# cross-family-correct + genesis-safe + non-forgeable by the DB attacker (object time is authority-set)
vk = keys.resolve_kid_for_object_time(row.kid, stored.receipt.authority_time) else latch{phase:'kid'}; Halt
verify_anchor_jws(vk, row.kid, row.manifest_bytes, decode(stored).jws)? else latch{phase:'jws'}; Halt
if row.kid != keys.current_kid(): warn!(rotation visibility)
# (4) CAS confirm under the family fence (no tx across S3; lease heartbeats during I/O)
chain_anchor_transition_confirm(row.id, family_lease.token, prev.id, prev.manifest_hash, fingerprint(row))
prev = row
# (C) FAILED-ANCHOR evidence path (divergence detection independent of the emitter)
# failed ⇒ outcome ∈ {divergent, error} (D4). Both may have a stored object at the attempt key;
# an 'error' attempt that never stored yields NotFound (benign — the emitter retries).
for row in select failed where current_attempt.outcome in ('divergent','error') order by anchor_seq:
okey = AnchorObjectKey{scope.instance, family, row.attempt.recovery_token, row.anchor_seq}
match reader.get(okey, row.attempt.external_version_id):
NotFound → return # emitter retries; a genuinely absent object is not (yet) evidence
Ok(stored) → if not semantically_equal(stored, row): latch AnchorIntegrity{phase:'failed-evidence'}; Halt
Engine seam (a rejection finding — the named seam did not match the check). Today
fetch_and_check_anchor is PRIVATE, takes only the security pool, and does steps 1–3 (row metadata /
topology / constants); the tip-prefix check is check_tip_prefixes (needs the chain pool), composed
only by manifest_check. So MR-5 adds a PUBLIC check_anchor_by_id(security, chain, scope, family, id) =
fetch_and_check_anchor (id-selected, expected_state='submitted') plus check_tip_prefixes — the
both-pools, tip-prefix-inclusive check the confirm walk calls; manifest_check keeps its
LatestConfirmed fetch BYTE-IDENTICAL (the 24-test engine suite stays pinned to it), refactored to share
the selector AnchorSelector{ LatestConfirmed | ById(id, expected_state) }. check_continuity(chain,
scope, prev.tips) is EXPLICIT new MR-5 engine work (reuse cursor_hash_at — assert the current chain
hash at each prev tip seq equals prev.tips[shard].last_hash, same epoch/topology), NOT part of
check_anchor_by_id. The confirm walk composes: linkage → check_continuity → check_anchor_by_id →
external evidence → CAS.
Confirm wiring + the missing types (a rejection finding). VerifierPools (chain_verify/mod.rs, today
{chain, security}) gains confirm: Option<ConfirmDeps>; spawn_family_task’s signature and
`verifier_pass → leased_pass → scheduled_pass thread it to family_confirm_step. None ⇒ confirm
dormant (anchors age to stale, never a crash); CHAIN_ANCHOR_CONFIRM_ENABLED=true with no reader creds
is a STARTUP error (D11). The types ConfirmDeps + AnchorKeyStore are CONFIRMER-only and live in
canopy-security (it holds sqlx + canopy-signing; reader: Arc<dyn ExternalAnchorAuthority> is the only
authority-crate type they reference) — the authority crate stays DB/key-free:
pub struct ConfirmDeps {
pub reader: Arc<dyn ExternalAnchorAuthority>, // reader creds + ListBucket only
pub keys: AnchorKeyStore,
pub confirm_batch: usize,
pub authority_id: String,
pub key_prefix: String,
}
/// DB-backed over signing_key_history (program 'chain-anchor'); small in-memory cache.
pub struct AnchorKeyStore { /* pool + cache */ }
impl AnchorKeyStore {
/// The kid valid at an object's authority time: `kid` is a registered chain-anchor key AND
/// `object_authority_time ∈ [kid.registered_at, successor.registered_at)` (current kid: to ∞),
/// tie-broken (registered_at, kid) (D3 — uses the EXISTING registered_at; no seq column; genesis-safe,
/// cross-family-correct, non-forgeable via the authority-set object time). None ⇒ unregistered/out-of-window.
pub async fn resolve_kid_for_object_time(&self, kid: &str, object_authority_time: DateTime<Utc>) -> Option<VerifyingKey>;
pub async fn current_kid(&self) -> Option<String>;
}
Genesis confirmation is a one-shot CUTOVER step, not the steady family pass (a rejection finding). The
steady family_confirm_step runs inside the leased family pass, which verifier_pass reaches only for an
active topology (host.rs:104 returns idle on NotActive, and fetch_topology maps installing →
NotActive with no scope). But Amendment 6 requires the genesis anchor VERIFIER-CONFIRMED before
chain_epoch_activate, i.e. while the epoch is still installing. Reworking the whole verifier host to
run its family pass during installing is invasive and wrong; instead genesis confirmation is a dedicated
confirm_genesis(security, chain, reader, keys, scope) entry point that reads chain_topology/
chain_sources DIRECTLY (like the emitter’s read_topology_any_state — a new
crates/canopy-common/src/chain_append.rs helper, added to Files Touched), runs the SAME confirm ladder
(linkage/continuity/check_anchor_by_id/external-evidence) against the installing-epoch genesis anchor,
and calls chain_anchor_transition_confirm’s genesis fence arm (D4: `p_lease_token=NULL, structurally
fenced on the epoch being installing) — NOT the family-lease arm, because no family pass/lease/checkpoint
row exists during installing (the earlier "runs the SAME confirm and calls confirm" glossed this; the
fence source is now explicit). It is invoked as an explicit step of the #1279 cutover
sequence (which is already a coordinated operation: quiesce → genesis → notarize → CONFIRM GENESIS →
activate → reopen) — genesis confirmation belongs there, before activation. This is not a "manual verify"
in the deprecated sense; it is a bounded cutover step. The earlier "genesis-confirm now in-process, no
manual path" framing is corrected: steady-state periodic_tip confirmation is in-process (the family pass);
genesis confirmation is the cutover’s own step.
D7 — the anchor census (MR-5)
Appended to family_census_step (same cadence stamp). Combines local integrity + the external frontier:
-
No cursor — a bounded full walk each cadence. Anchors accrue ~1 per emit interval (~9k/year/family, so ~88k/decade, at hourly cadence), so even a decade of history is a cheap full walk (hash + JWS re-verify) needing no resume state — which is fortunate, because the family checkpoint row is CHECK-forbidden from carrying a cursor (Amendment 9’s family-shape invariant) and the verify role cannot write checkpoints outside
chain_checkpoint_advance(a round-3 finding killed the "durable cursor on the family checkpoint" idea). If a family ever exceeds ~100k anchors (~a decade out), a resume cursor would be a SEPARATE dedicated table + guarded writer, filed then — not the family row. -
Local: seq contiguity,
manifest_hash == sha256(manifest_bytes),previous_anchor_hashlinkage, strict decode + seq/kind/epoch cross-check, JWS re-verify (non-pending, kid valid at the object time), head consistency — detects owner-level local rewrites. -
External: re-
list+ spot-get(version-pinned, capped) a bounded sample per cadence to detect external-side rewrite/rollback the confirm walk hasn’t re-touched;TooLargeon any object rejects it metadata-first. -
Epoch-spanning (walks all local anchors, all epochs) — so a manual family
okrun genuinely re-verifies old-epoch anchors, which is what makes D8’s resolution gate sound.
D8 — status: coverage freshness, the resolution gate, the max-age tightening (MR-5)
-
Coverage-lag input. A compromised emitter could emit fresh anchors with FROZEN tips forever; age-based staleness alone would stay green while everything above the frozen tips sits outside manifest protection. Status computes
max over shards (head.last_seq − trusted-manifest tip.last_seq)(from data status already loads) and degrades tostalevia a NEW additive reasonmanifest_coverage_laggingwhen it exceedsCHAIN_MANIFEST_MAX_LAG. Reason vocabulary 15→16 — additive wire change (domain enumchain_verify/status.rs, wire enumchain.rs, serde pin, CLI parse pin, service bijection); no path or schema-shape change. An idle chain (equal tips, heads not advancing) does NOT trip it. -
Age from the LANDED path —
read_trusted_manifest→chain_anchor_trusted_v→chain_anchors.created_at(status.rs:441-459), UNCHANGED. This is BEST-EFFORT liveness (D0.3): in the honest case a frozen frontier / dead confirmer ages tostale; a full privileged-DB attacker who rewritescreated_atAND freezes the confirmer is NOT defeated by read-time age — the privileged-rollback defense is the confirmer’s external-ahead-of-local enumeration + the authority-time discrepancy latch (eventual, D6 step A) and the independent auditor, NOT this read-time age. (A live-restampedauthority_observed_atwas rejected — it fights the family-row-no-cursor CHECK for a best-effort signal.) -
CHAIN_MANIFEST_MAX_AGE_SECSdefault 604800 → 21600 (6h), with the relationship≥ 3 × CHAIN_ANCHOR_EMIT_INTERVAL_SECS(startup error). -
Resolution gate (review Q).
anchor_integrityis family-global; the immutablechain_incident_resolverequires a manualokfamily run of the detected loop newer than detection. That is sound ONLY if a family run’sokoutcome is GATED on the external arm: a family run CANNOT reportokwhile any submitted/failed anchor or latchedanchor_integrityexists unless the confirmer’s external checks (confirm walk + census) actually ran and passed this run. A dormant confirmer therefore cannot produce anokthat clears an external breach.
D9 — the emitter role / LOGIN model (MR-3), resolved against the verify precedent
The landed hardening reconcile raises on BOTH a non-test_% member of canopy_chain_anchor_emitter
(:66-75) AND the role itself having LOGIN (:47-55), and re-runs every replay — so neither a
member-carrier nor a migration-flipped LOGIN works. The verify side proves the only viable model:
canopy_security_verify is flipped LOGIN operationally at #1279 (never in the migration set), so every
replay runs with it NOLOGIN and the reconcile passes. The emitter follows this exactly:
canopy_chain_anchor_emitter is the login identity, flipped LOGIN+password by #1279’s operational
credential-activation step, and stays NOLOGIN in all migration runs. Tested (review S’s "test with the
role already LOGIN"): apply the full migration set (role NOLOGIN → reconcile green), then operationally
ALTER ROLE … LOGIN, then prove the emitter connects + emits + is refused confirm; and document the
cutover invariant — the substrate/hardening migrations are not re-applied against a cluster whose carriers
are LOGIN (uniform for verify/incident-admin/emitter; a #1279-wide property, recorded on #1279, tested
here). Grants added in D4.
D10 — retention: a per-jurisdiction, per-family ruleset value (MR-3)
Per ADR-006, the federal floor is a jurisdiction-overridable input, not a source constant; per C7,
retention is per-jurisdiction, per-family. #1278 INTRODUCES the chain-retention ruleset key (which
#1208/#1247 reuse — one source of truth): a [chain.retention] table in
rulesets/{jurisdiction}/jurisdiction.toml with per-family *_retention_years values + a sibling
citations.toml entry (ADR-011), bounded below by the family’s federal floor (Pub 1075 §4 5y for FTI;
the applicable floor for audit) — the emitter’s boot load rejects a value below the floor.
Enforcement mechanism is pinned by MR-1’s empirical gate (D1): if object_store 0.13.2 exposes
per-object Object-Lock retain-until on put_opts, the emitter sets each object’s retain-until from the
ruleset value at store time (application-controlled); otherwise the retain-until is a BUCKET DEFAULT set
at provisioning from the same ruleset value (documented in the provisioning-requirements runbook, D13) —
the ruleset is the single source either way, and the object time ≥ retention is the immutability window
that makes "external ahead of local" durable evidence. Object Lock ENABLEMENT (bucket-level) is
provisioning (D13); the retention VALUE is ruleset policy (this section).
D11 — config: the capability matrix (ChainAnchorConfig type + parse: MR-2 in the authority crate; capability-matrix validation consumed MR-4/MR-5)
The parsed chain_anchor namespace + its ObjectStoreConfig derivation live in the shared
canopy-anchor-authority crate (D2), env-prefix PARAMETERIZED — so the emitter deployable and
canopy-security’s confirmer each construct their own view without either depending on the other’s service
lib (a rejection finding: "one struct at SecurityConfig reused by the emitter" would make the emitter
compile-depend on canopy-security, undoing the arm split). Validated to capability-specific matrices —
every violation a STARTUP error, never a clamp:
-
Emitter capability (emit enabled): writer creds + reader creds (AlreadyExists GET + probes) + signing key + emitter DB URL(s) + bucket + authority_id + jurisdiction (retention).
-
Confirmer capability (
CHAIN_ANCHOR_CONFIRM_ENABLED): reader creds +ListBucket+ verify pools; NO writer creds, NO signing key. Confirm-on-without-reader-creds fails at startup.
Knobs (all CANOPY_ANCHOR_EMITTER… on the emitter deployable / CANOPY_SECURITY… for the confirmer
side): CHAIN_ANCHOR_EMIT_ENABLED (false) · CHAIN_ANCHOR_CONFIRM_ENABLED (false) · TICK_MS 15000
(250..=60000) · FIRST_TICK_DELAY_SECS 30 (0..=3600) · EMIT_INTERVAL_SECS 3600 (60..=604800) ·
SUBMIT_BATCH 4 · CONFIRM_BATCH 4 · STORE_TIMEOUT_MS 10000 (1000..=120000) ·
ANCHOR_DATABASE_URL (+TANF/MEDICAID per-family, RedactedUrl, lazy) ·
STORE_{BACKEND s3|local(dev-only via CANOPY_ENV), BUCKET, S3_ENDPOINT, S3_REGION, WRITER_ACCESS_KEY/SECRET,
READER_ACCESS_KEY/SECRET, ENUMERATOR_*, ALLOW_HTTP (dev-only), KEY_PREFIX (grammar/length), MAX_OBJECT_BYTES} ·
AUTHORITY_ID (nonempty ≤128) · EXPECTED_AWS_ACCOUNT_OWNER · SIGNING_KEY (canopy-secrets; .keys/chain-anchor-private.pem dev fallback) ·
CHAIN_MANIFEST_MAX_LAG 1000000 (≥1, documented SLO derivation) · existing CHAIN_MANIFEST_MAX_AGE_SECS
→ 21600 with the ≥ 3× emit interval rule.
Access keys/secrets use secret wrappers with redacted Debug/serialize (coding-conventions:304).
bucket/prefix/authority_id are IMMUTABLE once anchors exist — a persistent authority-registry row
(in canopy_security) rejects a silent trust-domain swap at boot.
D12 — attestation (MR-4): behavioral probes, honestly scoped
At the emitter task’s first pass when CANOPY_ENV != development (fail-closed), under the reserved
disjoint {prefix}/_probe/ namespace (never a burnable anchor key; stable canary key, lifecycle-bounded):
-
conditional-create enforced (
put_createon the canary → Ok first-ever or AlreadyExists thereafter; a secondput_createMUST be AlreadyExists); -
unconditional overwrite DENIED — an explicit unconditional PUT MUST be rejected and the original version MUST remain current (the "create twice" check alone proves only create semantics, not overwrite-denial);
-
writer delete DENIED; reader write DENIED; reader read + list work.
Probe failure ⇒ the emitter goes dormant on capped backoff + error! (anchoring fails closed) WITHOUT
killing the process. Control-plane facts (Object Lock retention/legal-hold, versioning, the bucket
policy, split IAM) are NOT reachable through object_store’s data plane — they are provisioning-time
operator attestation (D13). No fake attestation.
D13 — production provisioning is a SEPARATE deployment concern (OUT of #1278)
Ratified: #1278 does NOT ship production IaC or perform account/bucket/credential provisioning. #1278
delivers: the code (works against any S3-compatible authority), the config surface (D11), the behavioral
attestation harness (D12, runnable against a real account as the #1279 go/no-go evidence), the Garage
devstack, and a security-operations.adoc provisioning-requirements runbook enumerating what
production MUST provide — bucket + versioning + Object Lock compliance mode + the retention default (from
D10’s ruleset value) + the bucket policy (require s3:if-none-match; deny DeleteObject/
DeleteObjectVersion/unconditional overwrite; grant reader GetObject+GetObjectVersion+prefix-scoped
ListBucket) + split writer/reader/enumerator IAM + the production signing-key generation ceremony. The
actual account/credential provisioning is recorded as a deployment concern on the tracker (a separate
ops issue linked from #1278), not silently deferred into #1279.
D14 — ADR-014 Amendment 10 (0b commit)
Amendment 10 ratifies (discharging Amendment 6’s delegation): the authority selection (S3-compatible
object store over canopy-store/object_store; production = AWS S3, versioning, Object Lock compliance
retention, split IAM, the conditional-write-required + delete-denied policy; Garage devstack
functional-not-adversarial); the ENUMERABLE transparency frontier + authority-derived time as the
rollback-defeating root of trust; the RFC 7638 kid identity + ANCHOR_SIGNING_VERSION→2 + kid retirement
as a signing_key_history.registered_at window checked against the object’s authority time; the
unpredictable-recovery-token key layout + envelope
v1; monotonic replay protection (enumeration + conditional create + version-pinned reads + confirm-time
byte equality); the ordered/linked/fenced/version-pinned confirm contract + failed-anchor evidence path;
the anchor census; the anchor_integrity kind + family-global scope + the family-run ok resolution
gate; the coverage-lag input + max-age tightening; per-jurisdiction ruleset retention (ADR-006); the
separate-emitter-deployable arm split; and the honest attestation split (behavioral probes vs
provisioning attestation). It also records that #1280/#1208/#1247 own their own confirm-check arms.
Failure-transition table (C5; the plan owns this byte-level table — Amendment 10 summarizes)
| # | Failure | Actor | Effect / recovery |
|---|---|---|---|
1 |
manifest build/append error |
emitter |
warn + backoff; cadence gap → age (authority time) → stale |
2 |
cross-replica emission race |
emitter |
advisory-lock loser skips; append RAISE benign backstop |
3 |
sign failure |
emitter |
stays |
4 |
store outage/timeout at |
emitter |
no state change; submission halts this tick; retry rides emit interval |
5 |
AlreadyExists, semantically EQUAL |
emitter |
→ |
6 |
AlreadyExists, DIVERGENT |
emitter |
→ |
7 |
PermissionDenied mid-run |
emitter |
unchanged; re-attestation loop (IAM drift fail-closed) |
8 |
attestation probe failure at boot |
emitter |
dormant + capped retry + error; process alive; no anchors → stale |
9 |
confirm: chain-check reject |
confirmer |
existing manifest kinds latch; family halts |
10 |
confirm: external outage/timeout |
confirmer |
stays |
11 |
confirm: frontier ahead / missing / version-gone / bytes / jws / kid / non-canonical / linkage / continuity |
confirmer |
latch |
12 |
key rotation mid-flight |
both |
an anchor’s kid verifies iff the object’s authority time is in that kid’s |
13 |
genesis at cutover |
emitter + cutover |
emitter SUBMITS via the installing-state read (rows 4–6 cover crash windows); the one-shot |
14 |
census: local rewrite / linkage / head tamper / bad stored JWS / external rewrite |
confirmer |
latch |
15 |
burned canonical key (leaked writer preplant) |
operator |
submission wedges at seq (fail-closed) → rotate writer cred THEN |
16 |
recovery rotate concurrent/retry |
operator |
idempotent by operation_id; fenced on expected_token; stale old-token store cannot mutate (attempt carries old token) |
17 |
confirmed anchor mutation attempt |
anyone |
impossible via fns (immutable); out-of-band owner rewrite caught by census (row 14) |
Test inventory (~80 named; grouped by MR)
MR-1 (canopy-store): put_create_returns_split_receipt; put_create_conflict_is_already_exists;
get_version_pins_the_recorded_version; get_version_detects_new_current_version_and_delete_marker;
get_capped_rejects_oversized_metadata_first + …streaming_cap; list_prefix_enumerates;
from_config_omits_empty_endpoint_for_native_aws; permission_denied_maps_typed; devstack:
garage_conditional_create_probe, garage_versioning_probe, garage_list_probe,
object_lock_retain_until_support_probe, provisioned_split_key_roundtrip_reader_write_denied,
recreate_and_boot_preserves_the_authority.
MR-2 (crates): p256_jwk_thumbprint_matches_rfc7638_kat (+ leading-zero coords, PEM-norm,
wrong-curve/malformed refusal); anchor_kid_is_prefixed_thumbprint; anchor_vectors_regenerate_at_v2;
anchor_vectors_kid_is_the_real_derivation; verify_anchor_jws_strict_negatives;
object_path_zero_pads_lexicographic (1 vs 2 vs 10 vs 2^53-1); envelope_round_trips_canonically;
envelope_rejects_noncanonical/foreign-keys/unknown-version/oversized;
envelope_preserves_manifest_bytes_exactly; proptest_envelope_decode_encode_identity;
canopy-anchor-authority: store_authority_maps_store_errors_to_authority_errors,
store_authority_get_by_version_and_version_gone, store_authority_list_enumerates,
chain_anchor_config_parses_under_two_env_prefixes (no canopy-security dep).
MR-3 (substrate): emit_reshape_writes_an_attempt_row_including_on_failed;
confirmed_requires_confirmed_attempt_and_prod_version; attempts_are_append_only;
recovery_rotate_is_idempotent_by_operation_id; recovery_rotate_fences_on_expected_token;
stale_old_token_store_cannot_mutate_after_rotate; recovery_journal_records_actor_reason_incident;
anchor_integrity_is_family_global_and_dedups; anchor_integrity_evidence_has_anchor_seq_not_seq
(position_seq NULL); confirm_cas_refuses_stale_lease_and_mismatched_fingerprint;
emitter_reads_topology_epochs_sources_heads_in_all_three_dbs
emitter_cannot_read_preimage_views (ADR-004); emitter_and_verify_can_read_signing_key_history;
signing_key_registration_select_and_compare_rejects_kid_mismatch;
retired_kid_refuses_new_anchor_but_verifies_old (registered_at window vs object authority time);
genesis_kid_resolves_for_the_genesis_object (first key covers the seq-1 genesis, cross-family);
migration_set_replays_green_twice_with_emitter_nologin; emitter_role_login_connect_emit_refuse_confirm;
retention_ruleset_key_below_floor_is_rejected.
MR-4 (emitter): emitter_dormant_by_default; emitter_config_capability_matrix_and_redaction;
emitter_emits_periodic_tip_after_interval + …skips_recent + …skips_when_installing;
cross_replica_emission_race_yields_one; emitter_submits_and_records_full_attempt;
crash_replay_adopts_stored_evidence_semantically (kid-rotation-across-crash ADOPTS, no false breach);
divergent_object_marks_failed_and_records_attempt; transient_outage_leaves_pending_retry_on_interval;
store_timeout_bounded; permission_denied_reattests; sign_failure_leaves_pending;
fti_slot_mismatch_refuses; genesis_pickup_via_real_xtask; attestation:
conditional_create_enforced, unconditional_overwrite_denied_original_stays_current,
writer_delete_denied, reader_write_denied, probe_failure_is_dormant_not_fatal;
anchor_key_boot_barrier_blocks_task_start_until_registered.
MR-5 (confirmer/status): confirm_happy_path_advances_trusted_ref_same_pass;
confirm_ordered_walk_stops_at_oldest_unresolved; confirm_linkage_and_continuity_enforced
(old-prefix rewrite with advanced tip caught); confirm_cas_refuses_lost_lease;
confirm_noncanonical_external_ref_or_authority_refuses;
confirm_version_pinned_read_detects_new_version_delete_marker_identical_bytes;
external_ahead_of_local_latches_rollback (the real anti-rollback defense);
local_authority_time_rewrite_is_latched (the D6 step-A discrepancy latch);
status_age_from_created_at_is_best_effort_liveness (honest — not claimed to defeat a frozen-confirmer attacker);
confirm_genesis_during_installing_before_activation (the one-shot cutover step, genesis fence arm);
kid_window_by_registered_at_vs_object_time_refuses_out_of_window;
failed_anchor_divergent_object_latches_without_emitter;
failed_anchor_error_outcome_missing_object_does_not_latch;
unknown_kind_confirm_defers_without_latch;
census_detects_local_rewrite/linkage/head_tamper/bad_jws/external_rewrite;
census_full_walk_is_bounded_by_anchor_count_no_cursor; census_rejects_oversized_object_metadata_first;
coverage_lag_degrades_to_stale_reason_16_while_fresh_frozen_anchors_exist;
idle_chain_equal_tips_confirms_clean;
manifest_max_age_default_21600_and_relationship_enforced;
family_run_ok_gated_on_external_arm_cannot_clear_dormant;
anchor_integrity_resolves_only_via_manual_ok_family_run_with_external_checks;
status_wire_shape_unchanged_except_additive_reason (insta snapshots green);
engine_manifest_check_byte_identical_after_selector_refactor (24-test suite green);
arm_split_confirmer_holds_no_signing_key_or_writer_creds; dormant_confirmer_no_activity.
Files Touched (by MR)
| Area | Change |
|---|---|
|
MR-1 hardening + probes |
|
MR-1 Garage provisioning + persistence + |
|
MR-2 kid + freeze + vectors |
|
MR-2 |
|
MR-2 |
|
MR-3 schema + attempts + recovery journal + roles + retention key |
|
MR-3/MR-5 |
|
MR-3 the |
|
MR-4 the emitter deployable (depends on canopy-anchor-authority; loads the ruleset for retention) |
|
MR-5 confirmer + census + status |
|
0b Amendment 10 |
|
0b + per-MR docs |
Scope — explicitly OUT
-
Production account/bucket/IAM/credential PROVISIONING (D13 — a separate deployment concern; ops issue).
-
The
epoch_closure(#1280) andarchive_watermark/purge_boundary(#1208/#1247) confirm-check arms — this plan pins genesis+periodic_tip; those kinds submit+store here but confirm DEFERS them. -
Widening the JWKS endpoint’s
Programenum to serve thechain-anchorslug (filed follow-up). -
The #1279 operational LOGIN flip + the repo-wide serde_jcs migration (#1281).
Verification
-
cargo xtask validate(fmt, clippy-D warnings, no unwrap/expect/panic in prod paths) before each push. -
cargo nextest run --workspace— all suites incl. the ~80 above; Garage-dependent tests skip wheninfrastructure_available()is false. -
cargo xtask dev restart(schema) between MR-3 and later local runs;touch crates/canopy-test-lib/src/db.rs. -
Every MR proves DORMANCY (flags off ⇒ zero emitter/confirmer activity); the attestation harness runs green against Garage (and is runnable against a real S3 account for the #1279 go/no-go).
-
cargo xtask plan-lintgreen; the doc gate (check-docs) green.
Documentation Updates
-
ADR-014 Amendment 10 (0b).
-
configuration-reference.adoc— theCHAIN_ANCHOR_*matrix + the retention ruleset key. -
security-operations.adoc— the provisioning-requirements runbook (D13) + rotation + recovery-token runbooks. -
data-models/canopy-security.adoc(+canopy-tanf.adocrole notes) — attempts table, recovery journal,recovery_token, split receipt columns, emitter grants. -
services.adoc— the newcanopy-anchor-emitterdeployable pointer. -
CHANGELOG.adoc == Unreleased— Added (anchor authority, emitter deployable) + Changed (ANCHOR_SIGNING_VERSION→2, additive reason 16,CHAIN_MANIFEST_MAX_AGE_SECSdefault, theAnchorSubmission/AnchorReceiptrelocation from canopy-chain to canopy-anchor-authority superseding the substrate DTO-home note). -
nav.adoc— this plan under Active (0b), moved to Archive at Step 6.