ADR-014: FTI Audit Hash-Chain Integrity

Add two nullable TEXT columns to fti_audit_log and fti_audit_log_archive in both canopy-tanf and canopy-medicaid:

Each service has its own database (ADR-001), so the migrations live in each service’s migrations/ directory and apply independently. Existing rows have NULL for both columns; verification skips contiguous NULL-block prefixes (genesis tail) and starts the chain at the first row with a non-NULL event_hash.

Included: every column an IRS Pub 1075 §4 auditor cares about — who accessed FTI, when, why, what fields, from which system, what kind of action, against what resource.

Excluded: request_id (request-scoped, not access-defining), ip_address (operationally useful but can change without altering the access semantics — DHCP / proxy churn would cause spurious chain breaks), success (binary; an attacker flipping false → true would not be caught by chaining alone, but Pub 1075 §4 does not require it; auditors get the field in the row, not the hash). created_at is excluded for the same reason it’s excluded from the audit_events chain — it’s an internal bookkeeping column written by clock_timestamp() after the hash is computed.

Each FTI database (canopy-tanf, canopy-medicaid) uses its own pg_advisory_xact_lock ID, distinct from any other writer in that database:

Pre-#423 the lock ID was the magic number 2 for both services. Per the amendment in #423 (2026-05-12), the lock ID is now derived from a hash of the human-readable name "<originating_system>.fti_chain" via fti_chain_lock_id (a private helper in canopy-common that mirrors canopy-db::advisory::advisory_lock_id — kept inline to avoid a canopy-db → canopy-common → canopy-db dep cycle). The chain hash itself is unchanged; only the lock-acquisition machinery moved from magic numbers to hash-based naming, matching the canopy-wide convention introduced in E0.4.

(Lock IDs are scoped to a Postgres database, so the same numeric ID in two different databases doesn’t contend. Pre-#423 lock 1 was reserved by canopy-security for audit_events; post-#423 every chain uses its own hashed name and the reserved-number registry is gone.)

Non-FTI writers (program eligibility logic, policy queries) do not take this lock. The advisory lock contends only with concurrent FTI inserts in the same service — workload is small (eligibility determinations call log_access once per FTI read, typically a few per case-touch), so the serialisation overhead is negligible.

PostgresFtiAuditLogger::log_access (crates/canopy-common/src/fti_audit.rs:225) is rewritten to follow the audit_events insert pattern exactly:

The Bug 6 + Bug 7 fixes are mandatory — implementations that write to_rfc3339() or rely on the column default now() are not Pub 1075-compliant under this ADR.

A new crates/canopy-common::fti_audit::verify_fti_chain(pool: &PgPool) → sqlx::Result<Result<usize, (Uuid, String)>> helper mirrors canopy_security::store::verify_chain:

When rows age past Pub 1075’s 5-year retention floor (figure corrected to 7 years, Pub 1075 AU-11 — see Amendment 12 / ADR-041; #1363), archive_expired_records (crates/canopy-common/src/fti_audit.rs:406) moves them from fti_audit_log to fti_audit_log_archive. The chain MUST continue across this boundary so that an auditor inspecting both tables can verify a single contiguous chain from genesis.

Pattern (precedent: services/canopy-security/migrations/20260409000000_align_archive_hash_columns.sql):

A chain break is reportable under IRS Pub 1075 §9 (incident response). When verify_fti_chain returns Errrow_id, reason:

The 503 is not a routine API behaviour — it is the contractual signal that the integrity guarantee has been violated and out-of-band incident response (per Pub 1075 §9) is required. The endpoint MUST NOT serve cached "last good" results while a breach is unresolved.

A daily job services/canopy-security/src/jobs/fti_chain_verify.rs walks each FTI service’s chain and emits the appropriate event. canopy-security is the natural host because (a) it already runs the breach-detection background loop for audit_events, (b) it has read access to no FTI columns — only the chain integrity columns, which are not FTI under Pub 1075 §4 (they are metadata about FTI access events, not the FTI itself).

The verify job uses a read-only Postgres connection per FTI database, configured via per-service connection strings in the canopy-security settings. It does not write to FTI tables; it does not log to fti_audit_log (a verify is not an FTI access).

Per FTI insert (steady state):

Eligibility determinations are not in the hot path of high-RPS workloads (they’re case-scoped, not session-scoped). The serialisation overhead is acceptable for the integrity guarantee.

Column-level row signatures (e.g., per-row HMAC) detect column tampering but not row deletion or reordering. Hash chaining detects all three at the cost of one extra column. HSM-based signing was the alternative considered in ADR-004 itself for the same reason; rejected there for cost. The chain pattern has no marginal infrastructure cost.

Each FTI database is independent (ADR-001). Cross-database locking would require a coordinator service or a shared lock database. Per-database locks are simpler, contention-free across services, and consistent with how canopy-security already serialises its own chain.

Pub 1075 §9 expects out-of-band incident response, not API workarounds. A cached success response would mask an unresolved integrity violation from anyone polling the endpoint — exactly the wrong behaviour. The 503 is the API surface of the breach state; clearing it requires investigator action, not auto-recovery.

created_at is written by clock_timestamp() after the hash is computed. Including it in the hash would create a circular dependency. The same reasoning applies in canopy-security::store::compute_event_hash — created_at is a chain-ordering column, not a chain-content column.

Both can change without altering the FTI access itself: request_id is request-scoped (the same access can have different request IDs across retries), ip_address can change with proxy / load-balancer churn. Including either would produce spurious chain breaks during normal operations. They remain in the row for auditors but not in the chain.

Alternative 1: Per-row signature (HMAC) instead of chain. Detects column tampering, not row deletion or reordering. Rejected — Pub 1075 §4 expects "an audit trail" that is end-to-end verifiable; per-row signatures break "end-to-end".

Alternative 2: Shared chain across canopy-tanf and canopy-medicaid via canopy-security. Would require either replicating FTI rows to canopy-security (violates ADR-004) or a cross-service chain coordinator. Per-database chains preserve isolation; the auditor endpoint aggregates verification status across services without aggregating data.

Alternative 3: Hash includes created_at. Would require splitting the INSERT into two phases (insert with placeholder, update with hash) or reading clock_timestamp() before the INSERT and reusing it. Extra round-trip + complexity for no integrity gain — created_at only orders rows; chain integrity comes from previous_hash / event_hash.

Before the redesign, a fix: MR contains the live hazards without waiting on chain-v2: replace the originating_system-derived lock with a constant per-chain lock (+ a concurrency test proving two distinct originating_system values cannot fork), and fail closed every current full-walk / archive-mutation path — the sync archive endpoint (api/mod.rs:601), the sync FTI/audit verify endpoints (:692, :547), and the boot verify tick — so no full-table walk or inline archive move runs on any request or boot thread. Chain-status reports an explicit interim unknown, never stale-green. (Delaying the boot tick alone is insufficient — the sync manual endpoints still OOM.)

The ADR pins these clauses; the substrate child (#1246) owns the byte-level schema, KAT vectors, and restricted roles.

C1 — Identity + position. The canonical position of every event is the tuple (chain_instance_id, chain_family, chain_epoch, shard_id, chain_seq). chain_instance_id is a non-reusable, trusted per-chain identity minted at each reset; chain_family/service/database identity is enumerated — "FTI" is two instances (canopy-tanf, canopy-medicaid), distinct from audit_events and from ele_grant_events. The full identity is bound into rows, hashes, heads, checkpoints, and anchors. Timestamps are queryable metadata, never position.

C2 — Hash (rules pinned; KAT vectors owned by the substrate child). event_hash = SHA-256 over a typed RFC-8785 / JCS canonical encoding (FTI converges off its concat/comma-join onto the audit JCS approach) of: the domain tag "canopy-chain-v2", the full C1 identity tuple, hash_formula_version (mandatory, fixed = 2, no legacy values — the protocol is named chain-v2 and the domain tag is "canopy-chain-v2" specifically to disambiguate from the pre-existing audit_events hash_version = 2 of Amendment 1, migration 20260621000000), previous_hash, and the canonical_event_payload. The ADR carries a per-family field-coverage table classifying every stored column as hashed / derived / excluded-with-rationale / forbidden (today FTI excludes request_id/ip_address/success/created_at; audit excludes the row id / ip / receipt timestamp). UUID / timestamp / JSONB / string / array normalization, genesis, and the anchor canonical representation are pinned as rules; frozen known-answer test vectors live in the substrate child and gate it. canonical_event_payload must be obtainable and stored so it re-hashes identically — the audit PG-JSONB normalization pattern (store/mod.rs:125, SELECT $1::jsonb once at insert, hash + store the normalized bytes) is the reference.

C3 — Append (durable head + FOR UPDATE). A chain_heads row per (instance, family, epoch, shard) holds last_seq + last_hash — the authoritative durable tip. Writers always chain from chain_heads, never from the newest live row. Per batch: validate + canonicalize before locking; process each shard in its own transaction — SELECT … FOR UPDATE that shard’s head, assign a contiguous chain_seq range, hash + insert, advance the head in the same tx. N is per-shard; a single tx does NOT lock all touched heads (that recreates global serialization + cross-shard rollback). A multi-shard write inside one FTI determination tx computes all shards first, then locks heads in shard-id order. Shard = a server-derived hash (function + modulo pinned) of an immutable event id — the ingestion path pins which id (bus envelope id for #-queue events, server-minted row id for direct ingest). Genesis / empty-head hash, the epoch-0 root, empty-shard genesis, and the first-row previous_hash are all specified. Direct ingest (POST /security/audit/ingest, api/mod.rs:235) enqueues into the same durable staging transport (not a synchronous append); its 202 is redefined to "accepted + durably staged" (chained by the per-shard drainer), replacing today’s "accepted and chained".

C4 — Topology + epochs (fenced state machine). A chain_epochs / topology registry records the active epoch, the fixed shard count, the routing version, the state open|closing|anchored|closed, shard membership, and FKs. Head rows are pre-created — locking a missing row locks nothing, so lazy-create is a fork race. Transitions are fenced so a stale binary is structurally unable to append to a closed epoch or use a different shard count. Shard count changes only at an epoch boundary; a new epoch’s shard genesis references the previous epoch’s anchor. Sharding loses global append order: chronology comes from hashed timestamps / query metadata, and the amendment records that no API or auditor contract relies on the old global order.

C5 — Anchors = external notarized head-manifests. Distinct anchor kinds — periodic tip-notarization, epoch-closure, archive-watermark, purge-boundary. Each manifest binds {chain_instance_id, family, epoch, hash_formula_version, shard-count, routing-version, per-shard (last_seq, last_hash), anchor-kind, previous-anchor-hash} under a pinned canonical encoding, is signed and stored in an external append-only authority (outside the chain-owning DB), is verifier-confirmed, and is emitted on a cadence/SLO. Periodic tip notarization is what closes the coherent-privileged- rewrite gap. The amendment pins the authority + credentials + append-only storage + signature/key retention + monotonic replay protection + every failure transition. Epoch closure is a crash-resumable cross-database state machine, not a single transaction.

C6 — Verification (sharded). Per-(instance, family, epoch, shard) targets + progress + an aggregate run manifest. Two loops: tail (capture a fixed (target_seq, target_hash) per shard, verify bounded batches up to it, expose verified_through + lag) and historical scrub (bounded, resumable, over the full retained history to its own fixed trusted target so it terminates under continuous writes; detects mutation behind the tail). Verification MUST reject: missing/unexpected shards or heads; duplicate, missing, or noncontiguous seq; wrong previous_hash / formula version; a head that is not its terminal row; rows beyond the head; invalid genesis / epoch-anchor linkage; orphan heads or rows; and divergence from the latest external manifest. Status states are unknown|verifying|healthy|stale|error| breached with separate tail vs scrub coverage/freshness (per-shard aggregation, lag/max-age thresholds, derived-state precedence); stale is computed at read time (a worker dying before it records an error cannot leave cached green); a clean pass never auto-clears breached — resolution needs an authorized actor + reason + evidence + revalidation. Citation path: event-specific coverage (the cited event’s shard is verified at/below verified_through and within the notarized manifest) or a dedicated attestation endpoint; fail closed for newer-than-checkpoint / unknown / stale / error / breached. The status DTOs (crates/canopy-contracts-security/src/{chain,fti}.rs) are replaced with per-shard coverage + tail/scrub times + lag + incident id + HTTP mappings. A nonzero staging backlog (staged-but-unchained events) MUST degrade status (never healthy) — events are durably staged
inbox-stamped before ack, and backlog depth surfaces in status. The status/checkpoint store separates mutable progress (cursors/lease) from immutable run results from the sticky incident record; checkpoint identity = (instance, family, epoch, shard, loop-kind) + fence/CAS; the async manual verify returns a job id + polling URL (truly async, not sync-then-202).

C7 — Archive + purge. Archive/purge move a contiguous chain_seq prefix per shard, never a timestamp-selected set. Retention decides eligibility; chain_seq decides the boundary = the greatest leading seq for which every row is eligible (not the largest eligible seq — received_at / accessed_at are not monotonic with seq). The retention bound + purge-enablement are per-jurisdiction ruleset values, per family (loaded from the jurisdiction ruleset, legal-hold aware, bounded below by the Pub 1075 §4 floor). Isolation contract: verifier snapshot semantics, duplicate handling, movement generation, single-archiver ownership, archived_through advanced in the same tx; the move is one transactional DELETE … RETURNING → INSERT with exact count/content equality and rollback on conflict (killing the ON-CONFLICT-DO-NOTHING data-loss path). Purge order: determine the contiguous eligible prefix → verify from the previous trusted boundary → externally sign/store + acknowledge a per-shard boundary manifest → delete locally → advance the local boundary. Failures leave rows intact; anchor-without-purge is safe and retryable. Archival is scheduled only after both verifiers prove the archive-aware path. The audit_events archive/purge is #1208; the FTI twin is #1247.

C8 — ADR-004 preservation + least-privilege. Restricted-data isolation is preserved: per-record Pub 1075 §4 granularity (a batched chain append never collapses per-access rows), and no FTI on canopy.events (manifests and breach events carry no FTI). The amendment corrects the inaccurate "only integrity columns" claim in §8 and the Consequences: verification reads the hashed-preimage projection (the access metadata needed to re-hash — accessed_by, purpose_code, data_elements_accessed, action, resource_type, …), not SELECT * and not "only the chain columns". It pins the minimum projection + grants + restricted DB roles/procedures for the live / archive / head / epoch / anchor tables — the real containment of the maintenance-GUC threat (the append-only guard’s statement-level shape is not the defect). The canopy-security cross-database read is reconciled against ADR-004 by naming the authorized read-only verification role (or amending ADR-004).

P0 containment (#1245) → this amendment (#1236) → chain-v2 substrate (#1246: schema + KATs + roles
empty-genesis) → verifier children (#1205 audit, #1206 FTI) landed dormant → coordinated-downtime cutover (quiesce, reset live + archive + integrity, mint chain_instance_id, install topology + heads, deploy v2 writers + verifier, verify + notarize the empty genesis, reopen) → append transport (#1207: durable staging + leased per-shard drainer) → finish the verifier children (background jobs, citation coverage, status DTOs, singleton scheduling) → archive/purge children last (#1208 audit + #1247 FTI). ele_grant_events migration is deferred (#1248). #1197 stays closed; its predecessor-query EXPLAIN tests are retargeted or retired, and which created_at indexes remain is an ordinary-query / retention decision, not a chain concern.

The substrate research proved serde_jcs 0.1 (the then-JCS dependency; retired repo-wide by #1281 with per-site byte-identity evidence) is not a conforming RFC 8785 implementation: it sorts serialized UTF-8 key bytes (quotes + escapes included) where the RFC requires raw-name UTF-16 code-unit order — divergent even on plain-ASCII keys — and it emits i64 verbatim beyond 2^53−1, which a conforming verifier cannot reproduce. Since C5’s whole point is independent external verification, chain-v2 pins: a conforming implementation (serde_json_canonicalizer, selected per the plan’s D-CANON) plus a recursive I-JSON validation layer that rejects non-finite numbers and integers with |n| > 2^53−1 before hashing (conforming serializers silently ROUND oversize integers through f64 — a semantic collision an audit chain must refuse; floats pass). The legacy call sites' migration is #1281 (out of chain-v2’s scope).

C1’s "chain_family/service/database identity is enumerated" is made columnar and hash-bound: every v2 event row and every anchor manifest carries chain_source (canopy-security | canopy-tanf | canopy-medicaid), bound into the C2 preimage, validated by the append functions against the chain_sources registry (source → instance, installed at genesis), with the source literal baked into each service’s rendered append function at migration-authoring time — never caller-supplied at runtime. The two FTI instances therefore remain distinct end-to-end, per C1’s original intent.

The C4 state set open|closing|anchored|closed becomes installing|active|closing|anchored|closed: genesis installs epoch 0 as installing (structurally NOT appendable — the append functions require state = 'active' AND the epoch equal to the topology pointer’s current_epoch); the single installing → active transition is the chain_epoch_activate CAS, executed at the coordinated-downtime cutover (#1279) only after the genesis anchor is externally notarized and confirmed. This closes the appendable-before-notarization window. The transition executor for closing → anchored → closed remains #1280’s, under the pinned epoch→head lock-order protocol.

C5’s anchor kinds gain a fifth: genesis — the notarized empty-genesis manifest (anchor_seq = 1, previous_anchor_hash = the zero sentinel), distinct from periodic_tip so the trust root is self-describing. The C5 requirement that "the amendment pins the authority + credentials" is discharged in stages: the substrate pins the manifest bytes, the local record, and the DTOs; the concrete authority/credential selection (#1278) is RATIFIED by a further amendment here when it lands.

C2’s "hash + store the normalized bytes" reading is corrected: PostgreSQL JSONB stores a semantic value, not bytes. The pinned semantics are refetch-and-recanonicalize — both the insert side and every verify side canonicalize the same refetched JSONB value, which is deterministic for any JSON because normalization happened once through $1::jsonb at intake.

Domain tags: canopy-chain-v2 (event-hash preimage), canopy-chain-v2/routing/v1 (shard routing), canopy-chain-v2/genesis (empty-head hashes), canopy-chain-v2/anchor (anchor manifests). Independent versions — a byte-level change bumps ITS OWN version, never the others: event_hash_formula_version=2, routing_version=1, genesis_version=1, anchor_manifest_version=1, anchor_signing_version=1. The frozen KAT corpus (substrate MR-1) pins all of them; changing a frozen vector requires the corresponding version bump plus an amendment here.

C3 said "server-minted row id for direct ingest"; the row PK is append-minted and excluded from the hash, so it cannot be the routing id. The corrected recommendation is the server-minted envelope event_id (the ingest endpoint already mints it); the binding pin remains #1207’s, and shard_for is id-agnostic.

The verification role’s read-only posture governs the chained data (especially the cross-database FTI reads: projection views only, never SELECT *); its own verification-STATE tables (checkpoints/runs/incidents/anchors, all homed in canopy_security) are written by design — exclusively through guarded SECURITY DEFINER CAS/transition functions, never direct DML, with incident RESOLUTION authority separated from the background verifier entirely.

The cutover cannot precede the append transport: at cutover direct DML becomes forbidden and v1 writers cannot call the v2 functions, so nothing could write. The corrected sequence — substrate (#1246) → #1207 writers (dormant) + #1205/#1206 verifiers (dormant) + #1278 anchor authority delivered → #1279 coordinated-downtime cutover LAST, depending on all of them (go/no-go gate in #1279). The parent plan carries the same correction.

Amendment 6 recommended the server-minted envelope event_id for direct ingest and left the binding pin to 1207. BOUND: direct ingest (POST /v1/security/audit/ingest) routes on the envelope id the endpoint already server-mints; -queue events route on the publisher-minted envelope id. Both ARE the payload’s hashed event_id — the routing id is hash-bound and immutable, and the staging store enforces the payload/column equality structurally (a CHECK constraint; a divergent pair is unrepresentable — lands in #1207 MR-2).

One chained row per event_id per family table, enforced by a UNIQUE expression index on the live v2 table ((canonical_event_payload→>'event_id') — lands in #1207 MR-2; the FTI tables already discharge it via the payload-derived primary key). This is the PERMANENT replay identity: staging rows are dequeued on chaining and inbox dedup rows are reaped after seven days, so without it a sufficiently-late redelivery would double-chain. Exact late replays (equal canonical-payload digest) are silently absorbed; same-id/different-content replays fail closed and are observable. Residual, documented: once #1208 archival moves rows out of the live table (years later, per retention) the live index no longer covers them — the replay horizon is days, the archive horizon years; #1208 carries the note.

The drain work unit — claim staged rows (FOR UPDATE SKIP LOCKED) → head lock → hash → append → dequeue — is ONE database transaction, so ownership IS the row locks: there is no lease table, TTL, renewal, or steal path (the lease-steal defect class is unrepresentable). Release on failure is bounded by the pinned per-transaction lock_timeout / statement_timeout and the session idle_in_transaction_session_timeout — NOT by wall-clock lease expiry, and not "instant" under a wedged session (PostgreSQL releases a partitioned session’s locks only when the server notices; the timeouts bound that window). This satisfies Amendment 5’s pinned drainer invariants — "multiple replicas + manual triggers cannot overlap ownership" and "restart resumes at every batch boundary, in O(batch) memory" — discharging a delegated byte-level decision, never overriding "leased". (Contrast: the transactional-outbox drainer needs timed leases because its AMQP publish necessarily happens outside the claiming transaction.)

202 = "accepted + durably staged" (chained asynchronously by the per-shard drainer), with an {event_id} receipt body. Dormant until the #1279 cutover flips chain_v2_append_enabled; until then the endpoint keeps today’s synchronous-append 202.

The substrate’s checkpoint functions are reshaped (pre-1.0, dormant — the old signatures are DROPPED, never overloaded): chain_checkpoint_acquire is the ONLY row-creating, fence-raising path — it mints a unique per-acquisition token under the checkpoint row lock, respects an unexpired foreign lease, and REQUIRES an initial cursor on first acquire (an existing checkpoint row can never hold a NULL cursor hash); chain_checkpoint_advance becomes existing-row-only, exact-token-bound, and cursor-monotonic. Run recording and incident latching are token-guarded functions — a fenced-out worker cannot write state, latch a breach, or contaminate runs; incident dedup is a partial-unique DB invariant. Wall-clock expiry gates only WHEN a claim may be contested; the token is the correctness mechanism. This is deliberately NOT Amendment 7’s structural lease: the verify read (a cross-database SELECT for FTI) and the checkpoint write cannot share one transaction — Amendment 7’s own stated precondition for timed leases.

Beyond seq/linkage/formula/hash: closed per-family key/type sets; derived-column ↔ payload consistency (a mutated typed projection column breaks verification even when payload + hash are internally consistent — the field-coverage tables made executable); routing placement (shard_for(routing id, shard count) == shard_id — discharging the substrate plan’s delegated placement validation); stored chain_source vs the registry; FTI row.id == payload.id. Persisted-data decode/canonicalization failures are integrity findings (latched), never transient errors.

Archive ∪ live with the seam at archived_through — archived-row mutation is detected by the scrub, not deferred. Cycle targets and the boundary-cursor reset persist through ONE atomic acquire write (restart resumes the same cycle to the same fixed target); cycle_completed_at is the coverage stamp, distinct from activity. Only the purge-boundary manifest machinery remains with the archive children (#1208/#1247); a purged boundary row fails closed until it lands.

trusted_manifest_ref is written only after the verifier compares an anchor’s decoded canonical bytes against the anchor row’s own metadata (the seven row columns: seq, kind, epoch, previous hash, manifest hash, instance, family — chain_anchors carries no source column; the decoded source is hash-bound inside the manifest bytes and checked against the topology), the topology, and the chain (per-tip prefix consistency, anchor fetched BEFORE heads, one refetch before latching divergence — a newer anchor confirmed mid-check is normal operation). A newly confirmed but never-checked anchor authorizes nothing. Status consumes a manifest-age threshold that #1278’s cadence/SLO tightens; the residual window — a rewrite strictly newer than the trusted manifest — is exactly what periodic-tip cadence bounds.

chain_anchor_append (and the pending→submitted/failed transitions) move to a dedicated emitter role; canopy_chain_verify keeps the submitted→confirmed transition only (verifier-confirmed, C5). One credential can no longer both fabricate and confirm an anchor.

breached > error > stale > unknown > verifying > healthy; stale derives at read time from checkpoint activity stamps, scrub completed-cycle stamps, tail lag, and trusted-manifest age; the backlog inputs (raw staged/parked from the #1207 snapshot, inbox parks, DLQ depth) hold status at best verifying, and an input that has never sampled successfully blocks healthy. HTTP: healthy/verifying → 200; unknown/stale/error/breached → 503 — the #1245 fail-closed-by-status-code posture, preserved on the same paths.

202 = "job accepted" with a {job_id, poll_url} body over a guarded queue (enqueue/ claim/finalize/reap functions only — no raw DML; claim lease + heartbeat + attempts
stale-claim reclaim; closed public error codes; requester-scoped polling). Manual coverage is all-or-nothing (coverage_incomplete, never a partial ok), and incident resolution requires a revalidation run that COVERS the finding’s loop — enforced in the guarded resolve, with the actor recorded from the session, never caller-supplied. The old synchronous FTI verify path is deleted (pre-1.0 Changed).

Amendment 8 said the fail-closed HTTP posture was "preserved on the same paths"; the paths themselves are now REPLACED (pre-1.0 Changed). One namespace carries the whole surface: GET /v1/security/chain/status?family=[&service=], POST /v1/security/chain/verify, GET /v1/security/chain/verify-jobs/{id}, GET /v1/security/chain/attest?event_id=&family=[&service=]. The historical GET /v1/security/verify-chain, POST /v1/security/fti/chain-verify, and GET /v1/security/fti/chain-status are deleted. What is PRESERVED is the posture — healthy/verifying → 200, everything else → 503 with a parseable body — and the #1245 guarantee that a latched legacy FTI breach stays visible: the unified FTI status arm ORs the v1 latched row into breached until #1279 drops the table (the earlier classification of that read as a compat cut is withdrawn — it is a safety invariant). Every API operation has a canopy CLI subcommand (ADR-007 parity).

loop_kind gains 'family': one family-level checkpoint row per (instance, family, epoch) at shard_id 0, PK-distinct from shard rows by loop kind. Pass order is fetch_topology (read-only; supplies the instance) → FAMILY lease → halt gate → manual jobs → manifest + census under the family token → per-shard tail/scrub under shard leases. The family lease structurally serializes the census and manifest check (one replica per family per cadence), makes the holder the only servicer of that family’s manual jobs, and is the authority for family-scoped writes: run recording validates the FAMILY token; family-scoped findings (missing/unexpected heads, genesis mismatch, manifest divergence, boundary-unavailable at init) latch under it with a NULL shard. Scale-out is BY FAMILY — this trivially discharges the no-overlap invariant at the pass level while shard tokens remain the write-fencing mechanism underneath. The trusted manifest reference lives on the FAMILY row ONLY. The family row carries no cursor at all (shape-CHECKed) — the deliberate, single exception to Amendment 8’s "an existing checkpoint row can never hold a NULL cursor hash"; tail/scrub rows keep that rule unchanged.

The verify role’s raw SELECT on chain_verification_checkpoints is REVOKED — it reads a token-free view, so the ONLY way to hold a lease token is to have minted it via acquire. Takeover is EXPIRY-ONLY for everyone (owner text is display-only; a restarted process waits out its own lease). Acquire/advance take a bounded DURATION (seconds, domain-checked in SQL); expiry is computed inside the locked function — no caller clocks. A token stays valid PAST expiry until a takeover mints a successor: correctness never reads the clock; expiry only opens the contest window, and a finding produced by a long batch is never lost to it. The lease ≥ 3× statement-timeout rule is a LIVENESS heuristic, not a correctness proof.

Scrub cycle-start is a cursor-CAS inside acquire (the reset applies only if the stored cursor equals the caller’s expected value — a delayed worker’s stale cycle-start loses under the row lock, and the CAS stamps cycle_started_at). chain_checkpoint_advance proves its stamps relationally: on scrub keys cycle_completed_at only when the presented cursor EQUALS the stored target (seq and hash, checked in-function); trusted_manifest_ref only when it references a CONFIRMED anchor row of the same (instance, family), and only on the family key. The one non-proven stamp is named: on the family key cycle_complete is the census-cadence marker — token-gated observability with no relational witness, and no status rule reads it. Scrub staleness falls back to cycle_started_at when no cycle has ever completed — a first cycle that never finishes goes stale. Error state is per-scope: runs carry an optional shard, and a shard’s error clears only against that shard’s own success stamp.

chain_incidents stores detected_loop_kind ∈ {tail, scrub, family} at latch — never inferred from the kind. Resolution requires a MANUAL revalidation run (job-linked; scheduled runs never resolve) of the DETECTED loop covering the incident’s scope, enforced in the guarded resolve alongside the session-recorded actor. Dedup is a UNIQUE NULLS NOT DISTINCT partial index over (instance, family, epoch, shard, kind, detected loop) — family-scoped incidents with NULL position dedup correctly. The kind vocabulary is a closed CHECK constraint; evidence is a bounded JSON object; the verify role reads incidents only through an evidence-free view (the incident-admin role keeps the full read).

Job rows carry the durable work definition — requested loop (tail | scrub | family-full), optional incident reference, and the instance/epoch/ target vector captured ONCE at first claim, so a reclaim resumes the SAME targets. Claims are DB-minted-token-based (heartbeat/finalize validate the token — the checkpoint ABA fix applied symmetrically) and target-scoped (each family task claims only its own family/source). One active job per target (a second enqueue returns the active job’s id → 409); a bounded queue refuses at capacity; enqueue is refused entirely for unconfigured verifier targets (no phantom queue). Runs record mode ∈ {scheduled, manual} + the job id: manual outcomes NEVER feed the status machine in either direction. Reap has an in-function ≥ 7-day floor, terminal-only, with incident-referenced rows exempt until resolved.

Every batch walk fetches archive ∪ live as ONE UNION ALL statement with a side tag — one MVCC snapshot; a concurrent archive move can never tear a read, and a row present on both sides latches as a duplicate. Rows are side-checked against the boundary (wrong-side and archived-orphan rows latch). The structural census extends to the archive side: identity-filtered whole-range counts, rows above the boundary, and foreign instance/epoch/shard detection cover archived rows too. The byte budget always admits at least one row; a single row above a hard 4 MiB ceiling (double the ingress body cap) latches malformed_row.

RFC 8785 renders numbers through f64, so values that collide in f64 (2^53 vs 2^53 + 1) canonicalize identically and hash recomputation alone cannot detect the mutation class. The fence closes it: a raw number-token scanner in canopy-chain accepts a token iff its DECIMAL VALUE equals the decimal value of the shortest round-trip representation of its f64 — value-level, so PostgreSQL’s jsonb numeric reformatting passes while every f64-collision mutation is caught. It runs verify-side over the fetched payload text (violation = malformed_row) and intake-side over raw ingest bodies and the staging consumer’s delivery bytes (no new collision-class value can enter). serde_json’s `arbitrary_precision feature is deliberately NOT used — cargo feature unification would change number parsing workspace-wide.

The authority is an S3-compatible object store reached through canopy-store (production: AWS S3 with versioning + Object Lock compliance-mode retention + split writer/reader/enumerator IAM + a policy requiring conditional writes and denying delete/overwrite; devstack: Garage, functional-not-adversarial). Its trust does NOT derive from local rows: the verifier ENUMERATES the authority (prefix-scoped ListBucket) to learn the true external high-water mark, so a local-DB rollback that hides newer objects is caught as external-ahead-of-local (a breach, family-global anchor_integrity); the confirmer’s per-pass re-list also latches on a beyond-tolerance discrepancy between an object’s LIVE LastModified and its stored authority_time (a local-rewrite signal); read-time status-age stays on the landed local path and is best-effort liveness only (see the status subsection below); and each object embeds the full manifest, so an auditor with only the bucket + the public key reconstructs the anchor chain with zero database access. Object keys carry an unpredictable recovery token namespace ({prefix}/{instance}/{family}/{token}/{seq:020}), not a guessable integer, so a leaked writer cannot preplant future keys; recovery from a burned key is an operator-gated, idempotent, fenced token rotation (chain_anchor_recovery_rotate) performed after the writer credential is rotated, journaled with actor + reason + incident.

The verifier confirms a submitted anchor only after: (1) anchor-chain linkage (seq = prev+1, previous_anchor_hash = prev.manifest_hash, genesis sentinel), walking the frontier strictly in seq order and halting at the oldest unresolved anchor; (2) chain CONTINUITY (the current chain hash at the previous trusted tip seq still equals the previous manifest’s tip hash, per shard, same epoch — tip-seq monotonicity alone does not prove continuity); (3) the full anchor-vs-chain check on THAT anchor; (4) external evidence at the CANONICAL key (derived from the row’s identity + recovery token, never by following external_ref, which must itself equal the derived key, with external_authority equal to the configured id), fetched at the RECORDED version (a new current version or delete-marker over it is a breach even with identical bytes); (5) a semantic byte + JWS check (kid resolved from signing_key_history by the registered_at window covering the object’s authority-set LastModified — a retired key verifies old anchors but authorizes no new ones; NOT a seq window, which cannot cover genesis seq-1 and collides across the three families' restarting seq spaces). The confirm transition is a CAS under the family lease fence (expected predecessor
checked fingerprint) — no DB transaction spans the S3 I/O, so a lease-lost verifier cannot confirm. The emitter never confirms; the confirmer never emits (grant-enforced) AND runs as a separate deployable (canopy-anchor-emitter) holding only the signing key + writer creds, so a single process compromise cannot forge-and-self-confirm. Divergent external objects always reach the verifier via an append-only chain_anchor_attempts record (recovery never overwrites evidence).

Anchor kid = canopy-chain-anchor:{RFC 7638 P-256 JWK thumbprint} (distinct from the determination PEM-text scheme); ANCHOR_SIGNING_VERSION bumps to 2 as the frozen provisional vectors are re-derived; the public key is retained in signing_key_history (program slug chain-anchor); kid retirement is a registered_at-window check against the anchor OBJECT’s authority-set time (LastModified — genesis-safe, cross-family-correct, non-forgeable by the privileged-DB attacker) so a retired key still verifies old anchors but authorizes no new ones. Object Lock retention is a per-jurisdiction, per-family ruleset value (ADR-006 / ADR-011), bounded below by the family’s federal floor — #1278 introduces the ruleset key that #1208/#1247 reuse. Read-time status freshness is BEST-EFFORT liveness (a frozen frontier / dead confirmer ages to stale in the honest case; a full privileged-DB attacker who both rewrites the local freshness stamp and freezes the confirmer is NOT defeated by read-time status — the anti-rollback guarantee is the confirmer’s external-ahead-of-local enumeration + discrepancy latch, eventual and observer-dependent, plus the independent auditor). Status gains a coverage-lag input (manifest_coverage_lagging) so a compromised emitter emitting fresh anchors with frozen tips still degrades in the honest case; CHAIN_MANIFEST_MAX_AGE_SECS tightens (default 6h, ≥ 3× emit interval); anchor_integrity latches family-global and resolves only via a manual ok family run whose ok outcome is gated on the external arm actually having run. Attestation is honest: behavioral boot probes prove conditional-create, unconditional-overwrite-denial, and credential-scope; control-plane facts (Object Lock, versioning, the bucket policy, split IAM) are provisioning-time operator attestation — production account/bucket provisioning is a separate deployment concern, out of #1278, which delivers the code, config, the attestation harness, and a provisioning-requirements runbook. The epoch_closure (#1280) and archive/purge-boundary (#1208/#1247) confirm-check arms are defined in their own issues; #1278 pins genesis + periodic_tip (unsupported kinds submit + store but confirmation defers them).

Amendment 10 rooted trust in enumeration + signatures + conditional-create + authority LastModified. That defeats a hidden-rollback attacker but NOT a truncation attacker: a privileged writer (or a store admin) can delete the suffix of the chain and, on a store without WORM, recreate a shorter valid suffix — signatures still verify, ListBucket returns exactly what remains, and LastModified is attacker-influenced. Enumeration reports what is PRESENT, never what was REMOVED; the bucket alone has no memory of the expected root. These Amendment 10 assertions are WITHDRAWN:

What Amendment 10 got right and is RE-RATIFIED: the separate emitter deployable (arm split); the append-only chain_anchor_attempts evidence rule; RFC 7638 anchor kid identity + ANCHOR_SIGNING_VERSION → 2; per-jurisdiction/per-family ruleset retention (ADR-006/011); coverage-lag status; and production account/bucket/IAM + signing-identity PROVISIONING as a separate deployment concern (out of #1278).

The load-bearing anti-truncation guarantee requires a WORM / Object-Lock store (AWS S3 / MinIO / Ceph RGW in compliance mode, with versioning + a deny-delete/overwrite bucket policy) as the C5 append-only authority: on such a store, deletion and truncation are impossible by construction. The CODE stays portable (canopy-store over object_store, any backend), but ASSURANCE is TIERED:

WORM enablement is CONTROL-PLANE: the pinned object_store API cannot set or read Object Lock, so it is provisioned and attested out of band (IaC / operator attestation + a provisioning-requirements runbook). Boot data-plane probes attest only BEHAVIORAL facts (create-if-absent enforced, unconditional overwrite denied, delete denied, credential scope).

Because the bucket cannot testify to what was removed, the trust root is: the WORM store (can’t-delete) + a PINNED genesis/root catalog (the genesis manifest hash + the authoritative chain_instance_id, published out of band) + an authenticated instance-succession and key-transition lineage. An auditor walks FORWARD from the pinned genesis over the WORM contents and detects truncation (tip below the pinned/known tip) or substitution (linkage / signature failure). Open sub-decision, resolved in the replacement plan + #1279: whether the pinned catalog’s authoritative home is the public repo release record or an operator-signed out-of-band record — both preserve the guarantee.

ANCHOR_MANIFEST_VERSION → 2: the signed preimage gains recovery_token (the object location is signed, so an object cannot be replayed under a fresh token path) and issued_at (freshness, coverage, and kid retirement ride SIGNED time, never the mutable LastModified). ANCHOR_SIGNING_VERSION → 2 (header freeze; KAT corpus regenerated + negative KATs). The confirmer/auditor derives each next canonical key from the hash-linked chain and probes it by HEAD (seq+1, seq+2, … to the tip) — a bounded FORWARD hash-walk, not an unordered list/offset scan (object_store list ordering is not guaranteed and the S3 list path returns version=None, so an offset cursor can permanently miss objects). A separate, capped, periodic prefix LIST detects foreign/spam keys but is NOT the frontier.

Confirmation is a CAS holding the checkpoint-row lock through commit (no TOCTOU) with the FULL fingerprint (manifest hash, canonical key, external authority, version/etag, kid, issued_at, attempt_id) and external JWS == stored == attempt; no DB transaction spans the store I/O; every remote op is timeout-bounded. Emit is fenced (recovery_token == head under the head lock) + idempotent, rides SIGNED coverage state (a store outage cannot append an anchor per tick), and HALTS the family on a transient submission (no N+1 while N pending). Cross-DB genesis extends the existing crash-resumable saga (Phase A target DB → Phase B canopy-security anchor DB → Phase C revalidate; installing until chain_epoch_activate) with an idempotent genesis-confirm step invoked at #1279 — NOT a SQL fence across databases. Key transitions are an externally-authenticated bundle registered through a narrow SECURITY DEFINER function (no table-wide INSERT to the emitter); retirement is the registered_at window vs the SIGNED issued_at. Incident gating is reachable (manual jobs run before the halt gate; resolution is SQL-enforced against an external-check witness). The census uses a DEDICATED guarded cursor (not the cursor-free family checkpoint row); retention is the ruleset value realized as the WORM bucket-default (per-object retain-until is impossible via object_store).

C5 (Amendment 5) requires an external APPEND-ONLY authority. A generic object store is NOT append-only; only a WORM-configured one is — so the "portable, no store features" reframe in Amendment 10 VIOLATED C5, and this amendment restores it: the WORM tier IS the C5 append-only authority, and portability is an assurance tier below it, not a substitute. Alternatives weighed (per the architectural-recommendation protocol): (a) crypto-only portable (Amendment 10) — REJECTED, cannot prove completeness; (b) external witness / transparency-log co-signer — defers the trust root to another append-only service with the SAME requirement one layer out, adding a network dependency + a second operator; held as future hardening, not the #1278 baseline; (c) WORM capability tier (this amendment) — the minimal model that delivers C5 with the pinned toolchain, portable code, and honest tiered assurance. The #1280 (epoch closure) and #1208/#1247 (archive/purge boundary) confirm-check arms remain defined in their own issues; #1278 pins genesis + periodic_tip.