Plan: chain-v2 verifiers — family-leased tail + scrub engine, token-confidential fencing, the unified chain namespace, citation attestation (#1205 audit, #1206 FTI, epic &73)

On this page
NOTE

Implements ADR-014 Amendment 5 C6 (with the C5 manifest-divergence consumer arm) under the Amendment 6 corrections, the Amendment 7 transport bindings, the Amendment 8 verifier bindings, and the Amendment 9 revisions this version introduces (accepted amendments are immutable — Amendment 9 is the formal revision vehicle). Parent rollout: ADR-014 chain-v2 (Step 4).

Review state: v7. v1→v4: three internal contextless rounds. A FIRST external review rejected v4 (~50 integrity/fencing/recovery/activation findings); the v5 rework (W1–W11) was approved, committed (MR !1054, merge a2d2b33f), and corrected by a 0c fold (f6fca4c7). A SECOND external review then REJECTED v5 with ~14 blocking + ~25 high
~10 material findings — token confidentiality, family-scoped fencing, durable jobs, proven health stamps, single-snapshot reads, the number-collision class, and a silently-erased #1245 safety invariant. v7 is the full rework, approved as workstreams X1–X10 (2026-08-01): lease-token confidentiality + honest lease semantics (X1); the FAMILY lease ordering manifest/census/shard loops/jobs under one holder (X2); the detected-loop incident model with NULL-safe dedup and the evidence split (X3); durable token-claimed target-scoped bounded jobs (X4); scrub/status correctness under the lock — cycle-start CAS, PROVEN completion/manifest stamps, first-cycle staleness, per-scope error clearing (X5); one-statement archive ∪ live reads + whole-history census + bounded rows (X6); the JSON number fence CLOSING the collision class, not observing it (X7); legacy FTI breach visibility PRESERVED until #1279 — reversing a wrong v5 cut (X8); the completed wire contract — the unified /v1/security/chain/* namespace, full de-pseudocoded SQL/DTOs, CLI parity, the honest ripple inventory — ratified as ADR-014 Amendment 9 (X9); and scheduler/config/constraint/test closure (X10). file:line verified against main == f6fca4c7 (0b123d38 → f6fca4c7 is docs-only — the v5 plan commits themselves).

Status

Step Description Status

0

Plan lifecycle: v5 committed (!1054 + the 0c fold) → SECOND external review REJECTED → X1–X10 rework approved (2026-08-01) → apply v7 (this text) + ADR-014 Amendment 9, commit direct to main (docs-only, full battery) → external re-review of the COMMITTED v7 artifact → user sign-off → 0a tracker pass (now including the #1205/#1206 AC updates + filing the incident-UI issue). Implementation starts only after re-review + sign-off.

Done (2026-08-01) — v7 committed at ddecb378; sign-off received; 0a complete (#1205/#1206 reopened + v7 ACs, #1289 filed, tracker notes on #1208/#1247/#1278/#1279/#1280/#1285)

1

MR-1 substrate hardening + verify engine + the number fence (feature/1205-chain-verify-engine, Relates to #1205 + Relates to #1206): the hardening migration set (token-confidential acquire/advance reshape with in-fn PROOFS, the family loop kind, guarded run/incident/resolve fns, the detected-loop incident model, token-free + evidence-free verify views, CHECK matrices, NULLS-NOT-DISTINCT dedup, anchor role/arm split, archive attestation index; FTI preimage views gain id in tanf/medicaid migrations) + canopy_common::chain_verify (single-statement whole-preimage batch walker over archive ∪ live, whole-history census, manifest binding, the VerifierDb token client) + canopy_chain::canon::raw_number_fence (verify-side #1285 closure) + canopy-chain/test-lib additives + engine suites. As-built deltas (living spec): the FTI id binding is check 9’s ALONE (id_mismatch — the derived-column compare deliberately skips id so the ADR-named finding is reachable); the pre-hardening substrate suites' grant/ACL/projection assertions updated to the reshaped surface in the same MR (Files); the task-level engine-inventory entries (head_regression_latches, halt_gate_rechecked_between_batches) move to MR-2’s host suite where verifier_pass exists; newly_confirmed_anchor_mid_check_refetches_not_latches is realized injection-free as manifest_tip_beyond_head_latches_after_refetch; VerifierDb::latch_incident carries an explicit shard: Option<u16> scope param; and the hardening migration CLEARS the C6 state tables before reshaping ("empty pre-cutover" is an assumption an open-CAS devstack can violate — pre-cutover state is definitionally scratch, so the clear makes the CHECK/NOT-NULL installs unconditional, forward-only per ADR-016).

Done (2026-08-01) — implemented on feature/1205-chain-verify-engine; 24-test engine suite + 2×5 FTI two-schema suites + 28 updated substrate tests green; merged 003bfd58 (+ the #1291/#1292 review-fold fixes, merged 26e3601a)

2

MR-2 audit verifier + the unified chain namespace + jobs + web + CLI + perf (feature/1205-audit-verifier, Closes #1205): projections/jobs migration (durable job table + guarded fns), the audit verifier task (family lease → halt gate → manifest/census → shard loops, job-first servicing), the C6 status assembly, the unified /v1/security/chain/ endpoints (status/verify/verify-jobs/attest; GET /v1/security/verify-chain + POST /v1/security/fti/chain-verify DELETED — path count 17), the intake-side number fence, canopy-web typed-terminal-503 client + badge + citation (citation.typ), canopy CLI chain parity commands (ADR-007), cargo xtask perf chain-verify (audit dimension — #1205’s AC is proven BEFORE its close), OpenAPI snapshots, the full doc ripple. *As-built deltas (living spec): chain_incidents_app_v realizes D6’s "position" as position_seq extracted from evidence→>'seq' (the expected/got hashes stay admin-only); a seventh view chain_verify_job_runs_v carries the poll endpoint’s manual-run summary (the scheduled-only runs view structurally excludes them); _app polls jobs through a TOKEN-FREE view, not a raw-table SELECT (the X1 class applied symmetrically — claim_token never reaches _app); chain_job_enqueue additionally validates a revalidation job’s loop COVERS the incident’s detected loop at the door (family-full or exact match) and RAISEs the queue cap under SQLSTATE 54000 for the handler’s 503 mapping; attest serves definitive negatives as 200 + attested: false (only verifier_unavailable is a 503, typed body both ways) and the decision is a pure fn (attest_decision); the D8 request-error matrix is validated manually in-handler so each refusal carries its closed code verbatim; the DLQ-depth input rides a new canopy_mq::metrics::queue_depth probe (file-list addition — the readable twin of the #452 gauge poller); per_family_task_isolation lands as the broken-pool-backoff arm (cross-family isolation becomes testable at MR-3’s second family); the web row banner’s typed matcher is live but anchors only once the events wire carries chain positions (post-#1279 v2 read surface — noted for #1279’s list). Pre-commit review fold (adversarially-verified findings, all applied in-MR): shard-lease contention during a job yields RESUME (Progress, claim retained), never a terminal coverage_incomplete — on a resume pass the "holder" is the job’s own previous pass, and D8’s give-up path is claim-lapse → attempts → crashed alone (coverage_incomplete remains the captured-epoch-no-longer-active arm); the held claim survives transient errors in PassState (only a FENCED heartbeat drops it — attempts never burn on DB blips); covered shards are skipped without budget charge or lease so resume progress is monotonic at any shard-count/budget ratio; a scrub job finishes a mid-flight scheduled cycle at its stored target and then plans a fresh CAS cycle at the CAPTURED target (coverage judged only against the captured vector); family-full records one manual run PER LOOP (tail/scrub whole-family NULL-shard + the family primary) so tail/scrub incidents are resolvable from a family-full job; environment errors past the family lease record a best-effort error run (the D6 error state is producible; pre-lease failures remain staleness-only); ScopeRunInput carries the loop — error clearing is per-(loop, shard), never cross-loop; a latched incident derives breached even when the topology is missing/not-active (D6 rule 1 is unconditional); the runs manual ⇔ job CHECK is one-directional (scheduled ⇒ no job) with the FK ON DELETE SET NULL so reap can delete terminal jobs while their runs survive; the enqueue cap check is advisory-lock-serialized and the unique-violation race arm can no longer return a NULL-id row (40001 → 503 retryable); the D14 consumer arm rides a new canopy_mq::Subscriber::subscribe_raw seam (float-formed collision tokens are invisible post-parse; unpark replays fence the stored rendering — the schema-parked sliver is normalized-not-rejected, documented on the fence); job-poll scoping applies to SERVICE callers only (admin sees all; service_id()’s azp fallback made admin tokens scope as the BFF client) and admin enqueues attribute as `admin:{sub}; a malformed incident_id is the matrix’s 404 unknown_incident (indistinguishable, closed vocabulary — invalid_incident_id never existed on the wire) and the fn’s loop-coverage RAISE maps to 400 invalid_loop; the FTI status arm’s 503 carries the TYPED ChainStatusResponse (unknown/verifier_disabled) like every other 503; CHAIN_JOB_HEARTBEAT_SECS drives a real per-batch claim pulse. The evidence-shape defect this review surfaced in MERGED MR-1 code (evidence_json emits prose, not D7’s positions+hex — position_seq dead) is #1292, fixed on its own branch with #1291 before this MR merges (merged in !1056; on the rebase this MR adds the end-to-end pin — a real tail-walk hash_mismatch feeding breached_position through the real writer, breached_position_rides_the_real_evidence_writer).

Done (2026-08-01) — implemented on feature/1205-audit-verifier; 20-test host suite + engine/substrate extensions + web/CLI/perf/docs green; merged 06b4a3b1 (!1057; #1205 closed; audit 2M perf evidence all-green)

3

MR-3 FTI verifier (feature/1206-fti-verifier, Closes #1206): tanf/medicaid verify pools + per-family tasks, the FTI status/attest arms WITH the preserved legacy latched-breach input (#1245 safety invariant — X8), GET /v1/security/fti/chain-status deleted (final path count 16), the v1 DTO chain narrowed to the breach-bit reader, archive-aware in-vivo proof, dormancy + paused-time tests, the FTI perf dimension, docs. As-built deltas (living spec): the X8 input is a FIFTEENTH closed status reason legacy_breach_latched — its own domain/wire variant riding rule 1’s breached precedence (StatusInputs.legacy_breach: bool), never a fabricated incident_id (the response carries incident_id: None for a legacy-only breach), and it holds UNCONDITIONALLY: a dormant/unconfigured FTI service with a latched v1 row still serves breached/legacy_breach_latched (the old endpoint’s dormant-visibility guarantee survives the endpoint); the breach bit is EXISTS(…​ broken) over ALL rows, not the latest row — a later clean row can never mask a latched break (the #1245 no-clean-re-run posture made structural); the service assembly generalized as assemble_family_status + a FamilySources descriptor (chain pool + heads relation + samplers + legacy bit) with audit reading heads from chain_status_v on the app pool and FTI from chain_heads on its verify pool (D6), FTI wire backlog: None — typed applicability, not zeroed; resolve_fti_position probes the id PRIMARY KEY on both preimage sides; the X10 (family, source) topology-slot check is a PRE-LEASE HostError::TopologySlotMismatch in verifier_pass — a swapped tanf/medicaid URL pair fails that family loudly on every pass with zero checkpoint writes; per-family dormancy is pool-presence (ChainVerifyRuntime::fti_pool), so enqueue/attest refuse per service; chain_archive_prefix{audit,fti} remain #1208/#1247 RAISE stubs, so the archive-aware in-vivo tests and the perf harness superuser-move prefixes directly (the documented idiom); the perf harness gained a --families audit,fti matrix (FamilyDim descriptor; FTI corpus on the devstack TANF database via chain_append_rows_fti; the job-claim plan gate stays audit-dimension-only since chain_verify_jobs lives in the security schema — logged, not silent); paused-time coverage rides MR-2’s run_family_loop tests unchanged (the loop is family-generic), with FTI-specific dormancy pinned at the HTTP layer.

Done (2026-08-02) — implemented on feature/1206-fti-verifier; 92 security + 41 CLI + 292 library tests green; FTI 2M perf evidence all-green (catch-up 30.2k/38.9k rows/s, RSS 80/147 MiB); merged 7c9a4386 (!1058; #1206 closed)

4

Post-merge docs close-out: verify auto-closure of #1205 + #1206 and post the closing comments (SHAs); parent Step 4 → Done; this plan → Archive + nav; perf cells posted on #1279. (Docs-only, committed direct to main per git-workflow.)

Done (2026-08-02) — #1205/#1206 auto-closure verified + closing comments posted (impl + merge SHAs); audit + FTI perf cells posted on #1279 with the v1-retirement scope reminder; epic &73 updated; this commit archives the plan (nav → Archive)

Epic &73
Issues #1205 (critical) + #1206 (critical) — blocked by #1246 (Done), #1236 (Done), #1207 (Done); both block #1279; #1285 (number-boundary adjudication) is CLOSED by this plan’s D14 fence
Branches v7 rework commits direct to main (docs-only), then per the Status table

Local cargo xtask validate runs before each MR’s first push.

Step 0 — lifecycle

Order (v7 rework round): apply X1–X10 to this artifact + append ADR-014 Amendment 9
parent-plan/architecture.adoc/CHANGELOG.adoc touches → ONE single-agent coherence pass (self-consistency after the large rework — not a new review round) → commit direct to main (docs-only, full battery) → external re-review against the COMMITTED artifact → user sign-off → 0a → MR-1.

  1. 0a — tracker reconciliation (post-sign-off, before any code). FIRST action: file the incident-resolution UI follow-up issue (out of plan scope but tracked in GitLab, per plan-lifecycle). Then:

    • #1205 + #1206 own ACs: update both issues' acceptance criteria to the v7 surface — the unified /v1/security/chain/* endpoints (the AC text still names the deleted paths), the path counts (17 → 16), the family-lease model, the durable-job manual verify, and the #1285 fence dependency — BEFORE any implementation MR references them.

    • #1208 + #1247: the verifiers cover archived rows (scrub + attestation + census span archive ∪ live); ONLY the purge-boundary manifest machinery and the movers remain the archive children’s. A purged boundary surfaces as fail-closed boundary_unavailable.

    • #1280: #1205/#1206 verify the single currently-active epoch; closed/closing-epoch verification and cross-epoch genesis-anchor linkage extend with the rollover executor.

    • #1278: the anchor-consumption contract — the verifiers CHECK anchors and bind status/attestation to the last verifier-checked manifest; emission + submission + the authority integration are #1278’s, WITH the role/arm split (D2): the emitter role owns chain_anchor_append + the emitter transition arm; the verifier keeps chain_anchor_transition_confirm only. The D5 confirmer-contract note is the #1278-facing interface. Status consumes a manifest-age threshold #1278 tightens.

    • #1285: closed by D14 — note the disposition (verify-side fence MR-1, intake-side fence MR-2) and close on MR-2’s merge.

    • #1279: go/no-go additions — verify LOGIN + pool URLs + flag; delayed first tick; unknown → verifying → healthy after genesis-anchor confirmation with zero backlog; badge in vivo; attestation gated on the first periodic tip; the legacy fti_chain_verifications TABLE drop retires the X8 breach-bit reader.

  2. 0b — the v7 plan commit (docs-only, direct to main): this plan (already nav-linked) + ADR-014 Amendment 9 + parent plan (Amendments 8–9 wording) + architecture.adoc
    CHANGELOG.adoc.

  3. 0c — external re-review runs against the COMMITTED artifact. Implementation MRs start only after it passes AND the user signs off.

ADR-014 Amendment 9 (committed with this text — full content in the ADR, summarized): the unified /v1/security/chain/* namespace (the two historical status paths + the FTI verify POST deleted, pre-1.0; Amendment 8’s "preserved on the same paths" sentence formally revised); the FAMILY lease + pass ordering; lease-token confidentiality
expiry-only takeover + duration-bounded DB-computed expiry; PROVEN health stamps (cycle-start CAS, relational completion/manifest proofs, first-cycle staleness); the detected-loop incident model (stored loop, NULL-safe dedup, closed vocabulary, the evidence split); durable target-scoped token-claimed jobs with manual/scheduled run separation; single-snapshot archive ∪ live reads + the whole-history census + bounded rows; the JSON number fence (#1285 closed); legacy FTI breach visibility preserved until #1279; CLI parity (ADR-007) for the chain surface.

Context (recon facts — verified file:line at f6fca4c7)

Contract (ADR-014 Amendment 5 C6 adr-014-fti-audit-hash-chain.adoc:479-499; C5 :469-477; invariants :527-546; Amendment 8 :749-830):

  • Two loops per (instance, family, epoch, shard): tail (fixed captured (target_seq, target_hash), bounded batches, expose verified_through + lag) and historical scrub (bounded, resumable, its OWN fixed trusted target; over the FULL retained history). v7 adds the family pass level above them (D3).

  • MUST reject: missing/unexpected shards or heads; duplicate/missing/noncontiguous seq; wrong previous_hash/formula version; a head that is not its terminal row; rows beyond the head; invalid genesis/epoch-anchor linkage; orphan heads or rows (v7: on BOTH sides of the archive seam); divergence from the latest external manifest.

  • States unknown|verifying|healthy|stale|error|breached; separate tail/scrub coverage/freshness PER SHARD; stale computed at READ time; breach latched until authorized actor + reason + evidence + revalidation; nonzero staging backlog never healthy; status DTOs in crates/canopy-contracts-security/src/{chain,fti}.rs replaced; checkpoint identity (instance, family, epoch, shard, loop-kind)
    fence/CAS; manual verify = job id + polling URL.

  • Citation: event-specific coverage or a dedicated attestation endpoint; fail-closed for newer-than-checkpoint / unknown / stale / error / breached.

  • C8: verification reads the hashed-preimage PROJECTION, never SELECT *; verification-state writes go through guarded SECURITY DEFINER fns; incident RESOLUTION authority separated from the background verifier (Amendment 6 :682-688). v7 extends C8 inward: the verify role itself loses raw SELECT on the checkpoint table (the token column) and the incident table (evidence) — D2.

Substrate as-built (#1246 20260910000000_chain_v2_substrate.sql; #1207 20260930000000_chain_append_staging.sql) — what v7 reshapes:

  • chain_verification_checkpoints (:291-307): PK (instance, family, epoch, shard, loop_kind), loop_kind CHECK ('tail','scrub') (:296 — gains 'family'), no CHECKs otherwise, NULL cursor hash representable, no token column. chain_verification_runs (:309-321): loop_kind CHECK at :314 (gains 'family'), no shard_id/mode/job_id, no useful index. chain_incidents (:323-340): nullable epoch/shard (:327-328), open kind, no dedup invariant, no detecting-loop column, resolution fields caller-supplied.

  • chain_checkpoint_advance (:583-620): INSERTs absent rows with a caller-supplied fence, accepts any p_fence >= fence from ANY caller, never checks owner, permits cursor regression — an open write path. Reshaped in D2.

  • chain_anchor_transition (:688-718): one fn, all edges, caller-supplied JWS/kid/refs — a single credential can fabricate AND confirm. Split in D2.

  • Grants: canopy_chain_verify = raw SELECT on ALL THREE C6 tables (:863 — the checkpoint SELECT leaks every replica’s lease token, the core X1 finding; the incident SELECT leaks evidence), INSERT runs (:864, revoked), EXECUTE checkpoint-CAS / incident-latch / anchor append+transition (:865-868 — the append grant and the emitter arm move to the emitter role); on tanf/medicaid strictly read-only (tanf :462; chain_status_v is NOT in the FTI verify grant — FTI head reads use chain_heads on the verify pool). canopy_security_app has NO C6 SELECT (:859 vs :863) — the D6 projections close that. Login carrier canopy_security_verify is NOLOGIN until #1279 (:56-57).

  • Preimage views: audit (:361-377) exposes position + hashes + chain_source
    canonical_event_payload + ALL typed derived columns (incl. event_id); FTI (tanf :192-204) exposes the same MINUS the hash-bound id PK — MR-1’s tanf/medicaid migrations add id to both FTI views (live + archive). Archive tables copy constraints but NOT indexes (LIKE copies none beyond re-declared UNIQUEs; :242-248, chain_append_staging.sql:138-139) — MR-1 adds the archive-side attestation indexes.

  • Genesis shape: heads pre-created last_seq = 0, last_hash = empty_head_hash(…​) (crates/canopy-chain/src/genesis.rs:54-77); epoch 0 installing until chain_epoch_activate; genesis anchor tips all last_seq = 0 (:135-139).

  • #1279 RENAMES the *_v2 tables at cutover — every request-path read here is view-mediated and survives.

Hash reproduction + the number hole (Rust; no SQL verify fn exists): ChainEnvelope preimage keys (envelope.rs:59-72); event_hash = SHA-256 over canonical_bytes (I-JSON validated RFC 8785, canon.rs:23-26); refetch-and-recanonicalize pinned by Amendment 6 (:625-630). canon.rs:48-70 (validate_number) range-checks i64/u64 and passes floats — so a FLOAT-FORMED token whose integer value sits near 2^53 (decimal or exponent syntax; bare over-range integer tokens are refused by the range check), mutated by ±1, canonicalizes to the SAME bytes (both render through f64), and hash recomputation alone CANNOT see the mutation. The second external review is right that a KAT merely observes this class; D14 CLOSES it with a value-level raw-token fence, and #1285’s adjudication resolves to that fence. AnchorManifest (anchor.rs:85-108) has encode + hash but NO decode — D1 adds from_canonical_bytes. shard_for (routing.rs:24-35) is the placement function the engine recomputes. NOTE: serde_json’s `arbitrary_precision feature is deliberately NOT used — cargo feature unification would silently change serde_json::Number behavior workspace-wide; the fence is a self-contained scanner.

#1207 hand-off (backlog inputs): ChainStagingSnapshot { staged, oldest_staged_age_secs, parked, sampled, degraded_reason } (chain_staging/health.rs:60-72), sampler always-on 30s (:208-220); staged rides RAW (:9-10); sampled starts false (:48-50) — an unsampled input BLOCKS healthy. Inbox parks: newer-schema envelopes only (#1131, subscriber.rs:923); count scoped queue_name = "canopy-security.audit" (a bare literal at main.rs:71; MR-2 extracts the constant). DLQ depth: passive queue_declare (metrics.rs:76-81 precedent) on dlq_queue_name("canopy-security.audit") via the subscriber’s ConnectionManager (available in boot, main.rs:111). Intake surfaces for the D14 fence: ingest_audit_event (api/mod.rs:236-265, POST /v1/security/audit/ingest) + the staging consumer (chain_staging/stage.rs — the home of the oversize-integer poison filter the fence extends).

v1 state being replaced — and the piece that SURVIVES (#1245 containment): all three chain endpoints hard-503 with ChainStatusInterim (api/mod.rs:553-568,590-605,699-736, 754-769); verify_chain is test-only (store/mod.rs:434-448); the FTI verify loop is already gone (#1245; main.rs:92-102 is the removal comment); fti_chain_verifications has readers but no writers (store/mod.rs:332-348). The FTI status handler deliberately keeps a latched v1 breach visible (api/mod.rs:676, pinned by the seeded-breach test tests/security_test.rs:683) — that is the #1245 "a breach is never silently swallowed" SAFETY INVARIANT, not compat, and v7 PRESERVES it until #1279 drops the table (D10; the v5 text calling it a compat cut is withdrawn). The rest of the v1 chain dies as planned: canopy-web still deserializes ChainVerificationResponse (also consumed by the test-lib client, crates/canopy-test-lib/src/clients/security.rs:253-263) and renders "Unable to verify chain" (stream.rs:148-183); the citation PDF hard-fails 502 (api/audit_log.rs:472-545). Wire shapes are pre-1.0: replacements are CHANGELOG Changed entries, every consumer migrated in the same MR. ChainStatusInterim/InterimChainState survive ONLY because run_archive (#1208’s untouched surface) still returns them (api/mod.rs:654-670); ChainStatusInterim.last_verification + FtiChainVerification(+Row) die in MR-3, with the store reader NARROWED to the breach bit (D10); the table drop is #1279’s.

Downstream consumers the second review surfaced (all in the Files ripple): the canopy-web generic client treats every non-2xx as Err and RETRIES 503s with discarded bodies (services/canopy-web/src/clients.rs:190) — breached could never render without the D9 typed-terminal-503 path; the CLI hardcodes the deleted status path (tools/canopy-cli/src/cmd/security.rs:66) — ADR-007 parity commands land in MR-2; the citation template (rulesets/georgia/notices/audit/citation.typ:111) and the backup-restore runbook (docs/modules/ROOT/pages/runbooks/database-backup-restore.adoc:453) reference the old paths/trigger; rbac-matrix.adoc:99 names the deleted FTI endpoints; plus auditor-handbook.adoc, nist-architecture-mapping.adoc, user-testing-guide.adoc, api/canopy-{tanf,medicaid}.adoc, the audit section Plugin.toml, and tests/e2e/specs/audit-rail.spec.ts.

Precedents: pass/idle-skip/rotation/config-domain from the #1207 drainer (chain_staging/drainer.rs:481-513, config.rs:110-229); delayed first tick (detection.rs:28-41); EphemeralSchema::new_for_<service> one-schema-one-pool constructors (db.rs:304-324) — FTI verify tests build TWO; SET ROLE grant probes; xtask perf home (xtask/src/cmd/perf.rs); NO secrecy crate in the workspace — pool URLs use the workspace settings pattern (String field + Debug redaction).

D1 — the shared verify engine (canopy_common::chain_verify, MR-1)

Home crates/canopy-common/src/chain_verify.rs (sibling of chain_append.rs; canopy-chain stays pure). Reused later by #1208/#1247 (purge-boundary verification) and #1280 (closed-epoch verification).

/// Which projections a family reads — ALWAYS view-mediated (C8 + rename
/// stability): audit → audit_hashed_preimage_v / _archive_v; fti →
/// fti_hashed_preimage_v / _archive_v (which gain `id` in MR-1).
/// EleGrant → UnsupportedFamily.
pub fn preimage_views(family: ChainFamily) -> Result<(&'static str, &'static str), ChainVerifyError>;

/// One row as read from a preimage view — the FULL projection. The payload is
/// fetched as TEXT (`canonical_event_payload::text` in the SELECT list — the
/// views themselves are unchanged) so the D14 number fence sees the stored
/// rendering BEFORE any serde normalization; it is parsed into `Value` only
/// after the fence passes.
pub struct PreimageRow {
    pub chain_seq: i64,
    pub previous_hash: EventHash,
    pub event_hash: EventHash,
    pub chain_source: ChainSource,
    pub formula: u16,
    pub side: RowSide,                // Live | Archive — the union tag (D1a)
    pub payload_text: String,         // fenced (D14), then parsed
    pub payload: serde_json::Value,   // STRUCTURAL-VALUE: canonical JSON, re-hashed after validation
    pub derived: DerivedColumns,      // per-family enum: every typed hashed column the view exposes
}
pub enum RowSide { Live, Archive }
pub enum DerivedColumns { Audit(AuditDerived), Fti(FtiDerived) }
// AuditDerived: event_id, event_type, event_timestamp, user_id, user_role, action,
//   resource_type, resource_id, source_service, household_id, metadata (fetched as
//   TEXT + fenced like the payload, then Value — STRUCTURAL-VALUE).
// FtiDerived: id, accessed_by, accessed_at, purpose_code, data_elements_accessed,
//   originating_system, action, resource_type, resource_id.

#[derive(Debug, thiserror::Error)]
pub enum ChainVerifyError {
    Database(#[from] sqlx::Error),        // availability — retry/backoff
    Topology(canopy_common::chain_append::ChainAppendError), // fetch_topology seam (explicit wrap, not From)
    Reject(#[from] VerifyReject),         // integrity — latch material, NEVER retried
    UnsupportedFamily { family: ChainFamily },
    Fenced,                               // lost the token/CAS — stop silently
}
/// Decode/canonicalization/number-fence failures on PERSISTED data construct
/// VerifyReject::MalformedRow — an integrity finding. canopy_chain::ChainError
/// from the environment stays an error path; there is NO blanket Chain→retry arm.

/// The C6 rejection taxonomy = the pinned chain_incidents.kind vocabulary,
/// now ALSO a CHECK constraint (D2) — Rust enum, SQL CHECK, and the kind→loop
/// mapping are test-pinned against each other:
/// hash_mismatch, linkage_break, noncontiguous_seq, duplicate_seq,
/// formula_version, payload_set_violation, derived_column_mismatch,
/// routing_mismatch, source_mismatch, id_mismatch, genesis_mismatch,
/// terminal_mismatch, head_regression, rows_beyond_head, missing_head,
/// unexpected_head, missing_shard_rows, target_hash_mismatch,
/// manifest_divergence, manifest_metadata_mismatch, boundary_unavailable,
/// malformed_row.
pub enum VerifyReject { /* one variant per kind, positional evidence fields */ }

pub struct VerifyTarget { pub seq: i64, pub hash: EventHash }
pub struct VerifyCursor { pub seq: i64, pub hash: EventHash }

D1a — the batch walk is ONE statement (X6). verify_batch fetches the window cursor.seq+1 ..= target.seq as a SINGLE UNION ALL statement over the archive and live projections with a side tag, ORDER BY chain_seq, LIMIT batch.rows — one MVCC snapshot. A mid-batch archive move can therefore never make rows vanish between two queries, and a row present on BOTH sides arrives as adjacent equal seqs and latches duplicate_seq. The byte budget is applied while CONSUMING the fetched rows: the batch always admits at least ONE row regardless of budget (no livelock); any single row whose payload exceeds the 4 MiB hard ceiling (double the 2 MiB ingress body cap — nothing legitimate can approach it) latches malformed_row; the config floor for CHAIN_VERIFY_BATCH_BYTES is 4 MiB so the ceiling always fits (D11).

/// Verify ONE bounded batch (single-statement archive ∪ live — D1a).
/// Per row, in order:
///   0. the D14 raw number fence over payload_text (and audit metadata text)
///      — violation → MalformedRow;
///   1. contiguity (seq == prev+1; <= prev → DuplicateSeq — including the
///      cross-side duplicate case, which arrives adjacent under D1a);
///   2. linkage (row.previous_hash == running hash);
///   3. formula == 2;
///   4. closed key/type set for the family (payload_set_violation — the same
///      key sets the append fns enforce, mirrored in Rust, test-pinned);
///   5. derived-column consistency: every DerivedColumns field == its payload
///      field (derived_column_mismatch);
///   6. routing placement: shard_for(routing_id, shard_count) == shard
///      (routing_mismatch; routing id = payload event_id for audit, payload id
///      for fti — the Amendment 7 binding);
///   7. source: row.chain_source == topo.source (source_mismatch);
///   8. side sanity: side == Archive requires seq <= archived_through(at
///      fetch); side == Live requires seq > archived_through — a wrong-side
///      row latches (the archived-orphan / wrong-side class, X6);
///   9. (fti) row.id == payload id (id_mismatch);
///  10. envelope rebuild + event_hash recompute == stored (hash_mismatch).
/// If the batch reaches target.seq the running hash MUST equal target.hash
/// (TargetHashMismatch). Returns the advanced cursor + rows/bytes consumed.
pub async fn verify_batch(
    chain: &PgPool, topo: &ActiveTopology, family: ChainFamily, shard: ShardId,
    cursor: VerifyCursor, target: &VerifyTarget, batch: BatchBudget,
) -> Result<BatchOutcome, ChainVerifyError>;
pub struct BatchBudget { pub rows: u32, pub bytes: u64 }
pub struct BatchOutcome { pub cursor: VerifyCursor, pub reached_target: bool,
    pub rows: u32, pub bytes: u64 }

/// Boundary hash for a cursor at `seq` (reads the CHAIN pool): seq == 0 →
/// empty_head_hash(instance, family, epoch, shard, None); else the event_hash
/// of the row at `seq` from whichever side holds it — absent entirely
/// (post-purge, #1208-era) → Reject::BoundaryUnavailable.
pub async fn cursor_hash_at(chain: &PgPool, topo: &ActiveTopology,
    family: ChainFamily, shard: ShardId, seq: i64) -> Result<EventHash, ChainVerifyError>;

/// One-statement heads snapshot (single MVCC snapshot).
pub struct ShardHead { pub shard: ShardId, pub last_seq: i64, pub last_hash: EventHash,
    pub archived_through: i64 }
pub async fn capture_heads(chain: &PgPool, topo: &ActiveTopology, family: ChainFamily)
    -> Result<Vec<ShardHead>, ChainVerifyError>;

/// Structural census (family-lease-serialized, its own cadence — one replica
/// per family per cadence, X2; WHOLE retained history, X6):
/// - head set complete over 0..shard_count, none beyond (MissingHead /
///   UnexpectedHead);
/// - genesis arm: an empty head (last_seq == 0) must carry last_hash ==
///   empty_head_hash (genesis_mismatch);
/// - terminal row exists with event_hash == last_hash, or last_seq <=
///   archived_through with the archive row matching (TerminalMismatch);
/// - rows beyond the CURRENT head — ONE statement per side unioned with the
///   head subselect (single snapshot);
/// - identity-filtered whole-range count == span, computed over archive ∪ live
///   in ONE union statement (MissingShardRows); rows carrying a foreign
///   instance/epoch/shard on EITHER side are detected (the archived-orphan
///   MUST now covers archived rows); on any mismatch the census RE-READS the
///   head/boundary and re-counts ONCE before latching (an archive move between
///   cadences is legal interleaving, never a torn-read breach).
pub async fn structural_census(chain: &PgPool, topo: &ActiveTopology,
    family: ChainFamily, heads: &[ShardHead]) -> Result<(), ChainVerifyError>;

/// Manifest binding — order matters: (1) fetch the latest CONFIRMED anchor row
/// (security pool); (2) decode manifest_bytes (from_canonical_bytes) and
/// compare the DECODED fields against the anchor ROW's caller-supplied
/// metadata — the SEVEN row columns: anchor_seq, kind, epoch,
/// previous_anchor_hash, manifest_hash, instance, family
/// (manifest_metadata_mismatch; chain_anchors has NO source column — the
/// decoded source is hash-bound inside manifest_bytes and checked in step 3);
/// (3) compare identity vs topology (instance/family/source/epoch/shard_count)
/// and vs the COMPILED formula/routing constants; (4) THEN capture heads and
/// check per-tip prefix consistency (tip.last_seq <= head.last_seq AND the
/// chain hash AT tip.last_seq equals tip.last_hash). A tip BEYOND the head
/// triggers ONE re-fetch + re-capture before latching manifest_divergence.
/// On success returns the checked anchor id — stamped on the FAMILY checkpoint
/// row as trusted_manifest_ref via the guarded advance (X5: the family row is
/// the ONE trusted-manifest source; per-shard stamping is REMOVED).
pub async fn manifest_check(security: &PgPool, chain: &PgPool, topo: &ActiveTopology,
    family: ChainFamily) -> Result<Option<CheckedManifest>, ChainVerifyError>;
pub struct CheckedManifest { pub anchor_id: Uuid, pub anchor_seq: i64,
    pub tips: Vec<ShardTipSeq> }

The token-fenced state client — VerifierDb (X1/X10). Every security-pool statement that promises SET LOCAL timeouts or transactional atomicity takes an explicit connection/transaction handle, never a bare &PgPool:

/// Owns the security pool + the verifier's own timeout knobs. Each call opens
/// a transaction, applies SET LOCAL lock_timeout/statement_timeout (the
/// router.rs:27-53 idiom, D11 knobs), runs the statement(s), commits. The
/// job-finalize path exposes the transaction so the run write and the job
/// finalize commit ATOMICALLY (D8).
pub struct VerifierDb { /* security: PgPool, timeouts: VerifyTimeouts */ }

pub struct CheckpointKey { pub instance: ChainInstanceId, pub family: ChainFamily,
    pub epoch: Epoch, pub shard: ShardId, pub loop_kind: LoopKind }
pub enum LoopKind { Tail, Scrub, Family }   // "tail" | "scrub" | "family"
/// The FAMILY key is (instance, family, epoch, shard 0, Family) — PK-distinct
/// from shard 0's tail/scrub rows by loop_kind (D2/D3).

pub struct Lease { pub token: Uuid, pub fence: i64 }

/// Reads go through the token-free view chain_checkpoints_verify_v (X1: the
/// verify role's raw SELECT on the table is REVOKED — the ONLY way to hold a
/// token is to have minted it via acquire).
pub struct CheckpointState { pub fence: i64, pub cursor: Option<VerifyCursor>,
    pub target: Option<VerifyTarget>, pub trusted_manifest_ref: Option<Uuid>,
    pub lease_expires_at: Option<DateTime<Utc>>, pub updated_at: DateTime<Utc>,
    pub cycle_started_at: Option<DateTime<Utc>>,
    pub cycle_completed_at: Option<DateTime<Utc>> }
impl VerifierDb {
  pub async fn read_checkpoint(&self, key: &CheckpointKey)
      -> Result<Option<CheckpointState>, ChainVerifyError>;

  /// chain_checkpoint_acquire: Some(Lease), or None = not acquired (an
  /// unexpired lease — ANY owner's, including our own crashed predecessor
  /// (X1: expiry-only takeover, owner display-only) — or a lost cycle-start
  /// CAS). Duration is a BOUNDED number of seconds; expiry is computed inside
  /// the locked fn (no caller clocks). `init` is REQUIRED for a first-ever
  /// tail/scrub acquire; family acquires carry neither init nor cycle.
  /// Scrub cycle-starts pass `cycle` — a cursor-CAS (X5): the target+reset
  /// write applies ONLY if the row's verified_through_seq still equals
  /// expected_seq, so a delayed worker's stale cycle-start LOSES under the
  /// row lock.
  pub async fn acquire(&self, key: &CheckpointKey, owner: &str, lease_secs: u32,
      init: Option<&VerifyCursor>, cycle: Option<&ScrubCycleStart>)
      -> Result<Option<Lease>, ChainVerifyError>;

  /// chain_checkpoint_advance: exact-token-bound, existing-row-only,
  /// cursor-monotonic. The token stays valid PAST expiry until a takeover
  /// mints a successor (X1: correctness never reads the clock; a finding from
  /// a long batch is never lost to a clock). Health stamps are PROVEN in-fn
  /// (X5): on SCRUB keys cycle_complete only when the presented cursor EQUALS
  /// the stored target (seq AND hash); on the FAMILY key cycle_complete is
  /// the census-cadence stamp — token-gated observability, deliberately NOT
  /// target-proven (census completion is a read-side check with no relational
  /// witness, and no D6 status rule reads the stamp); manifest_ref only on
  /// the FAMILY key and only when it references a CONFIRMED chain_anchors row
  /// of the same (instance, family). cursor is None exactly for family-key
  /// advances (lease refresh / census stamp — the family row has no cursor).
  pub async fn advance(&self, key: &CheckpointKey, lease: &Lease,
      cursor: Option<&VerifyCursor>, lease_secs: u32, cycle_complete: bool,
      manifest_ref: Option<Uuid>) -> Result<(), ChainVerifyError>;   // Err(Fenced) on token mismatch
}
pub struct ScrubCycleStart { pub expected_seq: i64, pub target: VerifyTarget,
    pub boundary: VerifyCursor }

/// Guarded writes (X2): run recording and incident latching validate a token
/// inside the DB. Runs validate the FAMILY token (run rows are family-scoped
/// with an optional shard for error attribution — the any-shard-token hole is
/// gone). Latches validate the token of the lease named by the finding's
/// scope: shard-scoped findings present the shard lease of the DETECTED loop;
/// family-scoped findings (missing/unexpected heads, genesis mismatch,
/// manifest divergence, boundary-unavailable at init) present the FAMILY
/// token with shard NULL.
pub struct VerifyRun { pub loop_kind: LoopKind, pub shard: Option<ShardId>,
    pub started_at: DateTime<Utc>, pub finished_at: DateTime<Utc>,
    pub outcome: RunOutcome, pub error: Option<String>, pub rows_verified: i64,
    pub mode: RunMode, pub job_id: Option<Uuid> }
pub enum RunOutcome { Ok, Rejected, Error }     // 'ok' | 'rejected' | 'error'
pub enum RunMode { Scheduled, Manual }          // manual runs NEVER feed status (X4)
impl VerifierDb {
  pub async fn record_run(&self, family_lease: &Lease, key_family: &CheckpointKey,
      run: &VerifyRun) -> Result<Uuid, ChainVerifyError>;
  /// Same statement inside a caller-owned transaction — the job-finalize
  /// atomicity seam (D8).
  pub async fn record_run_in(&self, tx: &mut PgConnection, family_lease: &Lease,
      key_family: &CheckpointKey, run: &VerifyRun) -> Result<Uuid, ChainVerifyError>;
  pub async fn latch_incident(&self, lease: &Lease, key: &CheckpointKey,
      detected: LoopKind, reject: &VerifyReject) -> Result<Option<Uuid>, ChainVerifyError>;
  /// Halt-gate read — via the evidence-free chain_incidents_verify_v (X3).
  pub async fn family_has_unresolved_incident(&self, instance: ChainInstanceId,
      family: ChainFamily) -> Result<Option<Uuid>, ChainVerifyError>;
}

canopy-chain additives (MR-1): AnchorManifest::from_canonical_bytes(&[u8]) — parse, newtype-decode, validate(), re-encode == input; PartialEq derives on AnchorManifest/ShardTip; the D14 canon::raw_number_fence + its KAT vectors. New shared types: ChainPosition and ChainBacklog live in canopy-contracts-security (wire types — D8); CheckedManifest/ShardTipSeq/ Lease/BatchBudget live in chain_verify (engine types).

test-lib additive (MR-1): append_chained_rows(pool, &SeededChain, shard, n, salt, dist: PayloadDist) — drives the REAL append_rows_on_shard in ≤500 batches; PayloadDist parameterizes the size distribution (X10 — the D13 mixes are arguments, not a hardcode).

D2 — the hardening migration: token confidentiality, the family loop, proofs, guarded writers

The MR-1 migration (20261010000000_chain_verification_hardening.sql, security DB) reshapes the dormant substrate — pre-1.0, zero compat, empty tables (no backfills). Everything follows the substrate’s own idioms: DO/EXECUTE format() with SET search_path, owner-transfer to canopy_chain_owner_security, PUBLIC EXECUTE revoked, per-role grants re-established, touch crates/canopy-test-lib/src/db.rs.

Tables:

-- Checkpoints: the family loop kind, the token, the cycle stamps, the CHECK matrix.
ALTER TABLE chain_verification_checkpoints
    DROP CONSTRAINT chain_verification_checkpoints_loop_kind_check;  -- the :296 inline CHECK
ALTER TABLE chain_verification_checkpoints
    ADD COLUMN lease_token UUID,
    ADD COLUMN cycle_started_at TIMESTAMPTZ,
    ADD COLUMN cycle_completed_at TIMESTAMPTZ,
    ADD CONSTRAINT ..._loop_kind CHECK (loop_kind IN ('tail','scrub','family')),
    -- The FAMILY row is (instance, family, epoch, shard 0, 'family') — PK-distinct
    -- from shard 0's tail/scrub rows. It carries NO cursor and NO target; it
    -- carries the family lease, trusted_manifest_ref (X5: the ONE source), and
    -- cycle_completed_at as the census-cadence stamp.
    ADD CONSTRAINT ..._family_shape CHECK (loop_kind <> 'family'
        OR (shard_id = 0 AND target_seq IS NULL AND verified_through_seq = 0
            AND verified_through_hash IS NULL)),
    ADD CONSTRAINT ..._cursor_pair CHECK (loop_kind = 'family'
        OR verified_through_hash IS NOT NULL),
    ADD CONSTRAINT ..._hash_len CHECK ((verified_through_hash IS NULL
            OR octet_length(verified_through_hash) = 32)
        AND (target_hash IS NULL OR octet_length(target_hash) = 32)),
    ADD CONSTRAINT ..._nonneg CHECK (verified_through_seq >= 0 AND fence >= 0
        AND (target_seq IS NULL OR target_seq >= 0)),
    ADD CONSTRAINT ..._target_pair CHECK ((target_seq IS NULL) = (target_hash IS NULL)),
    ADD CONSTRAINT ..._target_ge_cursor CHECK (target_seq IS NULL
        OR target_seq >= verified_through_seq),
    ADD CONSTRAINT ..._lease_trio CHECK ((lease_owner IS NULL) = (lease_expires_at IS NULL)
        AND (lease_owner IS NULL) = (lease_token IS NULL)),
    ADD CONSTRAINT ..._token_v7 CHECK (lease_token IS NULL
        OR uuid_extract_version(lease_token) = 7);

-- Runs: family loop kind, per-shard attribution, the manual/scheduled split.
ALTER TABLE chain_verification_runs
    DROP CONSTRAINT chain_verification_runs_loop_kind_check;         -- the :314 inline CHECK
ALTER TABLE chain_verification_runs
    ADD COLUMN shard_id SMALLINT,          -- NULL = family-scoped / whole-family coverage
    ADD COLUMN mode TEXT NOT NULL CHECK (mode IN ('scheduled','manual')),
    ADD COLUMN job_id UUID,                -- FK added in MR-2 (the jobs table lands there)
    ADD CONSTRAINT ..._loop_kind CHECK (loop_kind IN ('tail','scrub','family')),
    ADD CONSTRAINT ..._outcome CHECK (outcome IN ('ok','rejected','error')),
    ADD CONSTRAINT ..._finish_pair CHECK ((finished_at IS NULL) = (outcome IS NULL)),
    ADD CONSTRAINT ..._finish_order CHECK (finished_at IS NULL OR finished_at >= started_at),
    ADD CONSTRAINT ..._rows_nonneg CHECK (rows_verified >= 0);
CREATE INDEX chain_verification_runs_latest_idx ON chain_verification_runs
    (chain_instance_id, chain_family, chain_epoch, loop_kind, shard_id,
     finished_at DESC NULLS LAST)
    WHERE mode = 'scheduled';   -- per-SCOPE latest-run reads (X5); status reads
                                -- scheduled runs ONLY, and manual runs are
                                -- reached by id via their job — the partial
                                -- predicate matches the D6 view exactly

-- Incidents: the detecting loop is STORED, dedup is NULL-safe, the vocabulary
-- is closed in-schema, evidence is bounded.
ALTER TABLE chain_incidents
    ADD COLUMN detected_loop_kind TEXT NOT NULL
        CHECK (detected_loop_kind IN ('tail','scrub','family')),
    ADD CONSTRAINT ..._family_scope CHECK ((detected_loop_kind = 'family')
        = (shard_id IS NULL)),
    ADD CONSTRAINT ..._kind_vocab CHECK (kind IN ('hash_mismatch','linkage_break',
        'noncontiguous_seq','duplicate_seq','formula_version','payload_set_violation',
        'derived_column_mismatch','routing_mismatch','source_mismatch','id_mismatch',
        'genesis_mismatch','terminal_mismatch','head_regression','rows_beyond_head',
        'missing_head','unexpected_head','missing_shard_rows','target_hash_mismatch',
        'manifest_divergence','manifest_metadata_mismatch','boundary_unavailable',
        'malformed_row')),
    ADD CONSTRAINT ..._evidence_shape CHECK (jsonb_typeof(evidence) = 'object'
        AND pg_column_size(evidence) <= 16384);
CREATE UNIQUE INDEX chain_incidents_latched_uq ON chain_incidents
    (chain_instance_id, chain_family, chain_epoch, shard_id, kind, detected_loop_kind)
    NULLS NOT DISTINCT WHERE state = 'latched';   -- family-scoped (NULL epoch/shard) dedups (X3)

The confidentiality split (X1/X3) — the verify role loses every raw C6 table read; what it needs comes back through owner-transferred views:

CREATE VIEW chain_checkpoints_verify_v AS SELECT
    chain_instance_id, chain_family, chain_epoch, shard_id, loop_kind,
    target_seq, target_hash, verified_through_seq, verified_through_hash,
    trusted_manifest_ref, lease_owner, lease_expires_at, fence, updated_at,
    cycle_started_at, cycle_completed_at
  FROM chain_verification_checkpoints;            -- EVERYTHING except lease_token
CREATE VIEW chain_incidents_verify_v AS SELECT
    id, chain_instance_id, chain_family, chain_epoch, shard_id,
    detected_at, kind, detected_loop_kind, state
  FROM chain_incidents;                           -- no evidence, no resolution text
-- (owner-transfer both to canopy_chain_owner_security, the substrate $own$ pattern)
REVOKE SELECT ON chain_verification_checkpoints, chain_verification_runs,
    chain_incidents FROM canopy_chain_verify;     -- closes the :863 token/evidence leak
REVOKE INSERT ON chain_verification_runs FROM canopy_chain_verify;  -- the :864 open write
GRANT SELECT ON chain_checkpoints_verify_v, chain_incidents_verify_v
    TO canopy_chain_verify;
GRANT SELECT ON chain_incidents TO canopy_chain_incident_admin;  -- the D7 evidence reader

The reshaped checkpoint functions — durations, proofs, the family arm:

CREATE FUNCTION chain_checkpoint_acquire(
    p_instance uuid, p_family text, p_epoch int, p_shard smallint, p_loop text,
    p_owner text, p_lease_secs int,
    p_init_seq bigint DEFAULT NULL, p_init_hash bytea DEFAULT NULL,
    p_cycle_expected_seq bigint DEFAULT NULL,
    p_cycle_target_seq bigint DEFAULT NULL, p_cycle_target_hash bytea DEFAULT NULL,
    p_cycle_reset_seq bigint DEFAULT NULL, p_cycle_reset_hash bytea DEFAULT NULL
) RETURNS TABLE (lease_token uuid, fence bigint)
-- Domain: p_lease_secs BETWEEN 1 AND 600 (RAISE — X1: bounded DURATION; expiry
--   := now() + make_interval(secs => p_lease_secs) computed HERE, no caller
--   clocks, no lock-wait erosion).
-- Family arm: p_loop = 'family' requires p_shard = 0 and every init/cycle
--   param NULL (RAISE otherwise); the INSERT arm creates the family row with a
--   NULL cursor hash (the _family_shape/_cursor_pair CHECKs).
-- Tail/scrub arm: INSERT ... ON CONFLICT DO NOTHING with the REQUIRED init
--   cursor, then re-SELECT FOR UPDATE (concurrent first-acquires serialize;
--   absent row + NULL init → RAISE).
-- Takeover is EXPIRY-ONLY for everyone (X1): an unexpired lease — ANY owner,
--   including the caller's own text — returns EMPTY. Owner is display-only; a
--   restarted process waits out its own lease (bounded by p_lease_secs).
-- Cycle-start (scrub keys; all five p_cycle_* or none — RAISE on a partial
--   set) is a cursor-CAS (X5): it applies ONLY if verified_through_seq =
--   p_cycle_expected_seq; a stale worker's delayed cycle-start returns EMPTY
--   under the row lock and resets NOTHING. On success it writes target +
--   cursor reset + cycle_started_at := now() in the same UPDATE
--   (cycle_completed_at is NOT cleared — it stamps the LAST completed cycle).
-- Grant: fence := fence + 1, lease_token := uuidv7(); returns the minted
--   token + new fence — the ONLY fence-raising, row-creating path.

CREATE FUNCTION chain_checkpoint_advance(
    p_instance uuid, p_family text, p_epoch int, p_shard smallint, p_loop text,
    p_lease_token uuid, p_lease_secs int,
    p_verified_seq bigint DEFAULT NULL, p_verified_hash bytea DEFAULT NULL,
    p_cycle_complete boolean DEFAULT false, p_manifest_ref uuid DEFAULT NULL
) RETURNS boolean
-- Existing-row-only (absent → false). Token must EQUAL the stored lease_token
--   (else false — the fence); expiry is NEVER read (X1: a token stays valid
--   past expiry until a takeover mints a successor — a long batch's finding
--   is never lost to a clock; p_lease_secs, domain-checked 1..=600, renews
--   expiry := now() + interval on success).
-- Family arm: p_loop = 'family' requires p_verified_seq/hash NULL and permits
--   p_manifest_ref / p_cycle_complete (the census-cadence stamp). Tail/scrub
--   arms REQUIRE the cursor pair and REJECT p_manifest_ref (RAISE — X5: the
--   family row is the ONE trusted-manifest home).
-- Cursor monotonic: p_verified_seq < verified_through_seq → false; equal seq
--   with a DIFFERENT hash → RAISE 'chain-v2: checkpoint cursor hash
--   divergence' (corruption, never silent).
-- PROOFS (X5 — the scrub-coverage and manifest stamps are never caller
--   assertions; the family census stamp is the stated exception: token-gated
--   observability with no relational witness, and no D6 status rule reads it):
--   p_cycle_complete on a scrub key RAISEs unless p_verified_seq = target_seq
--   AND p_verified_hash = target_hash (checked against the STORED target
--   in-fn); on success stamps cycle_completed_at := now().
--   p_manifest_ref RAISEs unless it references a chain_anchors row with
--   notarization_state = 'confirmed' AND matching (chain_instance_id,
--   chain_family) — a relational check, not trust.
-- Never touches fence/target otherwise; stamps updated_at.

-- Guarded writers:
CREATE FUNCTION chain_run_record(
    p_family_token uuid,
    p_instance uuid, p_family text, p_epoch int, p_loop text, p_shard smallint,
    p_started timestamptz, p_finished timestamptz, p_outcome text, p_error text,
    p_rows bigint, p_mode text, p_job_id uuid
) RETURNS uuid
-- Validates p_family_token = the FAMILY row's lease_token for (p_instance,
-- p_family, p_epoch) — X2: runs are recorded ONLY by the family-lease holder
-- (returns NULL when fenced; the engine maps NULL → Fenced). INSERTs the
-- immutable run row (id := uuidv7()). p_job_id is stored from MR-1 (the FK
-- and the mode⇔job CHECK arrive with the MR-2 jobs table — same-signature
-- CREATE OR REPLACE there, explicitly NOT an overload).

CREATE FUNCTION chain_incident_latch(
    p_token uuid,
    p_instance uuid, p_family text, p_epoch int, p_shard smallint,
    p_detected_loop text, p_kind text, p_evidence jsonb
) RETURNS uuid
-- Scope rule (X3): p_detected_loop = 'family' ⟺ p_shard IS NULL (RAISE
-- otherwise). Token validation matches the finding's scope: family-scoped →
-- the FAMILY row's token; shard-scoped → the (p_shard, p_detected_loop) row's
-- token. p_kind is CHECK-constrained by the table; p_evidence bounded by the
-- _evidence_shape CHECK. INSERT ... ON CONFLICT (chain_instance_id,
-- chain_family, chain_epoch, shard_id, kind, detected_loop_kind)
-- WHERE state = 'latched' DO NOTHING; returns the existing latched id on
-- conflict (idempotent, race-free). Fenced token → NULL.

CREATE FUNCTION chain_incident_resolve(p_id uuid, p_reason text,
    p_evidence_ref text, p_revalidation_run uuid) RETURNS void
-- actor := session_user (recorded by the fn, never caller-supplied);
-- p_evidence_ref REQUIRED. The revalidation run must be: outcome = 'ok',
-- mode = 'manual' (X4: scheduled runs NEVER resolve), matching
-- instance + family, finished_at > incident.detected_at, loop_kind =
-- incident.detected_loop_kind (STORED, never inferred — X3), and
-- scope-covering: run.shard_id IS NULL (whole-family coverage) OR
-- run.shard_id = incident.shard_id.

The anchor role/arm split — the transition fn splits into NAMED arms (the substrate’s single fn :688-718 lets one credential fabricate and confirm):

CREATE ROLE canopy_chain_anchor_emitter NOLOGIN;   -- reconciled fail-closed like the rest
CREATE FUNCTION chain_anchor_transition_emit(
    p_id uuid, p_to text, p_jws text, p_kid text,
    p_external_ref text, p_external_version text) RETURNS void
-- The :701-707 edge matrix MINUS confirmation: pending→{submitted,failed},
-- submitted→failed, failed→submitted; p_to = 'confirmed' → RAISE. COALESCE
-- updates of jws/kid/external_ref/external_version as today.
CREATE FUNCTION chain_anchor_transition_confirm(p_id uuid) RETURNS void
-- submitted→confirmed ONLY; stamps verifier_confirmed_at := now(); touches
-- NO caller-supplied columns (the emitter arms own those).
REVOKE EXECUTE ON FUNCTION chain_anchor_append(uuid,text,int,bigint,text,bytea,bytea,bytea)
    FROM canopy_chain_verify;
GRANT EXECUTE ON FUNCTION chain_anchor_append(uuid,text,int,bigint,text,bytea,bytea,bytea)
    TO canopy_chain_anchor_emitter;
GRANT EXECUTE ON FUNCTION chain_anchor_transition_emit(uuid,text,text,text,text,text)
    TO canopy_chain_anchor_emitter;
GRANT EXECUTE ON FUNCTION chain_anchor_transition_confirm(uuid)
    TO canopy_chain_verify;

Archive attestation indexes (LIKE copies no indexes):

CREATE UNIQUE INDEX audit_events_archive_v2_event_id_uq
    ON audit_events_archive_v2 (((canonical_event_payload->>'event_id')));
-- and in the tanf/medicaid MR-1 migrations (alongside the view change):
--   CREATE UNIQUE INDEX fti_audit_log_archive_v2_id_uq
--       ON fti_audit_log_archive_v2 (id);

Old signatures DROPPED, never overloaded (CREATE OR REPLACE with a changed signature creates a NEW OVERLOAD in PostgreSQL — the substrate originals and their grants would survive as the exact open write paths this migration closes):

DROP FUNCTION chain_checkpoint_advance(uuid, text, int, smallint, text,
    bigint, text, timestamptz, bigint, bytea);
DROP FUNCTION chain_incident_latch(uuid, text, int, smallint, text, jsonb);
DROP FUNCTION chain_incident_resolve(uuid, text, text, text, uuid);
DROP FUNCTION chain_anchor_transition(uuid, text, text, text, text, text);
-- (chain_checkpoint_acquire and chain_run_record are new — nothing to drop.
-- The grant-matrix test asserts to_regprocedure(<each old signature>) IS NULL
-- post-migration.)

The protocol (Amendments 8–9): acquire → Lease{token, fence}; every write presents the token; ANY newer acquire mints a new token and the old holder’s next write returns Fenced (stop, re-enter next pass). Expiry-only takeover means the worst-case pause after a crash is one lease duration (default 30s). The lease ≥ 3× statement-timeout rule is a LIVENESS heuristic, not a correctness proof (X1): correctness rides the token; a batch outliving its lease merely gets fenced on its next write and the work is discarded. Every acquire/advance statement runs under the VerifierDb SET LOCAL timeouts (D11 knobs — never the drainer’s 30s default).

D3 — the family pass (X2): one lease orders everything

THREE independent tasks, one per family target (audit/security, fti/tanf, fti/medicaid) — a broken program DB back-pressures only its own family. Each task, per pass (verifier_pass(pools, target, cfg, pass_counter) — testable), in THIS order:

  1. fetch_topology(chain_pool, family) — read-only, supplies the INSTANCE (nothing below has a key without it); Missing/NotActive → idle-skip debounced (dormant pre-cutover). Topology-fetch is the only step preceding the family lease; its failure is logged and surfaces as read-time staleness (documented — no run row exists to record, and none is needed).

  2. Acquire the FAMILY lease — key (instance, family, epoch, shard 0, 'family'). None → another replica owns this family’s pass; skip the pass entirely. The family lease structurally serializes the census AND the manifest check (one replica per family per cadence — the O(range) census multiplier is gone) and makes the family holder the ONLY servicer of that family’s manual jobs (D8 — no cross-replica job lottery). Scale-out is BY FAMILY (three families today); shard tokens remain the write-fencing belt underneath (Amendment 9 records this honestly).

  3. Halt gate: family_has_unresolved_incidentSome(id) and no manual job in hand → skip with a debounced warn. DB-derived, restart-safe, replica-global. RE-CHECKED between batches — a latch propagates within one batch bound; the residual in-flight window (one batch) is documented.

  4. Manual jobs FIRST (X2): claim via chain_job_claim (target-scoped); a claimed job’s segments run under the same global budget before scheduled work (D8).

  5. Manifest check, then census (both under the family token): manifest_check per D1 — a passing check’s anchor id is stamped on the FAMILY row (advance(family_key, …, manifest_ref)); the census runs on its own cadence (CHAIN_CENSUS_INTERVAL_SECS), stamping the family row’s cycle_completed_at as the cadence marker. Family-scoped findings latch under the family token with shard NULL.

  6. Per shard, rotating start: acquire the tail key (first-ever acquire passes init = boundary cursor — genesis or archive-boundary hash via cursor_hash_at; a boundary_unavailable here latches under the FAMILY token — the shard lease does not exist yet); compare (cursor.seq, cursor.hash) against the captured head: seq beyond head, or EQUAL seq with a different hash → latch head_regression. Verify bounded batches within the pass’s GLOBAL budget, advancing after every batch. Then the scrub key likewise (D4).

  7. Every shard visit counts against the budget (X10) — including zero-work lease-refresh advances (the success stamp that clears error at read time). The family lease is refreshed by a family-row advance between shard segments — bounded chatter, and the refresh IS the family activity stamp.

  8. On Reject: token-guarded latch + run record (outcome 'rejected', the shard attributed); the halt gate stops the family. On Database/Topology errors: run record (family token, shard attributed where known), capped backoff (250ms→5s), never park, never green. Runs are recorded for passes that did work, erred, or rejected — never for idle ticks; scheduled runs carry mode = 'scheduled'.

Tick CHAIN_VERIFY_TICK_MS (default 1000); first tick delayed CHAIN_VERIFY_FIRST_TICK_DELAY_SECS (default 60) — sleep, then interval (the detection.rs:28-41 delayed-start precedent, generalized to a configurable delay); tests use paused tokio time (tokio/test-util dev-dep).

D4 — the historical scrub loop (full retained history)

Same task, loop_kind = 'scrub', CHAIN_SCRUB_BATCHES_PER_PASS (default 1) within the same global pass budget:

  • Coverage: the cycle walks 0 → target across ARCHIVE ∪ LIVE in single-statement batches (D1a) — archived rows re-verify under the full check set (a mutated archived payload with an intact stored hash IS detected). A boundary move mid-cycle is benign: the union is by seq range, side-agnostic, and each batch is one snapshot.

  • Cycle start (X5 — a cursor-CAS, race-free under the row lock): when the row shows target IS NULLcursor.seq == target.seq, the engine acquires with cycle = ScrubCycleStart{ expected_seq: <the cursor seq it just read>, target: tail cursor snapshot, boundary: (0, genesis hash) }. The fn applies the reset ONLY if verified_through_seq still equals expected_seq — a delayed worker whose read predates another’s cycle activity LOSES (empty result, no reset). Restart mid-cycle acquires plain and resumes from the stored cursor. cycle_started_at is stamped by the CAS — the first-cycle staleness input (D6).

  • Cycle end: reaching the target advances with cycle_complete = true — and the fn PROVES it (cursor == stored target, seq AND hash) before stamping cycle_completed_at (X5). A >0-rows cycle records a run.

  • A scrub Reject latches + halts identically. Empty-range cycles complete immediately.

D5 — census + the manifest trust chain

  • Census: family-lease-serialized, cadenced, whole-history (archive ∪ live — X6), retry-once-before-latch on count/boundary mismatches, explicit genesis arm.

  • Manifest: manifest_check per D1 — anchor-first ordering, decoded-bytes vs the SEVEN row-metadata columns vs topology vs chain, one re-fetch on a tip-beyond-head, then latch. The checked anchor id lands on the FAMILY row only (X5) — one unambiguous trusted_manifest_ref; status and attestation read ONLY the family ref — a newly confirmed but never-verifier-checked anchor authorizes nothing.

  • The #1278-facing confirmer contract (pinned for the anchor-authority child): the verifier confirms an anchor (chain_anchor_transition_confirm) ONLY after manifest_check passes on THAT anchor — bytes decoded, row metadata matched, topology matched, per-tip prefix consistency proven. #1278’s emitter submits (chain_anchor_transition_emit) and never confirms; the verifier confirms and never emits. The C5 divergence consumer arm rides the same check.

  • No confirmed anchor → manifest: absent → status caps at verifying. Manifest age (now - checked anchor’s created_at) over CHAIN_MANIFEST_MAX_AGE_SECSstale (default generous, 7 days; #1278 tightens). Post-cutover the confirmed genesis anchor exists (#1279 gate) — healthy is reachable at reopen; real-event ATTESTATION additionally needs the first periodic tip.

  • Residual (Amendment 8): a rewrite strictly newer than the trusted manifest is invisible until the next anchor — #1278’s cadence bounds that window.

D6 — the C6 status machine + backlog inputs + the _app projection

Inputs (assembled per read):

Input Source Availability

Tail/scrub coverage + freshness per shard (incl. cycle_started_at, cycle_completed_at) + the FAMILY row (trusted_manifest_ref, family activity)

chain_verification_status_v via the app pool (NO lease_token, NO lease_owner — tokens never reach _app)

always (0 rows pre-first-acquire)

Latest SCHEDULED run per (instance, family, epoch, loop, shard-scope)

chain_verification_runs_v (keyed + indexed by exactly that; WHERE mode = 'scheduled' — manual outcomes NEVER feed status, X4; raw error text EXCLUDED)

always

Unresolved incidents (position + kind + detected loop — never evidence)

chain_incidents_app_v

always

Trusted manifest (id, seq, age)

chain_anchor_trusted_v (joins the FAMILY rows' trusted_manifest_ref to chain_anchors; manifest_bytes exposed for attestation decode)

always

Head seqs for lag

audit: chain_status_v (_app, substrate :859); FTI: chain_heads on the verify pools

always

Staging backlog (audit only)

in-process ChainStagingSnapshot (#1207, always-on) + typed sampler_untrustworthy: bool (additive; the two sampler arms, health.rs:167-175)

sampled == false BLOCKS healthy

Inbox parks + DLQ depth (audit only)

ChainStatusInputs sampler (30s, always-on; queue literal extracted to a constant)

unsampled BLOCKS healthy

Derivation (derive_status — the domain enum and the pure derivation live in canopy_common::chain_verify::status; the wire enum maps from it explicitly in canopy-security, bijection test-pinned — X9). Precedence:

  1. breached — any unresolved incident for (instance, family). Latched; only D7 clears. The FIRST unresolved incident’s position populates breached_position (D8 — the typed row-banner source).

  2. error — PER SCOPE (X5): the latest scheduled run for a (loop, shard) has outcome = 'error' AND is newer than THAT shard’s checkpoint updated_at success stamp; family-scoped errors (shard NULL) compare against the FAMILY row’s updated_at. A shard’s error clears only against its own zero-work advance — cross-shard masking is structurally gone. Also: any backlog input reporting sampler_untrustworthy.

  3. stale — read-time: tail updated_at age > tail_max_age on any shard; tail lag > tail_max_lag on any shard; scrub cycle_completed_at age > scrub_max_age on any shard — falling back to cycle_started_at when no cycle has EVER completed (X5: a first cycle that never completes goes stale, exactly as the design claims); trusted-manifest age > manifest_max_age (from the FAMILY row).

  4. unknown — no checkpoint coverage (fresh/dormant) or topology absent/not-active.

  5. verifying — coverage advancing but incomplete/unanchored: initial catch-up, any backlog count nonzero, any backlog input NOT YET SAMPLED (with the nonzero clause, the exact complement of rule 6’s "SAMPLED and zero"), or manifest: absent.

  6. healthy — every shard’s tail at head within freshness + lag; every shard’s scrub cycle fresh; every backlog input SAMPLED and zero; trusted manifest present, checked, fresh; no unresolved incident; no live per-scope error.

HTTP mapping (Amendment 8, unchanged): healthy/verifying → 200; unknown/stale/error/breached → 503.

There is no separate census-staleness rule by construction: the census rides the family pass, so a family whose census stops has either a stopped pass (tail rows go stale) or a latched census finding (breached).

The _app projection migration (MR-2, 20261015000000_chain_verification_projections.sql + test-lib touch): the four views above (owner-transferred to canopy_chain_owner_security — a view runs with its owner’s rights, and only that role reads the C6/anchor bases) + the D8 jobs table + its guarded fns + the runs job_id FK + the mode ⇔ job CHECK (ALTER TABLE chain_verification_runs ADD CONSTRAINT …​_manual_job CHECK mode = 'manual') = (job_id IS NOT NULL, ADD CONSTRAINT …​_job_fk FOREIGN KEY (job_id) REFERENCES chain_verify_jobs(id)) + the same-signature CREATE OR REPLACE FUNCTION chain_run_record adding the job validation (identical signature — explicitly NOT an overload). chain_incident_resolve is untouched here: its mode/loop/scope checks are MR-1’s, and MR-2’s mode ⇔ job CHECK makes manual ⇒ job-linked structurally.

D7 — incidents: latch, evidence, resolution

  • Latch kinds = VerifyReject::kind_str() = the D2 CHECK vocabulary = the kind→loop mapping — all three test-pinned against each other. Evidence = positions
    expected/got hex only, bounded by the _evidence_shape CHECK.

  • The reader split is real (X3): the background verifier reads incidents ONLY through the evidence-free chain_incidents_verify_v (enough for the halt gate and status); _app reads chain_incidents_app_v (position/kind/detected-loop/state); evidence and resolution text are readable ONLY by canopy_chain_incident_admin.

  • Resolution (runbook in security-operations.adoc): inspect evidence under the incident-admin credential → ticket → trigger the manual revalidation job (bypasses the halt gate; runs the incident’s DETECTED loop — stored at latch, never inferred) → confirm outcome 'ok'chain_incident_resolve(id, reason, ticket, run_id) — the fn records actor = session_user and enforces manual mode + detected-loop match
    scope coverage + timing in SQL (D2). A clean scheduled pass never clears.

D8 — the unified /v1/security/chain/* surface + durable jobs

Namespace (X9, Amendment 9 — pre-1.0 CHANGELOG Changed, zero compat): ONE chain namespace replaces the historical scatter. Deleted: GET /v1/security/verify-chain
POST /v1/security/fti/chain-verify (MR-2), GET /v1/security/fti/chain-status (MR-3 — the interim handler survives untouched until then, serving the #1245 posture). Path-count assertion (api/mod.rs:781, currently 15): MR-2 → 17 (−2 +4); MR-3 → 16 (−1). Two staged OpenAPI snapshots, one per MR; devstack refresh BEFORE push (#1267).

  • GET /v1/security/chain/status?family={audit|fti}[&service=]ChainStatusResponse (200/503 per D6). MR-2: family=fti → 503 verifier_unavailable (routing live, target dormant until MR-3).

  • POST /v1/security/chain/verify {family, service?, loop?, incident_id?} → 202 ChainVerifyJobAccepted — or 409 verification_in_progress (an active job already exists for the target; body carries its job_id), 503 at the queue cap / verifier disabled / family unconfigured (no phantom queue — X4).

  • GET /v1/security/chain/verify-jobs/{id}ChainVerifyJobStatus. Requester-scoped: service callers see only their own jobs; admin sees all; unknown/foreign id → 404.

  • GET /v1/security/chain/attest?event_id={uuid}&family={audit|fti}[&service=]ChainAttestation. Position resolution is VIEW-mediated and INDEXED, across archive ∪ live: audit by the payload-expression predicate, FTI by the id column (both sides explicitly indexed — D2). Attested iff position found AND the position’s (instance, epoch) MATCH the active topology (the ChainPosition carries both — an old-epoch/old-instance row can never attest, X9) AND seq ⇐ tail.verified_through(shard) AND seq ⇐ TRUSTED manifest tip.last_seq(shard) AND state ∉ {unknown, stale, error, breached}. Pool discipline: position resolution runs on the VERIFY pools for BOTH families (the preimage views are verify-role-only — _app never gains them, C8); the status/coverage/trusted-tip inputs ride the _app projections. Dormant → 503.

Auth unchanged (the existing service-or-admin arms).

The request-error matrix (closed vocabulary; each cell is a named test):

Condition Status error code

family missing or not in {audit, fti}

400

invalid_family

family=fti without service

400

missing_service

family=audit WITH service

400

unexpected_service

service not in {canopy-tanf, canopy-medicaid}

400

invalid_service

event_id missing/malformed (attest)

400

invalid_event_id

loop not in {tail, scrub, family-full} (verify)

400

invalid_loop

incident_id unknown (verify)

404

unknown_incident

active job exists for the target (verify)

409

verification_in_progress

queue at CHAIN_JOB_MAX_QUEUED / verifier disabled / family unconfigured

503

verifier_unavailable

Replaced DTOs (crates/canopy-contracts-security/src/chain.rs — full serde-attributed shapes; every closed vocabulary is a real enum, wire strings test-pinned):

#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum ChainFamilyWire { Audit, Fti }
#[derive(Serialize, Deserialize)]
pub enum ChainServiceWire {
    #[serde(rename = "canopy-tanf")] CanopyTanf,
    #[serde(rename = "canopy-medicaid")] CanopyMedicaid }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum ChainVerifyState { Unknown, Verifying, Healthy, Stale, Error, Breached }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum ChainLoopWire { Tail, Scrub, Family }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "kebab-case")]
pub enum RequestedLoopWire { Tail, Scrub, FamilyFull }      // matches the DB vocabulary
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum JobStateWire { Queued, Running, Done, Error }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum JobErrorCode { CoverageIncomplete, VerifierError, IntegrityRejected, Crashed }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum AttestRefusal { UnknownEvent, NewerThanCheckpoint, BeyondTrustedManifest,
    StateNotAttestable, VerifierUnavailable, ForeignTopology }
// StatusReason: the closed reasons vocabulary (one variant per firing D6 input),
// #[serde(rename_all = "snake_case")], enumerated + test-pinned in the source.

pub struct ChainPosition { pub instance: Uuid, pub family: ChainFamilyWire,
    pub epoch: i32, pub shard: u16, pub seq: i64 }   // instance+family: X9 —
    // attestation and the row banner filter to the EXACT topology
pub struct ChainBacklog { pub staged: i64, pub parked: i64,
    pub inbox_parked: i64, pub dlq_depth: i64 }

pub struct ShardCoverage {
    pub shard_id: u16,
    pub tail_verified_through: i64,
    pub head_seq: i64,
    pub tail_lag: i64,
    pub tail_updated_at: Option<DateTime<Utc>>,
    pub scrub_verified_through: i64,
    pub scrub_target: Option<i64>,
    pub scrub_updated_at: Option<DateTime<Utc>>,
    pub scrub_cycle_started_at: Option<DateTime<Utc>>,    // first-cycle staleness input
    pub scrub_cycle_completed_at: Option<DateTime<Utc>>,  // coverage stamp
}

pub struct ChainStatusResponse {
    pub family: ChainFamilyWire,
    pub service: Option<ChainServiceWire>,
    pub state: ChainVerifyState,
    pub reasons: Vec<StatusReason>,
    pub epoch: Option<i32>,
    pub shards: Vec<ShardCoverage>,
    pub backlog: Option<ChainBacklog>,   // None for fti — typed applicability,
                                         // never zeroed i64s (X5)
    pub trusted_manifest: Option<TrustedManifestSummary>,
    pub incident_id: Option<Uuid>,
    pub breached_position: Option<ChainPosition>,  // the typed row-banner source (X9)
}
pub struct TrustedManifestSummary { pub anchor_id: Uuid, pub anchor_seq: i64,
    pub age_secs: i64 }

pub struct ChainVerifyJobAccepted { pub job_id: Uuid, pub poll_url: String }
pub struct ChainVerifyJobStatus { pub job_id: Uuid, pub state: JobStateWire,
    pub requested_loop: RequestedLoopWire, pub attempts: i32,
    pub run: Option<ChainRunSummary>, pub error_code: Option<JobErrorCode> }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum RunOutcomeWire { Ok, Rejected, Error }     // = the D2 CHECK vocabulary
pub struct ChainRunSummary { pub run_id: Uuid, pub loop_kind: ChainLoopWire,
    pub outcome: Option<RunOutcomeWire>, pub rows_verified: i64,
    pub started_at: DateTime<Utc>, pub finished_at: Option<DateTime<Utc>> }
pub struct ChainAttestation { pub attested: bool, pub event_id: Uuid,
    pub position: Option<ChainPosition>, pub verified_through: Option<i64>,
    pub trusted_anchor_seq: Option<i64>, pub state: ChainVerifyState,
    pub reason: Option<AttestRefusal> }

Retirement scope (pre-1.0): ChainVerificationResponse dies in MR-2 — producer + all four consumer sites (three canopy-web files + the test-lib client) migrate in the same MR. ChainStatusInterim/InterimChainState remain ONLY for run_archive (1208’s surface); ChainStatusInterim.last_verification + FtiChainVerification(+Row) die in MR-3 with the reader narrowed per D10 (FtiChainStatusParams survives). No [deprecated] anywhere; doc-comments carry the retirement notes; the stale references at api/mod.rs:578 and events.rs:191 are updated (J5).

The durable job store (X4 — MR-2, in the projections migration):

CREATE TABLE chain_verify_jobs (
  id UUID PRIMARY KEY CHECK (uuid_extract_version(id) = 7),
  chain_family TEXT NOT NULL CHECK (chain_family IN ('audit','fti')),
  fti_source TEXT CHECK (fti_source IN ('canopy-tanf','canopy-medicaid')),
  CHECK ((chain_family = 'fti') = (fti_source IS NOT NULL)),
  requested_loop TEXT NOT NULL CHECK (requested_loop IN ('tail','scrub','family-full')),
  incident_id UUID REFERENCES chain_incidents(id),   -- revalidation jobs
  requested_by TEXT NOT NULL CHECK (requested_by <> ''),
  requested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  state TEXT NOT NULL DEFAULT 'queued'
      CHECK (state IN ('queued','running','done','error')),
  -- The durable work definition, written ONCE at first claim (X4): reclaim
  -- resumes the SAME instance/epoch/target vector — a crash never re-captures
  -- weaker targets. Shape: {"shards":[{"shard":N,"seq":S,"hash":"<hex>"},…]}.
  chain_instance_id UUID, chain_epoch INT, captured_targets JSONB,
  CHECK ((captured_targets IS NULL) = (chain_instance_id IS NULL)
     AND (captured_targets IS NULL) = (chain_epoch IS NULL)),
  -- The claim is DB-minted-token-based (the checkpoint ABA fix, symmetric):
  claim_owner TEXT, claim_token UUID, claim_expires_at TIMESTAMPTZ,
  heartbeat_at TIMESTAMPTZ,
  attempts INT NOT NULL DEFAULT 0 CHECK (attempts >= 0),
  run_id UUID REFERENCES chain_verification_runs(id),
  error_code TEXT CHECK (error_code IN
      ('coverage_incomplete','verifier_error','integrity_rejected','crashed')),
  finished_at TIMESTAMPTZ,
  -- The full state matrix (X4):
  CHECK ((state = 'running') = (claim_owner IS NOT NULL)),
  CHECK ((claim_owner IS NULL) = (claim_token IS NULL)
     AND (claim_owner IS NULL) = (claim_expires_at IS NULL)
     AND (claim_owner IS NULL) = (heartbeat_at IS NULL)),
  CHECK ((state IN ('done','error')) = (finished_at IS NOT NULL)),
  CHECK (state <> 'done' OR run_id IS NOT NULL),
  CHECK ((state = 'error') = (error_code IS NOT NULL)),
  CHECK (state <> 'queued' OR (run_id IS NULL AND captured_targets IS NULL
         AND attempts = 0))
);
CREATE INDEX chain_verify_jobs_queued_idx ON chain_verify_jobs (requested_at)
    WHERE state = 'queued';
CREATE INDEX chain_verify_jobs_reclaim_idx ON chain_verify_jobs (claim_expires_at)
    WHERE state = 'running';                                    -- reclaim scan (X4)
CREATE INDEX chain_verify_jobs_reap_idx ON chain_verify_jobs (finished_at)
    WHERE state IN ('done','error');                            -- reap scan (X4)
CREATE UNIQUE INDEX chain_verify_jobs_active_target_uq ON chain_verify_jobs
    (chain_family, fti_source) NULLS NOT DISTINCT
    WHERE state IN ('queued','running');       -- ONE active job per target (X4)

Guarded job fns (ALL with the full SECURITY DEFINER discipline — owner-transfer, SET search_path, REVOKE PUBLIC — plus PUBLIC probes in the grant test; NO raw DML grants to anyone, _app gets SELECT for polling):

chain_job_enqueue(p_family text, p_source text, p_loop text, p_requested_by text,
    p_incident uuid, p_max_queued int) RETURNS TABLE (job_id uuid, created boolean)
--   An existing active job for the target → (its id, false) — the handler maps
--   to 409 verification_in_progress (idempotent: no duplicate work, the caller
--   learns the in-flight id). Global queued count >= p_max_queued → RAISE
--   (handler → 503). p_incident must reference an existing incident.  [_app]
chain_job_claim(p_family text, p_source text, p_worker text, p_claim_secs int)
    RETURNS SETOF chain_verify_jobs
--   TARGET-SCOPED (X4): each family task claims ONLY its own (family, source).
--   FOR UPDATE SKIP LOCKED, oldest queued OR running-with-expired-claim
--   (reclaim). Mints claim_token := uuidv7(), attempts := attempts + 1,
--   claim_expires_at := now() + p_claim_secs (domain 1..=600 in-fn).
--   captured_targets is PRESERVED on reclaim.                        [verify]
chain_job_capture(p_id uuid, p_claim_token uuid, p_instance uuid, p_epoch int,
    p_targets jsonb) RETURNS void
--   Token-validated; writes the work definition ONLY when captured_targets IS
--   NULL (first capture wins; a reclaimer sees non-NULL and RESUMES).  [verify]
chain_job_heartbeat(p_id uuid, p_claim_token uuid, p_claim_secs int) RETURNS void
--   Token-validated; extends claim_expires_at, stamps heartbeat_at.   [verify]
chain_job_finalize(p_id uuid, p_claim_token uuid, p_state text, p_run uuid,
    p_error_code text) RETURNS void
--   Token-validated; state ∈ {done, error}; nulls the claim fields; called in
--   the SAME transaction as chain_run_record (VerifierDb::record_run_in).
--                                                                     [verify]
chain_job_reap(p_older_than_days int) RETURNS bigint
--   Floor enforced IN the fn: p_older_than_days >= 7 (RAISE below).
--   Terminal-only; rows whose incident_id references a still-latched incident
--   are EXEMPT until resolved.                                        [verify]

Job execution (crash-safe, family-lease-integrated — X2/X4): the family-lease holder claims its target’s jobs FIRST each pass. First claim: capture instance/epoch/targets from live heads (chain_job_capture); reclaim: RESUME the captured vector. The job protocol iterates passes under the global budget until every shard of the family reaches the CAPTURED target for the requested loop (family-full = tail + scrub
census + manifest; it records one run per loop, the job’s run_id pointing at the family run). Heartbeat per batch (CHAIN_JOB_HEARTBEAT_SECS); claim lease CHAIN_JOB_CLAIM_SECS (validated ≥ 3× heartbeat). Manual runs carry mode = 'manual' + job_id — they NEVER feed the status machine in either direction (X4). Any shard unreached at claim expiry → the claim lapses and a reclaimer resumes; attempts > CHAIN_JOB_MAX_ATTEMPTSerror/crashed. Coverage is all-or-nothing: finalize done only with full self-verified coverage of the captured targets, else error/coverage_incomplete. Finalize + run write are ONE transaction. Reap after 7 days.

D9 — canopy-web: the typed-terminal-503 client, badge, banner, citation

  • The client path comes first (X9): the generic canopy-web client maps every non-2xx to Err and RETRIES 503s with discarded bodies (clients.rs:190) — under the D6 mapping, breached IS a 503 with a body, so the badge could never render it. MR-2 adds an endpoint-specific terminal fetch (clients.rs): no retry on 503, and the ChainStatusResponse body is deserialized on BOTH 200 and 503.

  • project_chain_status (stream.rs:148-183) rewritten over ChainStatusResponse: healthy → verified pill; verifying → neutral "Verification in progress" (+ backlog note); breached → broken pill + incident id; everything else / transport error → "Unable to verify chain". The dead per-row broken_at == event.id matcher (stream.rs:192-216) is REMOVED; the row banner derives from the TYPED breached_position field (instance/family-filtered — X9), not a string note.

  • case_detail/sections/audit.rs:105-145 + api/audit_log.rs:191-217: same swap; the audit section Plugin.toml is touched alongside.

  • Citation PDF (audit_log.rs:472-545): the flow is pinned as fetch-by-PK → attest-by-event_id (the handler resolves the cited row by its primary key, then attests the row’s event_id — rename-stable, archive-safe). attested: false or transport error → 502 "citation not issued" (posture preserved); PDF inputs = verified_through/trusted_anchor_seq. The TEMPLATE is in the file list: rulesets/georgia/notices/audit/citation.typ:111 renders the verification block — updated with the new fields + its notice render tests.

  • Test-lib client verify_chain() retargets to the unified status endpoint
    ChainStatusResponse; web api-docs snapshot refreshed.

CLI parity (ADR-007 — X9): the CLI hardcodes the deleted path (tools/canopy-cli/src/cmd/security.rs:66). MR-2: canopy security chain-status --family [--service], canopy security chain-verify --family [--service] [--loop] [--wait] (trigger + poll), canopy security chain-attest --event-id --family [--service] — thin wrappers over the unified endpoints; MR-3 activates the FTI arguments. The ADR-007 parity inventory is updated in the same MRs.

D10 — the FTI arm (#1206) + the preserved legacy breach (X8)

  • Three verify-pool URLs (D11) as canopy_security_verify (LOGIN at #1279), pool size 2. Startup validation is parse-only; connectivity is per-family and LAZY (X10): a present-but-unreachable FTI DB degrades THAT family to error/stale at read time and retries on cadence — one program outage never blocks the process or the other families (the fail-fast contradiction resolved in favor of isolation). The (family, source) topology-slot match runs at each family’s FIRST successful connect — a swapped tanf/medicaid URL pair still fails that family loudly, in its own lane. Dormant = unconfigured.

  • Per-family tasks share the engine + config; checkpoints/runs/incidents key by instance in the security DB; head/lag reads on the CHAIN pool’s chain_heads.

  • FTI status/attest arms resolve positions via the verify pools on the id column (both sides indexed — D2).

  • The legacy latched breach STAYS VISIBLE (X8 — reversing the v5 cut, which was wrong): #1245’s handler + seeded-breach test (api/mod.rs:676, security_test.rs:683) deliberately keep a latched v1 FTI breach visible — "a breach is never silently swallowed" is a SAFETY invariant. MR-3’s FTI status arm ORs the legacy latched row into breached (reason legacy_breach_latched) until #1279 drops the table. The store reader is NARROWED to the breach bit (legacy_fti_breach_latched(pool, service) → bool replacing the FtiChainVerification DTO chain); the seeded-breach test SURVIVES, retargeted to the unified endpoint.

  • Archive-aware in-vivo proof (both families): seed → append → superuser-move a prefix
    advance archived_through → scrub verifies THROUGH the archived range and the live suffix; mutate an archived payload (coherently: payload + derived columns together) → scrub latches hash_mismatch; mutate INCOHERENTLY (payload only) → derived_column_mismatch (the honest two-arm split, X10); purge the boundary row → boundary_unavailable.

  • No writer-path changes in tanf/medicaid services; their DATABASES gain exactly the MR-1 migration (views + id + archive index + grants).

D11 — configuration

Env var (CANOPY_SECURITY__…) Default Domain (out-of-domain = STARTUP ERROR)

CHAIN_V2_VERIFY_ENABLED

false

bool

CHAIN_VERIFY_TICK_MS

1000

10..=60000

CHAIN_VERIFY_FIRST_TICK_DELAY_SECS

60

0..=3600

CHAIN_VERIFY_BATCH_SIZE

1000

1..=10000

CHAIN_VERIFY_BATCH_BYTES

33554432 (32 MiB)

4 MiB..=256 MiB — the floor equals the D1a hard row ceiling, so one max-size row ALWAYS fits the budget

CHAIN_VERIFY_PASS_BUDGET

16

1..=256 — GLOBAL shard VISITS per pass, every visit counted, zero-work refreshes included (X10)

CHAIN_SCRUB_BATCHES_PER_PASS

1

1..=64, validated < CHAIN_VERIFY_PASS_BUDGET (the scrub share can never starve the tail)

CHAIN_CENSUS_INTERVAL_SECS

300

30..=86400

CHAIN_VERIFY_LEASE_SECS

30

5..=300 — shard AND family leases; also the worst-case crash-recovery pause (expiry-only takeover, X1)

CHAIN_VERIFY_STATEMENT_TIMEOUT_MS

5000

1000..=10000 — from_config validates lease_secs × 1000 ≥ 3 × statement_timeout_ms as a LIVENESS heuristic (X1: correctness rides the token; the check keeps a healthy holder from being contested mid-batch, nothing more)

CHAIN_VERIFY_LOCK_TIMEOUT_MS

2000

100..=10000

CHAIN_JOB_CLAIM_SECS

60

10..=600, validated ≥ 3 × CHAIN_JOB_HEARTBEAT_SECS

CHAIN_JOB_HEARTBEAT_SECS

15

1..=200

CHAIN_JOB_MAX_ATTEMPTS

3

1..=10

CHAIN_JOB_MAX_QUEUED

8

1..=64 — passed into chain_job_enqueue (fns read no config)

CHAIN_TAIL_MAX_AGE_SECS

300

≥30

CHAIN_TAIL_MAX_LAG

100000

≥1

CHAIN_SCRUB_MAX_AGE_SECS

172800

≥300 — against cycle_completed_at, falling back to cycle_started_at on a never-completed first cycle (X5)

CHAIN_MANIFEST_MAX_AGE_SECS

604800

≥300 (#1278 tightens)

CHAIN_VERIFY_DATABASE_URL / TANF / MEDICAID

String, Debug-redacted (workspace settings pattern); presence + URL-parse validated at boot when enabled; CONNECTIVITY is lazy per family (X10)

ChainVerifyConfig::from_config (the ChainDrainConfig pattern), unit-tested; every relationship above validated at startup, never clamped.

D12 — dormancy + the #1279 handshake

  • Flag off (default): no pools, no tasks, no job servicing; the unified status endpoint serves unknown → 503 (the #1245 posture by status code); POST verify → 503 verifier_unavailablechain_job_enqueue is never called for an unconfigured target (no phantom queue, X4). The ChainStatusInputs sampler runs always.

  • #1279 flips: LOGIN carriers (verify + incident-admin + the emitter for #1278) + three URLs + flag. Sequence after genesis-anchor confirmation: unknown → verifying → healthy with zero backlog; badge in vivo; ATTESTATION of real events additionally waits for #1278’s first confirmed periodic tip.

  • v1 retirement at #1279: the fti_chain_verifications table drop + the X8 breach-bit reader + the legacy_breach_latched reason retire together (tracked on #1279’s list at 0a).

D13 — perf evidence (numeric gates; cargo xtask perf chain-verify)

Audit dimension runs in MR-2 (before Closes #1205); MR-3 adds the FTI dimension. Release build, devstack PG, seeded via append_chained_rows through the real append path with the PayloadDist parameter (X10).

  • Corpus: 2,000,000 rows/family; payload mix 1 KiB typical / 64 KiB p99; shard_count ∈ {2, 8}; 30s warmup, 60s window, 2 reps, mean ± spread.

  • Catch-up: tail from genesis — PASS ≥ 5,000 rows/s per task at the 1 KiB mix, and ABSOLUTE process RSS < 256 MiB throughout the run (X10 — an absolute gate, not a delta a bloated baseline can hide).

  • Sustained: producer at 300/s concurrent — PASS: lag slope ≤ 0 over the window.

  • Independence: status GET and attest GET sampled at 0.5M/1M/2M rows — PASS: p95 < 50 ms at each size AND max/min p95 ratio < 1.5 (flat).

  • Plans: EXPLAIN assertions — the attestation lookups (live + archive, both families), the D1a union batch fetch, and chain_job_claim use index scans at 2M rows.

  • The D14 fence KATs + proptest gate in-battery (correctness, not perf).

D14 — the JSON number fence: #1285 CLOSED, not observed (X7)

The hole (canon.rs:48-70): validate_number range-checks integers but passes floats, and RFC 8785 renders every number THROUGH f64 — so distinct stored values that collide in f64 (the canonical example: 2^53 vs 2^53 + 1) canonicalize to identical bytes. Hash recomputation alone cannot detect that mutation class. A KAT can only demonstrate the collision; the fence CLOSES it.

Deliberate non-dependency: serde_json’s `arbitrary_precision feature is NOT enabled — cargo feature unification would change `serde_json::Number’s parse behavior for EVERY workspace crate. The fence is a self-contained scanner in canopy-chain.

The fence (canopy_chain::canon::raw_number_fence(text: &str) → Result<(), NumberFenceViolation>, ~80 lines + tests): walk the JSON text once, skipping string literals (escape-aware); for each number token t (RFC 8259 grammar):

  1. Parse t as f64 (overflow to ±inf → violation).

  2. Render the f64’s SHORTEST round-trip decimal (std Display — shortest-digit guaranteed).

  3. Compare AT VALUE LEVEL: normalize both t and the rendering to (sign, digit string, decimal exponent) and require equality. Accept iff decimal_value(t) == decimal_value(shortest_repr(f64(t))).

Value-level (not text-level) comparison is the load-bearing choice: PostgreSQL stores jsonb numbers as exact numeric and REFORMATS on output (1e30 prints as its 31-digit expansion), so text equality would false-positive on every legitimate rendering difference, while VALUE equality accepts exactly the values JCS can represent losslessly and rejects every f64-collision mutation: a stored 2^53 mutated to 2^53 + 1 prints a token whose decimal value differs from its f64 round-trip (9007199254740993 ≠ 9007199254740992) → violation; notation differences (1e30 vs the expansion vs 2.3e1 vs 23) normalize equal → pass. -0 normalizes to 0 → pass (JCS renders it "0").

Where it runs:

  • Verify-side (MR-1): step 0 of the D1a per-row checks, over the fetched canonical_event_payload::text (and the audit metadata column text) — violation → malformed_row latch with the offending token + offset as evidence. This catches every historical or mutated row.

  • Intake-side (MR-2): the same fence over the RAW BODY BYTES in ingest_audit_event (api/mod.rs:236-265; violation → the 422 I-JSON class extends by one code) and over the staging consumer’s delivery bytes before staging (chain_staging/stage.rs; violation → park as poison, its oversize-integer filter precedent) — no new collision-class value can enter.

Pinning: KAT vectors (2^53 − 1 / 2^53 / 2^53 + 1, exponent vs expansion forms, 0.1, -0, 1e400 overflow, value-preserving reformattings) + a proptest invariant — for every f64, raw_number_fence accepts its shortest repr AND every tested value-preserving reformatting; for every integer magnitude above 2^53 not exactly representable, the literal is rejected. #1285’s adjudication resolves to this fence (0a notes it; closed at MR-2 merge).

Scope — explicitly OUT

  • Anchor emission, the external authority, submission/confirmation loops, cadence/SLO → #1278 (WITH the emitter role/arm split + the D5 confirmer contract).

  • Epoch closure/rollover + closed-epoch verification → #1280.

  • Purge-boundary manifests + the archive/purge movers → #1208/#1247 (verification of RETAINED rows, archived included, is IN scope here).

  • The incident-resolution UI → the 0a-filed follow-up issue (0a’s FIRST action).

  • ele_grant → #1248. Cutover execution + table drops → #1279.

Verification (test inventory — both external reviews' cases included)

canopy-chain unit (MR-1): raw_number_fence_kats (the D14 vector set); raw_number_fence_value_roundtrip_props (proptest — the D14 invariant); manifest_decode_round_trips_and_rejects_noncanonical (+ KATs).

canopy-common unit: derive_status_precedence_table (exhaustive: every backlog input singly, unsampled-blocks-healthy, manifest-absent/aged, per-shard error isolation — shard A’s error never masks/clears via shard B (X5), first-cycle-never-completed goes stale (X5), dead-worker read-time stale); reject_kinds_are_closed_and_match_sql (Rust enum = the D2 CHECK vocabulary = the kind→loop mapping); verify_batch_math_props (proptest); payload_closed_sets_match_the_append_fns; dto_serde_shapes (every D8 enum’s wire strings pinned; domain→wire status bijection).

canopy-security tests/chain_verify_engine_test.rs (EphemeralSchema, audit family): confidentiality + fencingtoken_column_unreadable_by_verify_role (SET ROLE probe on the view + the table — X1); concurrent_cold_acquire_admits_exactly_one; acquire_requires_init_cursor_on_absent_row; expiry_only_takeover_same_owner_waits (X1 — no mid-lease self-reacquire); advance_rejects_unknown_token_and_absent_row; token_survives_expiry_until_takeover (X1); fenced_worker_cannot_latch_or_record; cursor_regression_rejected_equal_seq_hash_divergence_raises; family_lease_serializes_pass_and_gates_runs (X2 — run record demands the family token; a shard token cannot record); family_scoped_finding_latches_under_family_token_pre_shard_lease (X2); proofsadvance_cycle_complete_requires_target_equality (X5); advance_manifest_ref_requires_confirmed_anchor_same_identity (X5); advance_manifest_ref_rejected_on_shard_keys (X5 — family row only); cycle_start_cas_stale_reset_loses (X5); cycle_start_partial_param_set_raises; integritypayload_rewrite_detected; previous_hash_rewrite_detected; seq_gap_and_duplicate_detected (incl. the cross-side duplicate arriving adjacent — D1a); derived_column_mutation_detected_per_column; routing_misplacement_detected; foreign_source_detected; payload_set_violation_detected; malformed_persisted_row_latches_not_retries (incl. a D14 fence violation); head_regression_latches; genesis_mismatch_census_arm; whole_shard_deletion_detected; rows_beyond_head_single_snapshot_no_false_positive; wrong_side_and_archived_orphan_rows_detected (X6); coherent_post_capture_rewrite_hits_target_hash_mismatch; scrub/archivescrub_detects_mutation_behind_tail; scrub_covers_archived_rows_and_detects_archived_mutation (coherent arm → hash_mismatch; incoherent arm → derived_column_mismatch — X10); scrub_cycle_fixed_target_terminates_under_continuous_writes; archive_move_during_walk_single_statement_no_tear (X6); archive_move_concurrent_with_census_no_false_latch (retry arm); purged_boundary_fails_closed; oversized_first_row_admitted_then_ceiling_latch (X6 — one row always admitted; a > 4 MiB row latches malformed_row); manifestmanifest_row_metadata_mismatch_latches; newly_confirmed_anchor_mid_check_refetches_not_latches; attestation_binds_to_trusted_ref_not_newest_anchor; incidentslatch_is_idempotent_under_race (incl. family-scoped NULL epoch/shard — the NULLS-NOT-DISTINCT arm, X3); incident_evidence_unreadable_by_verify_role (X3); resolve_enforces_manual_mode_detected_loop_scope_and_session_actor (X3/X4 — a scheduled ok run and a wrong-loop manual run both REJECTED); resolution_runbook_restores_loops; clean_pass_never_clears_breached.

canopy-tanf + canopy-medicaid tests/chain_v2_verify_test.rs (two-schema topology): fti_tail_and_scrub_verify_real_carveout_appends; fti_view_exposes_id_and_id_mismatch_detected; fti_field_coverage_rehash_matches; fti_two_instances_are_disjoint; fti_attestation_resolves_on_id_column_indexed (EXPLAIN, live + archive).

canopy-security tests/chain_verifier_host_test.rs: dormant_flag_off_unknown_503_and_post_refuses; delayed_first_tick_no_boot_walk (paused time); pass_budget_counts_zero_work_visits_and_rotates (X10); per_family_task_isolation_broken_pool_stalls_one_family (paused time + a poisoned pool — the LAZY connectivity arm, X10); halt_gate_rechecked_between_batches; backlog_inputs_hold_verifying_and_unsampled_blocks_healthy; status_http_mapping_all_six_states; chain_request_error_matrix (one named assertion per D8 request-error row); job_lifecycle_queued_running_done; job_reclaim_preserves_captured_targets (X4 — crash → reclaim resumes the SAME vector); job_claim_token_fencing (a stale claimant’s heartbeat/finalize rejected); job_enqueue_409_on_active_target_and_503_at_cap; job_crash_reclaim_attempts_then_crashed; job_all_or_nothing_on_lease_contention (coverage_incomplete); job_poll_requester_scoped; job_finalize_and_run_are_one_tx; manual_runs_never_feed_status (X4 — a manual ok run clears nothing; a manual error run degrades nothing); grants_match_the_matrix (SET ROLE probes: app reads projections + polls jobs, cannot DML jobs or C6 tables; verify reads ONLY the token-free/evidence-free views, cannot resolve incidents, append anchors, or confirm-as-emitter; emitter cannot confirm; PUBLIC executes nothing; to_regprocedure(<each old signature>) IS NULL); openapi_path_count_17_and_snapshot (MR-2) / …_16 (MR-3); attestation_refuses_foreign_instance_or_epoch (X9).

canopy-web: six-state pill matrix; typed_terminal_503_renders_breached (X9 — the generic-client retry path proven bypassed); row banner from breached_position; citation attest-false → 502 / attest-true → PDF inputs (+ the citation.typ render tests); dead row-matcher removed.

canopy-security tests/security_test.rs (MR-3): the #1245 seeded-breach test SURVIVES retargeted — legacy_fti_breach_stays_visible_on_unified_status (X8).

CLI: chain-status/chain-verify/chain-attest argument matrices + a poll-loop test against a mock (MR-2), FTI args (MR-3).

Gates, all MRs: full battery; SPDX; proptest on the walker + the fence; B3a STRUCTURAL-VALUE markers + per-MR lock delta; B3b for test imports.

Files touched (by MR)

0b (this commit): this plan + adr-014-fti-audit-hash-chain.adoc (Amendment 9)
parent scale-audit-adr014-chain-v2.adoc (Amendments 8–9 wording)
architecture.adoc + CHANGELOG.adoc. (nav already links this plan.)

MR-1: services/canopy-security/migrations/20261010000000_chain_verification_hardening.sql (new — the full D2 set); services/canopy-{tanf,medicaid}/migrations/ 20261010000000_fti_preimage_id.sql (new — both FTI preimage views gain id
fti_audit_log_archive_v2_id_uq + re-grants); crates/canopy-test-lib/src/db.rs (touch); crates/canopy-chain/src/{canon.rs (raw_number_fence + NumberFenceViolation), anchor.rs (from_canonical_bytes, PartialEq)} + KAT vectors under crates/canopy-chain/tests/vectors/; crates/canopy-common/{src/chain_verify.rs (new), src/lib.rs, Cargo.toml}; crates/canopy-test-lib/src/chain.rs (append_chained_rows + PayloadDist); services/canopy-security/tests/chain_verify_engine_test.rs (new); services/canopy-{tanf,medicaid}/{tests/chain_v2_verify_test.rs (new), Cargo.toml}; the pre-hardening substrate suites updated to the reshaped surface — services/canopy-security/tests/chain_v2_substrate_test.rs (the transition-arm split, the token protocol smoke, token-guarded latch/resolve with the manual-run + session-actor proofs, the guarded-fns-only write matrix, the 17-object ownership + dropped-signature ACL matrix) and services/canopy-{tanf,medicaid}/tests/chain_v2_substrate_test.rs (the pinned preimage column lists gain the appended id); CHANGELOG.adoc; own Status row.

MR-2: services/canopy-security/migrations/20261015000000_chain_verification_projections.sql (new: _app views + jobs table + job fns + the runs FK/CHECK + the same-signature chain_run_record replace) + test-lib touch; services/canopy-security/src/ {chain_verify/{host,status,inputs,jobs}.rs (new), lib.rs, main.rs (queue constant), api/mod.rs (unified endpoints; `verify-chain GET + FTI verify POST deleted; the D14 intake fence in ingest_audit_event; :578 doc fix), config.rs, chain_staging/stage.rs (the delivery-bytes fence beside its oversize-integer filter), chain_staging/health.rs (additive sampler_untrustworthy), Cargo.toml (tokio test-util dev-dep)}`; crates/canopy-contracts-security/src/ {chain.rs (the D8 DTO set; `ChainVerificationResponse removed), fti.rs (doc-note), events.rs (:191 doc fix), paths.rs (−2 4)}` + contracts roundtrip tests; `crates/canopy-test-lib/src/clients/security.rs`; `services/canopy-web/src/{clients.rs (typed-terminal-503), audit/stream.rs, case_detail/sections/audit.rs, case_detail/sections/audit/Plugin.toml, api/audit_log.rs}`; `rulesets/georgia/notices/audit/citation.typ` + notice render tests; `tools/canopy-cli/src/cmd/security.rs` ( CLI tests); xtask/src/cmd/perf.rs (chain-verify, audit dimension); OpenAPI snapshots (security + web); docs — api/canopy-security.adoc, data-models/canopy-security.adoc, security-operations.adoc (resolution runbook + credential provisioning), configuration-reference.adoc, rbac-matrix.adoc (:99 names the old FTI paths), auditor-handbook.adoc, nist-architecture-mapping.adoc, user-testing-guide.adoc, runbooks/database-backup-restore.adoc (:453 — deleted paths + the old sync trigger), the ADR-007 parity inventory, CHANGELOG.adoc; tests/e2e/specs/ audit-rail.spec.ts; own Status row.

MR-3: services/canopy-security/src/chain_verify/ (FTI targets/pools) + config.rs (URLs) + api/mod.rs (FTI arms + the X8 legacy-breach OR; GET /v1/security/fti/chain-status deleted) + store/{mod,models}.rs (FtiChainVerification(+Row) + latest_fti_chain_verification deleted; the narrowed legacy_fti_breach_latched reader added); tests/security_test.rs (the retargeted seeded-breach test); crates/canopy-contracts-security/src/{fti.rs, paths.rs (−1)}; tools/canopy-cli/src/cmd/security.rs (FTI args); xtask/src/cmd/perf.rs (FTI dimension); host-test extensions + archive suites; OpenAPI snapshot (16); docs — api/canopy-{tanf,medicaid}.adoc, api/canopy-security.adoc, user-testing-guide.adoc (FTI flow), rbac-matrix.adoc (the GET /v1/security/fti/chain-status row dies here, not in MR-2), CHANGELOG.adoc; own Status row.

Step 4 (docs close-out): verify auto-closure of #1205 + #1206 + post closing comments; parent Step 4 → Done; plan → Archive + nav; perf cells on #1279.

Sequencing & review-risk notes

0b → MR-1 → MR-2 → MR-3 → Step 4, strictly. Honest scope statement: this plan performs a substrate-hardening migration set — two reshaped checkpoint fns (with in-fn proofs), three guarded writer fns, the family loop kind, the detected-loop incident model, the token/evidence confidentiality split, CHECK matrices, the NULL-safe dedup, a runs index, archive indexes, the anchor role/arm split, and the FTI view change — all on the dormant, pre-1.0 chain-owned surface, all following the substrate’s own idioms, all flagged in §Open decisions. The FTI databases gain a migration (views
index + grants). MR-2 carries the wire-shape replacement + two path deletions + the unified namespace (pre-1.0 Changed entries; every consumer migrated in-MR — web, CLI, test-lib, template, docs; the #1245 fail-closed posture preserved by the HTTP mapping and, for the legacy FTI breach, by the X8 arm in MR-3).

Open decisions for sign-off

  1. The substrate-hardening migration set (D2) — ratified for v5, EXTENDED in v7: token confidentiality (the verify role loses raw C6 SELECTs), duration-based DB-computed expiry, expiry-only takeover, in-fn health-stamp proofs, the family loop kind, the detected-loop incident model, the anchor ARM split. Hardening the dormant substrate remains the architecturally correct fix over client discipline.

  2. The FAMILY lease (X2) — one replica runs an entire family’s pass; scale-out is BY FAMILY while shard tokens keep write-level fencing. Ratified by the v7 approval; Amendment 9 records it.

  3. Backlog → verifying / HTTP 200 with enumerated reasons (ratified 2026-07-31).

  4. Family halt on latch (+ between-batch recheck; the bounded in-flight window is documented).

  5. Manual jobs: durable guarded queue, target-scoped, all-or-nothing coverage, 409 on an active target, manual runs quarantined from status (X4).

  6. Incident resolution stays runbook-only (guarded fn + session actor + manual-mode
    detected-loop enforcement; the UI follow-up is 0a’s first-filed issue).

  7. Archived-row verification in scope; single-statement union reads (X6).

  8. The number fence closes #1285 (D14) — verify-side MR-1, intake-side MR-2; no arbitrary_precision dependency.

  9. Legacy FTI breach visibility preserved until #1279 (X8) — a safety invariant, not compat; the v5 cut is withdrawn.

Edit this page · default