Plan: chain-v2 verifiers — family-leased tail + scrub engine, token-confidential fencing, the unified chain namespace, citation attestation (#1205 audit, #1206 FTI, epic &73)
On this page
- Status
- Step 0 — lifecycle
- Context (recon facts — verified
file:lineatf6fca4c7) - D1 — the shared verify engine (
canopy_common::chain_verify, MR-1) - D2 — the hardening migration: token confidentiality, the family loop, proofs, guarded writers
- D3 — the family pass (X2): one lease orders everything
- D4 — the historical scrub loop (full retained history)
- D5 — census + the manifest trust chain
- D6 — the C6 status machine + backlog inputs + the
_appprojection - D7 — incidents: latch, evidence, resolution
- D8 — the unified
/v1/security/chain/*surface + durable jobs - D9 — canopy-web: the typed-terminal-503 client, badge, banner, citation
- D10 — the FTI arm (#1206) + the preserved legacy breach (X8)
- D11 — configuration
- D12 — dormancy + the #1279 handshake
- D13 — perf evidence (numeric gates;
cargo xtask perf chain-verify) - D14 — the JSON number fence: #1285 CLOSED, not observed (X7)
- Scope — explicitly OUT
- Verification (test inventory — both external reviews' cases included)
- Files touched (by MR)
- Sequencing & review-risk notes
- Open decisions for sign-off
Implements ADR-014 Amendment 5 C6 (with the C5 manifest-divergence consumer arm) under the Amendment 6 corrections, the Amendment 7 transport bindings, the Amendment 8 verifier bindings, and the Amendment 9 revisions this version introduces (accepted amendments are immutable — Amendment 9 is the formal revision vehicle). Parent rollout: ADR-014 chain-v2 (Step 4).
Review state: v7. v1→v4: three internal contextless rounds. A FIRST external review
rejected v4 (~50 integrity/fencing/recovery/activation findings); the v5 rework (W1–W11)
was approved, committed (MR !1054, merge a2d2b33f), and corrected by a 0c fold
(f6fca4c7). A SECOND external review then REJECTED v5 with ~14 blocking + ~25 high
~10 material findings — token confidentiality, family-scoped fencing, durable jobs,
proven health stamps, single-snapshot reads, the number-collision class, and a
silently-erased #1245 safety invariant. v7 is the full rework, approved as workstreams
X1–X10 (2026-08-01): lease-token confidentiality + honest lease semantics (X1); the
FAMILY lease ordering manifest/census/shard loops/jobs under one holder (X2); the
detected-loop incident model with NULL-safe dedup and the evidence split (X3); durable
token-claimed target-scoped bounded jobs (X4); scrub/status correctness under the lock —
cycle-start CAS, PROVEN completion/manifest stamps, first-cycle staleness, per-scope
error clearing (X5); one-statement archive ∪ live reads + whole-history census + bounded
rows (X6); the JSON number fence CLOSING the collision class, not observing it (X7);
legacy FTI breach visibility PRESERVED until #1279 — reversing a wrong v5 cut (X8); the
completed wire contract — the unified /v1/security/chain/* namespace, full
de-pseudocoded SQL/DTOs, CLI parity, the honest ripple inventory — ratified as ADR-014
Amendment 9 (X9); and scheduler/config/constraint/test closure (X10). file:line
verified against main == f6fca4c7 (0b123d38 → f6fca4c7 is docs-only — the v5 plan
commits themselves).
Status
| Step | Description | Status |
|---|---|---|
0 |
Plan lifecycle: v5 committed (!1054 + the 0c fold) → SECOND external review REJECTED → X1–X10 rework approved (2026-08-01) → apply v7 (this text) + ADR-014 Amendment 9, commit direct to main (docs-only, full battery) → external re-review of the COMMITTED v7 artifact → user sign-off → 0a tracker pass (now including the #1205/#1206 AC updates + filing the incident-UI issue). Implementation starts only after re-review + sign-off. |
Done (2026-08-01) — v7 committed at ddecb378; sign-off received; 0a complete (#1205/#1206 reopened + v7 ACs, #1289 filed, tracker notes on #1208/#1247/#1278/#1279/#1280/#1285) |
1 |
MR-1 substrate hardening + verify engine + the number fence ( |
Done (2026-08-01) — implemented on |
2 |
MR-2 audit verifier + the unified chain namespace + jobs + web + CLI + perf ( |
Done (2026-08-01) — implemented on |
3 |
MR-3 FTI verifier ( |
Done (2026-08-02) — implemented on |
4 |
Post-merge docs close-out: verify auto-closure of #1205 + #1206 and post the closing comments (SHAs); parent Step 4 → Done; this plan → Archive + nav; perf cells posted on #1279. (Docs-only, committed direct to main per git-workflow.) |
Done (2026-08-02) — #1205/#1206 auto-closure verified + closing comments posted (impl + merge SHAs); audit + FTI perf cells posted on #1279 with the v1-retirement scope reminder; epic &73 updated; this commit archives the plan (nav → Archive) |
Epic &73
Issues #1205 (critical) + #1206 (critical) — blocked by #1246 (Done), #1236 (Done), #1207 (Done); both block #1279; #1285 (number-boundary adjudication) is CLOSED by this plan’s D14 fence
Branches v7 rework commits direct to main (docs-only), then per the Status table
Local cargo xtask validate runs before each MR’s first push.
Step 0 — lifecycle
Order (v7 rework round): apply X1–X10 to this artifact + append ADR-014 Amendment 9
parent-plan/architecture.adoc/CHANGELOG.adoc touches → ONE single-agent coherence
pass (self-consistency after the large rework — not a new review round) → commit direct
to main (docs-only, full battery) → external re-review against the COMMITTED artifact →
user sign-off → 0a → MR-1.
-
0a — tracker reconciliation (post-sign-off, before any code). FIRST action: file the incident-resolution UI follow-up issue (out of plan scope but tracked in GitLab, per plan-lifecycle). Then:
-
#1205 + #1206 own ACs: update both issues' acceptance criteria to the v7 surface — the unified
/v1/security/chain/*endpoints (the AC text still names the deleted paths), the path counts (17 → 16), the family-lease model, the durable-job manual verify, and the #1285 fence dependency — BEFORE any implementation MR references them. -
#1208 + #1247: the verifiers cover archived rows (scrub + attestation + census span archive ∪ live); ONLY the purge-boundary manifest machinery and the movers remain the archive children’s. A purged boundary surfaces as fail-closed
boundary_unavailable. -
#1280: #1205/#1206 verify the single currently-active epoch; closed/closing-epoch verification and cross-epoch genesis-anchor linkage extend with the rollover executor.
-
#1278: the anchor-consumption contract — the verifiers CHECK anchors and bind status/attestation to the last verifier-checked manifest; emission + submission + the authority integration are #1278’s, WITH the role/arm split (D2): the emitter role owns
chain_anchor_append+ the emitter transition arm; the verifier keepschain_anchor_transition_confirmonly. The D5 confirmer-contract note is the #1278-facing interface. Status consumes a manifest-age threshold #1278 tightens. -
#1285: closed by D14 — note the disposition (verify-side fence MR-1, intake-side fence MR-2) and close on MR-2’s merge.
-
#1279: go/no-go additions — verify LOGIN + pool URLs + flag; delayed first tick;
unknown → verifying → healthyafter genesis-anchor confirmation with zero backlog; badge in vivo; attestation gated on the first periodic tip; the legacyfti_chain_verificationsTABLE drop retires the X8 breach-bit reader.
-
-
0b — the v7 plan commit (docs-only, direct to main): this plan (already nav-linked) + ADR-014 Amendment 9 + parent plan (Amendments 8–9 wording) +
architecture.adoc
CHANGELOG.adoc. -
0c — external re-review runs against the COMMITTED artifact. Implementation MRs start only after it passes AND the user signs off.
ADR-014 Amendment 9 (committed with this text — full content in the ADR, summarized):
the unified /v1/security/chain/* namespace (the two historical status paths + the FTI
verify POST deleted, pre-1.0; Amendment 8’s "preserved on the same paths" sentence
formally revised); the FAMILY lease + pass ordering; lease-token confidentiality
expiry-only takeover + duration-bounded DB-computed expiry; PROVEN health stamps
(cycle-start CAS, relational completion/manifest proofs, first-cycle staleness); the
detected-loop incident model (stored loop, NULL-safe dedup, closed vocabulary, the
evidence split); durable target-scoped token-claimed jobs with manual/scheduled run
separation; single-snapshot archive ∪ live reads + the whole-history census + bounded
rows; the JSON number fence (#1285 closed); legacy FTI breach visibility preserved until
#1279; CLI parity (ADR-007) for the chain surface.
Context (recon facts — verified file:line at f6fca4c7)
Contract (ADR-014 Amendment 5 C6 adr-014-fti-audit-hash-chain.adoc:479-499; C5
:469-477; invariants :527-546; Amendment 8 :749-830):
-
Two loops per
(instance, family, epoch, shard): tail (fixed captured(target_seq, target_hash), bounded batches, exposeverified_through+ lag) and historical scrub (bounded, resumable, its OWN fixed trusted target; over the FULL retained history). v7 adds the family pass level above them (D3). -
MUST reject: missing/unexpected shards or heads; duplicate/missing/noncontiguous seq; wrong
previous_hash/formula version; a head that is not its terminal row; rows beyond the head; invalid genesis/epoch-anchor linkage; orphan heads or rows (v7: on BOTH sides of the archive seam); divergence from the latest external manifest. -
States
unknown|verifying|healthy|stale|error|breached; separate tail/scrub coverage/freshness PER SHARD;stalecomputed at READ time; breach latched until authorized actor + reason + evidence + revalidation; nonzero staging backlog neverhealthy; status DTOs incrates/canopy-contracts-security/src/{chain,fti}.rsreplaced; checkpoint identity(instance, family, epoch, shard, loop-kind)
fence/CAS; manual verify = job id + polling URL. -
Citation: event-specific coverage or a dedicated attestation endpoint; fail-closed for newer-than-checkpoint / unknown / stale / error / breached.
-
C8: verification reads the hashed-preimage PROJECTION, never
SELECT *; verification-state writes go through guarded SECURITY DEFINER fns; incident RESOLUTION authority separated from the background verifier (Amendment 6:682-688). v7 extends C8 inward: the verify role itself loses raw SELECT on the checkpoint table (the token column) and the incident table (evidence) — D2.
Substrate as-built (#1246 20260910000000_chain_v2_substrate.sql; #1207
20260930000000_chain_append_staging.sql) — what v7 reshapes:
-
chain_verification_checkpoints(:291-307): PK(instance, family, epoch, shard, loop_kind),loop_kindCHECK('tail','scrub')(:296— gains'family'), no CHECKs otherwise, NULL cursor hash representable, no token column.chain_verification_runs(:309-321):loop_kindCHECK at:314(gains'family'), noshard_id/mode/job_id, no useful index.chain_incidents(:323-340): nullable epoch/shard (:327-328), openkind, no dedup invariant, no detecting-loop column, resolution fields caller-supplied. -
chain_checkpoint_advance(:583-620): INSERTs absent rows with a caller-supplied fence, accepts anyp_fence >= fencefrom ANY caller, never checks owner, permits cursor regression — an open write path. Reshaped in D2. -
chain_anchor_transition(:688-718): one fn, all edges, caller-supplied JWS/kid/refs — a single credential can fabricate AND confirm. Split in D2. -
Grants:
canopy_chain_verify= raw SELECT on ALL THREE C6 tables (:863— the checkpoint SELECT leaks every replica’s lease token, the core X1 finding; the incident SELECT leaks evidence), INSERT runs (:864, revoked), EXECUTE checkpoint-CAS / incident-latch / anchor append+transition (:865-868— the append grant and the emitter arm move to the emitter role); on tanf/medicaid strictly read-only (tanf:462;chain_status_vis NOT in the FTI verify grant — FTI head reads usechain_headson the verify pool).canopy_security_apphas NO C6 SELECT (:859vs:863) — the D6 projections close that. Login carriercanopy_security_verifyis NOLOGIN until #1279 (:56-57). -
Preimage views: audit (
:361-377) exposes position + hashes +chain_source
canonical_event_payload+ ALL typed derived columns (incl.event_id); FTI (tanf:192-204) exposes the same MINUS the hash-boundidPK — MR-1’s tanf/medicaid migrations addidto both FTI views (live + archive). Archive tables copy constraints but NOT indexes (LIKEcopies none beyond re-declared UNIQUEs;:242-248,chain_append_staging.sql:138-139) — MR-1 adds the archive-side attestation indexes. -
Genesis shape: heads pre-created
last_seq = 0,last_hash = empty_head_hash(…)(crates/canopy-chain/src/genesis.rs:54-77); epoch 0installinguntilchain_epoch_activate; genesis anchor tips alllast_seq = 0(:135-139). -
#1279 RENAMES the
*_v2tables at cutover — every request-path read here is view-mediated and survives.
Hash reproduction + the number hole (Rust; no SQL verify fn exists): ChainEnvelope
preimage keys (envelope.rs:59-72); event_hash = SHA-256 over canonical_bytes
(I-JSON validated RFC 8785, canon.rs:23-26); refetch-and-recanonicalize pinned by
Amendment 6 (:625-630). canon.rs:48-70 (validate_number) range-checks i64/u64 and
passes floats — so a FLOAT-FORMED token whose integer value sits near 2^53 (decimal or
exponent syntax; bare over-range integer tokens are refused by the range check), mutated
by ±1, canonicalizes to the SAME bytes (both render through f64), and hash recomputation
alone CANNOT see the mutation. The
second external review is right that a KAT merely observes this class; D14
CLOSES it with a value-level raw-token fence, and #1285’s adjudication resolves to that
fence. AnchorManifest (anchor.rs:85-108) has encode + hash but NO decode —
D1 adds from_canonical_bytes. shard_for (routing.rs:24-35) is the placement
function the engine recomputes. NOTE: serde_json’s `arbitrary_precision feature is
deliberately NOT used — cargo feature unification would silently change
serde_json::Number behavior workspace-wide; the fence is a self-contained scanner.
#1207 hand-off (backlog inputs): ChainStagingSnapshot { staged, oldest_staged_age_secs,
parked, sampled, degraded_reason } (chain_staging/health.rs:60-72), sampler always-on
30s (:208-220); staged rides RAW (:9-10); sampled starts false (:48-50) — an
unsampled input BLOCKS healthy. Inbox parks: newer-schema envelopes only (#1131,
subscriber.rs:923); count scoped queue_name = "canopy-security.audit" (a bare
literal at main.rs:71; MR-2 extracts the constant). DLQ depth: passive queue_declare
(metrics.rs:76-81 precedent) on dlq_queue_name("canopy-security.audit") via the
subscriber’s ConnectionManager (available in boot, main.rs:111). Intake surfaces for
the D14 fence: ingest_audit_event (api/mod.rs:236-265,
POST /v1/security/audit/ingest) + the staging consumer (chain_staging/stage.rs —
the home of the oversize-integer poison filter the fence extends).
v1 state being replaced — and the piece that SURVIVES (#1245 containment): all three
chain endpoints hard-503 with ChainStatusInterim (api/mod.rs:553-568,590-605,699-736,
754-769); verify_chain is test-only (store/mod.rs:434-448); the FTI verify loop is
already gone (#1245; main.rs:92-102 is the removal comment);
fti_chain_verifications has readers but no writers (store/mod.rs:332-348). The FTI
status handler deliberately keeps a latched v1 breach visible (api/mod.rs:676, pinned
by the seeded-breach test tests/security_test.rs:683) — that is the #1245 "a breach is
never silently swallowed" SAFETY INVARIANT, not compat, and v7 PRESERVES it until #1279
drops the table (D10; the v5 text calling it a compat cut is withdrawn). The
rest of the v1 chain dies as planned: canopy-web still deserializes
ChainVerificationResponse (also consumed by the test-lib client,
crates/canopy-test-lib/src/clients/security.rs:253-263) and renders "Unable to verify
chain" (stream.rs:148-183); the citation PDF hard-fails 502 (api/audit_log.rs:472-545).
Wire shapes are pre-1.0: replacements are CHANGELOG Changed entries, every consumer
migrated in the same MR. ChainStatusInterim/InterimChainState survive ONLY because
run_archive (#1208’s untouched surface) still returns them (api/mod.rs:654-670);
ChainStatusInterim.last_verification + FtiChainVerification(+Row) die in MR-3, with
the store reader NARROWED to the breach bit (D10); the table drop is #1279’s.
Downstream consumers the second review surfaced (all in the Files ripple):
the canopy-web generic client treats every non-2xx as Err and RETRIES 503s with
discarded bodies (services/canopy-web/src/clients.rs:190) — breached could never
render without the D9 typed-terminal-503 path; the CLI hardcodes the deleted
status path (tools/canopy-cli/src/cmd/security.rs:66) — ADR-007 parity commands land
in MR-2; the citation template (rulesets/georgia/notices/audit/citation.typ:111) and
the backup-restore runbook
(docs/modules/ROOT/pages/runbooks/database-backup-restore.adoc:453) reference the old
paths/trigger; rbac-matrix.adoc:99 names the deleted FTI endpoints; plus
auditor-handbook.adoc, nist-architecture-mapping.adoc, user-testing-guide.adoc,
api/canopy-{tanf,medicaid}.adoc, the audit section Plugin.toml, and
tests/e2e/specs/audit-rail.spec.ts.
Precedents: pass/idle-skip/rotation/config-domain from the #1207 drainer
(chain_staging/drainer.rs:481-513, config.rs:110-229); delayed first tick
(detection.rs:28-41); EphemeralSchema::new_for_<service> one-schema-one-pool
constructors (db.rs:304-324) — FTI verify tests build TWO;
SET ROLE grant probes; xtask perf home (xtask/src/cmd/perf.rs); NO secrecy crate
in the workspace — pool URLs use the workspace settings pattern (String field + Debug
redaction).
D1 — the shared verify engine (canopy_common::chain_verify, MR-1)
Home crates/canopy-common/src/chain_verify.rs (sibling of chain_append.rs;
canopy-chain stays pure). Reused later by #1208/#1247 (purge-boundary verification) and
#1280 (closed-epoch verification).
/// Which projections a family reads — ALWAYS view-mediated (C8 + rename
/// stability): audit → audit_hashed_preimage_v / _archive_v; fti →
/// fti_hashed_preimage_v / _archive_v (which gain `id` in MR-1).
/// EleGrant → UnsupportedFamily.
pub fn preimage_views(family: ChainFamily) -> Result<(&'static str, &'static str), ChainVerifyError>;
/// One row as read from a preimage view — the FULL projection. The payload is
/// fetched as TEXT (`canonical_event_payload::text` in the SELECT list — the
/// views themselves are unchanged) so the D14 number fence sees the stored
/// rendering BEFORE any serde normalization; it is parsed into `Value` only
/// after the fence passes.
pub struct PreimageRow {
pub chain_seq: i64,
pub previous_hash: EventHash,
pub event_hash: EventHash,
pub chain_source: ChainSource,
pub formula: u16,
pub side: RowSide, // Live | Archive — the union tag (D1a)
pub payload_text: String, // fenced (D14), then parsed
pub payload: serde_json::Value, // STRUCTURAL-VALUE: canonical JSON, re-hashed after validation
pub derived: DerivedColumns, // per-family enum: every typed hashed column the view exposes
}
pub enum RowSide { Live, Archive }
pub enum DerivedColumns { Audit(AuditDerived), Fti(FtiDerived) }
// AuditDerived: event_id, event_type, event_timestamp, user_id, user_role, action,
// resource_type, resource_id, source_service, household_id, metadata (fetched as
// TEXT + fenced like the payload, then Value — STRUCTURAL-VALUE).
// FtiDerived: id, accessed_by, accessed_at, purpose_code, data_elements_accessed,
// originating_system, action, resource_type, resource_id.
#[derive(Debug, thiserror::Error)]
pub enum ChainVerifyError {
Database(#[from] sqlx::Error), // availability — retry/backoff
Topology(canopy_common::chain_append::ChainAppendError), // fetch_topology seam (explicit wrap, not From)
Reject(#[from] VerifyReject), // integrity — latch material, NEVER retried
UnsupportedFamily { family: ChainFamily },
Fenced, // lost the token/CAS — stop silently
}
/// Decode/canonicalization/number-fence failures on PERSISTED data construct
/// VerifyReject::MalformedRow — an integrity finding. canopy_chain::ChainError
/// from the environment stays an error path; there is NO blanket Chain→retry arm.
/// The C6 rejection taxonomy = the pinned chain_incidents.kind vocabulary,
/// now ALSO a CHECK constraint (D2) — Rust enum, SQL CHECK, and the kind→loop
/// mapping are test-pinned against each other:
/// hash_mismatch, linkage_break, noncontiguous_seq, duplicate_seq,
/// formula_version, payload_set_violation, derived_column_mismatch,
/// routing_mismatch, source_mismatch, id_mismatch, genesis_mismatch,
/// terminal_mismatch, head_regression, rows_beyond_head, missing_head,
/// unexpected_head, missing_shard_rows, target_hash_mismatch,
/// manifest_divergence, manifest_metadata_mismatch, boundary_unavailable,
/// malformed_row.
pub enum VerifyReject { /* one variant per kind, positional evidence fields */ }
pub struct VerifyTarget { pub seq: i64, pub hash: EventHash }
pub struct VerifyCursor { pub seq: i64, pub hash: EventHash }
D1a — the batch walk is ONE statement (X6). verify_batch fetches the window
cursor.seq+1 ..= target.seq as a SINGLE UNION ALL statement over the archive and
live projections with a side tag, ORDER BY chain_seq, LIMIT batch.rows — one MVCC
snapshot. A mid-batch archive move can therefore never make rows vanish between two
queries, and a row present on BOTH sides arrives as adjacent equal seqs and latches
duplicate_seq. The byte budget is applied while CONSUMING the fetched rows: the batch
always admits at least ONE row regardless of budget (no livelock); any single row whose
payload exceeds the 4 MiB hard ceiling (double the 2 MiB ingress body cap — nothing
legitimate can approach it) latches malformed_row; the config floor for
CHAIN_VERIFY_BATCH_BYTES is 4 MiB so the ceiling always fits (D11).
/// Verify ONE bounded batch (single-statement archive ∪ live — D1a).
/// Per row, in order:
/// 0. the D14 raw number fence over payload_text (and audit metadata text)
/// — violation → MalformedRow;
/// 1. contiguity (seq == prev+1; <= prev → DuplicateSeq — including the
/// cross-side duplicate case, which arrives adjacent under D1a);
/// 2. linkage (row.previous_hash == running hash);
/// 3. formula == 2;
/// 4. closed key/type set for the family (payload_set_violation — the same
/// key sets the append fns enforce, mirrored in Rust, test-pinned);
/// 5. derived-column consistency: every DerivedColumns field == its payload
/// field (derived_column_mismatch);
/// 6. routing placement: shard_for(routing_id, shard_count) == shard
/// (routing_mismatch; routing id = payload event_id for audit, payload id
/// for fti — the Amendment 7 binding);
/// 7. source: row.chain_source == topo.source (source_mismatch);
/// 8. side sanity: side == Archive requires seq <= archived_through(at
/// fetch); side == Live requires seq > archived_through — a wrong-side
/// row latches (the archived-orphan / wrong-side class, X6);
/// 9. (fti) row.id == payload id (id_mismatch);
/// 10. envelope rebuild + event_hash recompute == stored (hash_mismatch).
/// If the batch reaches target.seq the running hash MUST equal target.hash
/// (TargetHashMismatch). Returns the advanced cursor + rows/bytes consumed.
pub async fn verify_batch(
chain: &PgPool, topo: &ActiveTopology, family: ChainFamily, shard: ShardId,
cursor: VerifyCursor, target: &VerifyTarget, batch: BatchBudget,
) -> Result<BatchOutcome, ChainVerifyError>;
pub struct BatchBudget { pub rows: u32, pub bytes: u64 }
pub struct BatchOutcome { pub cursor: VerifyCursor, pub reached_target: bool,
pub rows: u32, pub bytes: u64 }
/// Boundary hash for a cursor at `seq` (reads the CHAIN pool): seq == 0 →
/// empty_head_hash(instance, family, epoch, shard, None); else the event_hash
/// of the row at `seq` from whichever side holds it — absent entirely
/// (post-purge, #1208-era) → Reject::BoundaryUnavailable.
pub async fn cursor_hash_at(chain: &PgPool, topo: &ActiveTopology,
family: ChainFamily, shard: ShardId, seq: i64) -> Result<EventHash, ChainVerifyError>;
/// One-statement heads snapshot (single MVCC snapshot).
pub struct ShardHead { pub shard: ShardId, pub last_seq: i64, pub last_hash: EventHash,
pub archived_through: i64 }
pub async fn capture_heads(chain: &PgPool, topo: &ActiveTopology, family: ChainFamily)
-> Result<Vec<ShardHead>, ChainVerifyError>;
/// Structural census (family-lease-serialized, its own cadence — one replica
/// per family per cadence, X2; WHOLE retained history, X6):
/// - head set complete over 0..shard_count, none beyond (MissingHead /
/// UnexpectedHead);
/// - genesis arm: an empty head (last_seq == 0) must carry last_hash ==
/// empty_head_hash (genesis_mismatch);
/// - terminal row exists with event_hash == last_hash, or last_seq <=
/// archived_through with the archive row matching (TerminalMismatch);
/// - rows beyond the CURRENT head — ONE statement per side unioned with the
/// head subselect (single snapshot);
/// - identity-filtered whole-range count == span, computed over archive ∪ live
/// in ONE union statement (MissingShardRows); rows carrying a foreign
/// instance/epoch/shard on EITHER side are detected (the archived-orphan
/// MUST now covers archived rows); on any mismatch the census RE-READS the
/// head/boundary and re-counts ONCE before latching (an archive move between
/// cadences is legal interleaving, never a torn-read breach).
pub async fn structural_census(chain: &PgPool, topo: &ActiveTopology,
family: ChainFamily, heads: &[ShardHead]) -> Result<(), ChainVerifyError>;
/// Manifest binding — order matters: (1) fetch the latest CONFIRMED anchor row
/// (security pool); (2) decode manifest_bytes (from_canonical_bytes) and
/// compare the DECODED fields against the anchor ROW's caller-supplied
/// metadata — the SEVEN row columns: anchor_seq, kind, epoch,
/// previous_anchor_hash, manifest_hash, instance, family
/// (manifest_metadata_mismatch; chain_anchors has NO source column — the
/// decoded source is hash-bound inside manifest_bytes and checked in step 3);
/// (3) compare identity vs topology (instance/family/source/epoch/shard_count)
/// and vs the COMPILED formula/routing constants; (4) THEN capture heads and
/// check per-tip prefix consistency (tip.last_seq <= head.last_seq AND the
/// chain hash AT tip.last_seq equals tip.last_hash). A tip BEYOND the head
/// triggers ONE re-fetch + re-capture before latching manifest_divergence.
/// On success returns the checked anchor id — stamped on the FAMILY checkpoint
/// row as trusted_manifest_ref via the guarded advance (X5: the family row is
/// the ONE trusted-manifest source; per-shard stamping is REMOVED).
pub async fn manifest_check(security: &PgPool, chain: &PgPool, topo: &ActiveTopology,
family: ChainFamily) -> Result<Option<CheckedManifest>, ChainVerifyError>;
pub struct CheckedManifest { pub anchor_id: Uuid, pub anchor_seq: i64,
pub tips: Vec<ShardTipSeq> }
The token-fenced state client — VerifierDb (X1/X10). Every security-pool statement
that promises SET LOCAL timeouts or transactional atomicity takes an explicit
connection/transaction handle, never a bare &PgPool:
/// Owns the security pool + the verifier's own timeout knobs. Each call opens
/// a transaction, applies SET LOCAL lock_timeout/statement_timeout (the
/// router.rs:27-53 idiom, D11 knobs), runs the statement(s), commits. The
/// job-finalize path exposes the transaction so the run write and the job
/// finalize commit ATOMICALLY (D8).
pub struct VerifierDb { /* security: PgPool, timeouts: VerifyTimeouts */ }
pub struct CheckpointKey { pub instance: ChainInstanceId, pub family: ChainFamily,
pub epoch: Epoch, pub shard: ShardId, pub loop_kind: LoopKind }
pub enum LoopKind { Tail, Scrub, Family } // "tail" | "scrub" | "family"
/// The FAMILY key is (instance, family, epoch, shard 0, Family) — PK-distinct
/// from shard 0's tail/scrub rows by loop_kind (D2/D3).
pub struct Lease { pub token: Uuid, pub fence: i64 }
/// Reads go through the token-free view chain_checkpoints_verify_v (X1: the
/// verify role's raw SELECT on the table is REVOKED — the ONLY way to hold a
/// token is to have minted it via acquire).
pub struct CheckpointState { pub fence: i64, pub cursor: Option<VerifyCursor>,
pub target: Option<VerifyTarget>, pub trusted_manifest_ref: Option<Uuid>,
pub lease_expires_at: Option<DateTime<Utc>>, pub updated_at: DateTime<Utc>,
pub cycle_started_at: Option<DateTime<Utc>>,
pub cycle_completed_at: Option<DateTime<Utc>> }
impl VerifierDb {
pub async fn read_checkpoint(&self, key: &CheckpointKey)
-> Result<Option<CheckpointState>, ChainVerifyError>;
/// chain_checkpoint_acquire: Some(Lease), or None = not acquired (an
/// unexpired lease — ANY owner's, including our own crashed predecessor
/// (X1: expiry-only takeover, owner display-only) — or a lost cycle-start
/// CAS). Duration is a BOUNDED number of seconds; expiry is computed inside
/// the locked fn (no caller clocks). `init` is REQUIRED for a first-ever
/// tail/scrub acquire; family acquires carry neither init nor cycle.
/// Scrub cycle-starts pass `cycle` — a cursor-CAS (X5): the target+reset
/// write applies ONLY if the row's verified_through_seq still equals
/// expected_seq, so a delayed worker's stale cycle-start LOSES under the
/// row lock.
pub async fn acquire(&self, key: &CheckpointKey, owner: &str, lease_secs: u32,
init: Option<&VerifyCursor>, cycle: Option<&ScrubCycleStart>)
-> Result<Option<Lease>, ChainVerifyError>;
/// chain_checkpoint_advance: exact-token-bound, existing-row-only,
/// cursor-monotonic. The token stays valid PAST expiry until a takeover
/// mints a successor (X1: correctness never reads the clock; a finding from
/// a long batch is never lost to a clock). Health stamps are PROVEN in-fn
/// (X5): on SCRUB keys cycle_complete only when the presented cursor EQUALS
/// the stored target (seq AND hash); on the FAMILY key cycle_complete is
/// the census-cadence stamp — token-gated observability, deliberately NOT
/// target-proven (census completion is a read-side check with no relational
/// witness, and no D6 status rule reads the stamp); manifest_ref only on
/// the FAMILY key and only when it references a CONFIRMED chain_anchors row
/// of the same (instance, family). cursor is None exactly for family-key
/// advances (lease refresh / census stamp — the family row has no cursor).
pub async fn advance(&self, key: &CheckpointKey, lease: &Lease,
cursor: Option<&VerifyCursor>, lease_secs: u32, cycle_complete: bool,
manifest_ref: Option<Uuid>) -> Result<(), ChainVerifyError>; // Err(Fenced) on token mismatch
}
pub struct ScrubCycleStart { pub expected_seq: i64, pub target: VerifyTarget,
pub boundary: VerifyCursor }
/// Guarded writes (X2): run recording and incident latching validate a token
/// inside the DB. Runs validate the FAMILY token (run rows are family-scoped
/// with an optional shard for error attribution — the any-shard-token hole is
/// gone). Latches validate the token of the lease named by the finding's
/// scope: shard-scoped findings present the shard lease of the DETECTED loop;
/// family-scoped findings (missing/unexpected heads, genesis mismatch,
/// manifest divergence, boundary-unavailable at init) present the FAMILY
/// token with shard NULL.
pub struct VerifyRun { pub loop_kind: LoopKind, pub shard: Option<ShardId>,
pub started_at: DateTime<Utc>, pub finished_at: DateTime<Utc>,
pub outcome: RunOutcome, pub error: Option<String>, pub rows_verified: i64,
pub mode: RunMode, pub job_id: Option<Uuid> }
pub enum RunOutcome { Ok, Rejected, Error } // 'ok' | 'rejected' | 'error'
pub enum RunMode { Scheduled, Manual } // manual runs NEVER feed status (X4)
impl VerifierDb {
pub async fn record_run(&self, family_lease: &Lease, key_family: &CheckpointKey,
run: &VerifyRun) -> Result<Uuid, ChainVerifyError>;
/// Same statement inside a caller-owned transaction — the job-finalize
/// atomicity seam (D8).
pub async fn record_run_in(&self, tx: &mut PgConnection, family_lease: &Lease,
key_family: &CheckpointKey, run: &VerifyRun) -> Result<Uuid, ChainVerifyError>;
pub async fn latch_incident(&self, lease: &Lease, key: &CheckpointKey,
detected: LoopKind, reject: &VerifyReject) -> Result<Option<Uuid>, ChainVerifyError>;
/// Halt-gate read — via the evidence-free chain_incidents_verify_v (X3).
pub async fn family_has_unresolved_incident(&self, instance: ChainInstanceId,
family: ChainFamily) -> Result<Option<Uuid>, ChainVerifyError>;
}
canopy-chain additives (MR-1): AnchorManifest::from_canonical_bytes(&[u8]) — parse,
newtype-decode, validate(), re-encode == input; PartialEq derives on
AnchorManifest/ShardTip; the D14 canon::raw_number_fence + its KAT
vectors. New shared types: ChainPosition and ChainBacklog live in
canopy-contracts-security (wire types — D8); CheckedManifest/ShardTipSeq/
Lease/BatchBudget live in chain_verify (engine types).
test-lib additive (MR-1): append_chained_rows(pool, &SeededChain, shard, n, salt,
dist: PayloadDist) — drives the REAL append_rows_on_shard in ≤500 batches;
PayloadDist parameterizes the size distribution (X10 — the D13 mixes are arguments,
not a hardcode).
D2 — the hardening migration: token confidentiality, the family loop, proofs, guarded writers
The MR-1 migration (20261010000000_chain_verification_hardening.sql, security DB)
reshapes the dormant substrate — pre-1.0, zero compat, empty tables (no backfills).
Everything follows the substrate’s own idioms: DO/EXECUTE format() with
SET search_path, owner-transfer to canopy_chain_owner_security, PUBLIC EXECUTE
revoked, per-role grants re-established, touch crates/canopy-test-lib/src/db.rs.
Tables:
-- Checkpoints: the family loop kind, the token, the cycle stamps, the CHECK matrix.
ALTER TABLE chain_verification_checkpoints
DROP CONSTRAINT chain_verification_checkpoints_loop_kind_check; -- the :296 inline CHECK
ALTER TABLE chain_verification_checkpoints
ADD COLUMN lease_token UUID,
ADD COLUMN cycle_started_at TIMESTAMPTZ,
ADD COLUMN cycle_completed_at TIMESTAMPTZ,
ADD CONSTRAINT ..._loop_kind CHECK (loop_kind IN ('tail','scrub','family')),
-- The FAMILY row is (instance, family, epoch, shard 0, 'family') — PK-distinct
-- from shard 0's tail/scrub rows. It carries NO cursor and NO target; it
-- carries the family lease, trusted_manifest_ref (X5: the ONE source), and
-- cycle_completed_at as the census-cadence stamp.
ADD CONSTRAINT ..._family_shape CHECK (loop_kind <> 'family'
OR (shard_id = 0 AND target_seq IS NULL AND verified_through_seq = 0
AND verified_through_hash IS NULL)),
ADD CONSTRAINT ..._cursor_pair CHECK (loop_kind = 'family'
OR verified_through_hash IS NOT NULL),
ADD CONSTRAINT ..._hash_len CHECK ((verified_through_hash IS NULL
OR octet_length(verified_through_hash) = 32)
AND (target_hash IS NULL OR octet_length(target_hash) = 32)),
ADD CONSTRAINT ..._nonneg CHECK (verified_through_seq >= 0 AND fence >= 0
AND (target_seq IS NULL OR target_seq >= 0)),
ADD CONSTRAINT ..._target_pair CHECK ((target_seq IS NULL) = (target_hash IS NULL)),
ADD CONSTRAINT ..._target_ge_cursor CHECK (target_seq IS NULL
OR target_seq >= verified_through_seq),
ADD CONSTRAINT ..._lease_trio CHECK ((lease_owner IS NULL) = (lease_expires_at IS NULL)
AND (lease_owner IS NULL) = (lease_token IS NULL)),
ADD CONSTRAINT ..._token_v7 CHECK (lease_token IS NULL
OR uuid_extract_version(lease_token) = 7);
-- Runs: family loop kind, per-shard attribution, the manual/scheduled split.
ALTER TABLE chain_verification_runs
DROP CONSTRAINT chain_verification_runs_loop_kind_check; -- the :314 inline CHECK
ALTER TABLE chain_verification_runs
ADD COLUMN shard_id SMALLINT, -- NULL = family-scoped / whole-family coverage
ADD COLUMN mode TEXT NOT NULL CHECK (mode IN ('scheduled','manual')),
ADD COLUMN job_id UUID, -- FK added in MR-2 (the jobs table lands there)
ADD CONSTRAINT ..._loop_kind CHECK (loop_kind IN ('tail','scrub','family')),
ADD CONSTRAINT ..._outcome CHECK (outcome IN ('ok','rejected','error')),
ADD CONSTRAINT ..._finish_pair CHECK ((finished_at IS NULL) = (outcome IS NULL)),
ADD CONSTRAINT ..._finish_order CHECK (finished_at IS NULL OR finished_at >= started_at),
ADD CONSTRAINT ..._rows_nonneg CHECK (rows_verified >= 0);
CREATE INDEX chain_verification_runs_latest_idx ON chain_verification_runs
(chain_instance_id, chain_family, chain_epoch, loop_kind, shard_id,
finished_at DESC NULLS LAST)
WHERE mode = 'scheduled'; -- per-SCOPE latest-run reads (X5); status reads
-- scheduled runs ONLY, and manual runs are
-- reached by id via their job — the partial
-- predicate matches the D6 view exactly
-- Incidents: the detecting loop is STORED, dedup is NULL-safe, the vocabulary
-- is closed in-schema, evidence is bounded.
ALTER TABLE chain_incidents
ADD COLUMN detected_loop_kind TEXT NOT NULL
CHECK (detected_loop_kind IN ('tail','scrub','family')),
ADD CONSTRAINT ..._family_scope CHECK ((detected_loop_kind = 'family')
= (shard_id IS NULL)),
ADD CONSTRAINT ..._kind_vocab CHECK (kind IN ('hash_mismatch','linkage_break',
'noncontiguous_seq','duplicate_seq','formula_version','payload_set_violation',
'derived_column_mismatch','routing_mismatch','source_mismatch','id_mismatch',
'genesis_mismatch','terminal_mismatch','head_regression','rows_beyond_head',
'missing_head','unexpected_head','missing_shard_rows','target_hash_mismatch',
'manifest_divergence','manifest_metadata_mismatch','boundary_unavailable',
'malformed_row')),
ADD CONSTRAINT ..._evidence_shape CHECK (jsonb_typeof(evidence) = 'object'
AND pg_column_size(evidence) <= 16384);
CREATE UNIQUE INDEX chain_incidents_latched_uq ON chain_incidents
(chain_instance_id, chain_family, chain_epoch, shard_id, kind, detected_loop_kind)
NULLS NOT DISTINCT WHERE state = 'latched'; -- family-scoped (NULL epoch/shard) dedups (X3)
The confidentiality split (X1/X3) — the verify role loses every raw C6 table read; what it needs comes back through owner-transferred views:
CREATE VIEW chain_checkpoints_verify_v AS SELECT
chain_instance_id, chain_family, chain_epoch, shard_id, loop_kind,
target_seq, target_hash, verified_through_seq, verified_through_hash,
trusted_manifest_ref, lease_owner, lease_expires_at, fence, updated_at,
cycle_started_at, cycle_completed_at
FROM chain_verification_checkpoints; -- EVERYTHING except lease_token
CREATE VIEW chain_incidents_verify_v AS SELECT
id, chain_instance_id, chain_family, chain_epoch, shard_id,
detected_at, kind, detected_loop_kind, state
FROM chain_incidents; -- no evidence, no resolution text
-- (owner-transfer both to canopy_chain_owner_security, the substrate $own$ pattern)
REVOKE SELECT ON chain_verification_checkpoints, chain_verification_runs,
chain_incidents FROM canopy_chain_verify; -- closes the :863 token/evidence leak
REVOKE INSERT ON chain_verification_runs FROM canopy_chain_verify; -- the :864 open write
GRANT SELECT ON chain_checkpoints_verify_v, chain_incidents_verify_v
TO canopy_chain_verify;
GRANT SELECT ON chain_incidents TO canopy_chain_incident_admin; -- the D7 evidence reader
The reshaped checkpoint functions — durations, proofs, the family arm:
CREATE FUNCTION chain_checkpoint_acquire(
p_instance uuid, p_family text, p_epoch int, p_shard smallint, p_loop text,
p_owner text, p_lease_secs int,
p_init_seq bigint DEFAULT NULL, p_init_hash bytea DEFAULT NULL,
p_cycle_expected_seq bigint DEFAULT NULL,
p_cycle_target_seq bigint DEFAULT NULL, p_cycle_target_hash bytea DEFAULT NULL,
p_cycle_reset_seq bigint DEFAULT NULL, p_cycle_reset_hash bytea DEFAULT NULL
) RETURNS TABLE (lease_token uuid, fence bigint)
-- Domain: p_lease_secs BETWEEN 1 AND 600 (RAISE — X1: bounded DURATION; expiry
-- := now() + make_interval(secs => p_lease_secs) computed HERE, no caller
-- clocks, no lock-wait erosion).
-- Family arm: p_loop = 'family' requires p_shard = 0 and every init/cycle
-- param NULL (RAISE otherwise); the INSERT arm creates the family row with a
-- NULL cursor hash (the _family_shape/_cursor_pair CHECKs).
-- Tail/scrub arm: INSERT ... ON CONFLICT DO NOTHING with the REQUIRED init
-- cursor, then re-SELECT FOR UPDATE (concurrent first-acquires serialize;
-- absent row + NULL init → RAISE).
-- Takeover is EXPIRY-ONLY for everyone (X1): an unexpired lease — ANY owner,
-- including the caller's own text — returns EMPTY. Owner is display-only; a
-- restarted process waits out its own lease (bounded by p_lease_secs).
-- Cycle-start (scrub keys; all five p_cycle_* or none — RAISE on a partial
-- set) is a cursor-CAS (X5): it applies ONLY if verified_through_seq =
-- p_cycle_expected_seq; a stale worker's delayed cycle-start returns EMPTY
-- under the row lock and resets NOTHING. On success it writes target +
-- cursor reset + cycle_started_at := now() in the same UPDATE
-- (cycle_completed_at is NOT cleared — it stamps the LAST completed cycle).
-- Grant: fence := fence + 1, lease_token := uuidv7(); returns the minted
-- token + new fence — the ONLY fence-raising, row-creating path.
CREATE FUNCTION chain_checkpoint_advance(
p_instance uuid, p_family text, p_epoch int, p_shard smallint, p_loop text,
p_lease_token uuid, p_lease_secs int,
p_verified_seq bigint DEFAULT NULL, p_verified_hash bytea DEFAULT NULL,
p_cycle_complete boolean DEFAULT false, p_manifest_ref uuid DEFAULT NULL
) RETURNS boolean
-- Existing-row-only (absent → false). Token must EQUAL the stored lease_token
-- (else false — the fence); expiry is NEVER read (X1: a token stays valid
-- past expiry until a takeover mints a successor — a long batch's finding
-- is never lost to a clock; p_lease_secs, domain-checked 1..=600, renews
-- expiry := now() + interval on success).
-- Family arm: p_loop = 'family' requires p_verified_seq/hash NULL and permits
-- p_manifest_ref / p_cycle_complete (the census-cadence stamp). Tail/scrub
-- arms REQUIRE the cursor pair and REJECT p_manifest_ref (RAISE — X5: the
-- family row is the ONE trusted-manifest home).
-- Cursor monotonic: p_verified_seq < verified_through_seq → false; equal seq
-- with a DIFFERENT hash → RAISE 'chain-v2: checkpoint cursor hash
-- divergence' (corruption, never silent).
-- PROOFS (X5 — the scrub-coverage and manifest stamps are never caller
-- assertions; the family census stamp is the stated exception: token-gated
-- observability with no relational witness, and no D6 status rule reads it):
-- p_cycle_complete on a scrub key RAISEs unless p_verified_seq = target_seq
-- AND p_verified_hash = target_hash (checked against the STORED target
-- in-fn); on success stamps cycle_completed_at := now().
-- p_manifest_ref RAISEs unless it references a chain_anchors row with
-- notarization_state = 'confirmed' AND matching (chain_instance_id,
-- chain_family) — a relational check, not trust.
-- Never touches fence/target otherwise; stamps updated_at.
-- Guarded writers:
CREATE FUNCTION chain_run_record(
p_family_token uuid,
p_instance uuid, p_family text, p_epoch int, p_loop text, p_shard smallint,
p_started timestamptz, p_finished timestamptz, p_outcome text, p_error text,
p_rows bigint, p_mode text, p_job_id uuid
) RETURNS uuid
-- Validates p_family_token = the FAMILY row's lease_token for (p_instance,
-- p_family, p_epoch) — X2: runs are recorded ONLY by the family-lease holder
-- (returns NULL when fenced; the engine maps NULL → Fenced). INSERTs the
-- immutable run row (id := uuidv7()). p_job_id is stored from MR-1 (the FK
-- and the mode⇔job CHECK arrive with the MR-2 jobs table — same-signature
-- CREATE OR REPLACE there, explicitly NOT an overload).
CREATE FUNCTION chain_incident_latch(
p_token uuid,
p_instance uuid, p_family text, p_epoch int, p_shard smallint,
p_detected_loop text, p_kind text, p_evidence jsonb
) RETURNS uuid
-- Scope rule (X3): p_detected_loop = 'family' ⟺ p_shard IS NULL (RAISE
-- otherwise). Token validation matches the finding's scope: family-scoped →
-- the FAMILY row's token; shard-scoped → the (p_shard, p_detected_loop) row's
-- token. p_kind is CHECK-constrained by the table; p_evidence bounded by the
-- _evidence_shape CHECK. INSERT ... ON CONFLICT (chain_instance_id,
-- chain_family, chain_epoch, shard_id, kind, detected_loop_kind)
-- WHERE state = 'latched' DO NOTHING; returns the existing latched id on
-- conflict (idempotent, race-free). Fenced token → NULL.
CREATE FUNCTION chain_incident_resolve(p_id uuid, p_reason text,
p_evidence_ref text, p_revalidation_run uuid) RETURNS void
-- actor := session_user (recorded by the fn, never caller-supplied);
-- p_evidence_ref REQUIRED. The revalidation run must be: outcome = 'ok',
-- mode = 'manual' (X4: scheduled runs NEVER resolve), matching
-- instance + family, finished_at > incident.detected_at, loop_kind =
-- incident.detected_loop_kind (STORED, never inferred — X3), and
-- scope-covering: run.shard_id IS NULL (whole-family coverage) OR
-- run.shard_id = incident.shard_id.
The anchor role/arm split — the transition fn splits into NAMED arms (the substrate’s
single fn :688-718 lets one credential fabricate and confirm):
CREATE ROLE canopy_chain_anchor_emitter NOLOGIN; -- reconciled fail-closed like the rest
CREATE FUNCTION chain_anchor_transition_emit(
p_id uuid, p_to text, p_jws text, p_kid text,
p_external_ref text, p_external_version text) RETURNS void
-- The :701-707 edge matrix MINUS confirmation: pending→{submitted,failed},
-- submitted→failed, failed→submitted; p_to = 'confirmed' → RAISE. COALESCE
-- updates of jws/kid/external_ref/external_version as today.
CREATE FUNCTION chain_anchor_transition_confirm(p_id uuid) RETURNS void
-- submitted→confirmed ONLY; stamps verifier_confirmed_at := now(); touches
-- NO caller-supplied columns (the emitter arms own those).
REVOKE EXECUTE ON FUNCTION chain_anchor_append(uuid,text,int,bigint,text,bytea,bytea,bytea)
FROM canopy_chain_verify;
GRANT EXECUTE ON FUNCTION chain_anchor_append(uuid,text,int,bigint,text,bytea,bytea,bytea)
TO canopy_chain_anchor_emitter;
GRANT EXECUTE ON FUNCTION chain_anchor_transition_emit(uuid,text,text,text,text,text)
TO canopy_chain_anchor_emitter;
GRANT EXECUTE ON FUNCTION chain_anchor_transition_confirm(uuid)
TO canopy_chain_verify;
Archive attestation indexes (LIKE copies no indexes):
CREATE UNIQUE INDEX audit_events_archive_v2_event_id_uq
ON audit_events_archive_v2 (((canonical_event_payload->>'event_id')));
-- and in the tanf/medicaid MR-1 migrations (alongside the view change):
-- CREATE UNIQUE INDEX fti_audit_log_archive_v2_id_uq
-- ON fti_audit_log_archive_v2 (id);
Old signatures DROPPED, never overloaded (CREATE OR REPLACE with a changed signature
creates a NEW OVERLOAD in PostgreSQL — the substrate originals and their grants would
survive as the exact open write paths this migration closes):
DROP FUNCTION chain_checkpoint_advance(uuid, text, int, smallint, text,
bigint, text, timestamptz, bigint, bytea);
DROP FUNCTION chain_incident_latch(uuid, text, int, smallint, text, jsonb);
DROP FUNCTION chain_incident_resolve(uuid, text, text, text, uuid);
DROP FUNCTION chain_anchor_transition(uuid, text, text, text, text, text);
-- (chain_checkpoint_acquire and chain_run_record are new — nothing to drop.
-- The grant-matrix test asserts to_regprocedure(<each old signature>) IS NULL
-- post-migration.)
The protocol (Amendments 8–9): acquire → Lease{token, fence}; every write presents
the token; ANY newer acquire mints a new token and the old holder’s next write returns
Fenced (stop, re-enter next pass). Expiry-only takeover means the worst-case pause
after a crash is one lease duration (default 30s). The lease ≥ 3× statement-timeout
rule is a LIVENESS heuristic, not a correctness proof (X1): correctness rides the
token; a batch outliving its lease merely gets fenced on its next write and the work is
discarded. Every acquire/advance statement runs under the VerifierDb SET LOCAL
timeouts (D11 knobs — never the drainer’s 30s default).
D3 — the family pass (X2): one lease orders everything
THREE independent tasks, one per family target (audit/security, fti/tanf,
fti/medicaid) — a broken program DB back-pressures only its own family. Each task, per
pass (verifier_pass(pools, target, cfg, pass_counter) — testable), in THIS order:
-
fetch_topology(chain_pool, family)— read-only, supplies the INSTANCE (nothing below has a key without it);Missing/NotActive→ idle-skip debounced (dormant pre-cutover). Topology-fetch is the only step preceding the family lease; its failure is logged and surfaces as read-time staleness (documented — no run row exists to record, and none is needed). -
Acquire the FAMILY lease — key
(instance, family, epoch, shard 0, 'family').None→ another replica owns this family’s pass; skip the pass entirely. The family lease structurally serializes the census AND the manifest check (one replica per family per cadence — the O(range) census multiplier is gone) and makes the family holder the ONLY servicer of that family’s manual jobs (D8 — no cross-replica job lottery). Scale-out is BY FAMILY (three families today); shard tokens remain the write-fencing belt underneath (Amendment 9 records this honestly). -
Halt gate:
family_has_unresolved_incident—Some(id)and no manual job in hand → skip with a debounced warn. DB-derived, restart-safe, replica-global. RE-CHECKED between batches — a latch propagates within one batch bound; the residual in-flight window (one batch) is documented. -
Manual jobs FIRST (X2): claim via
chain_job_claim(target-scoped); a claimed job’s segments run under the same global budget before scheduled work (D8). -
Manifest check, then census (both under the family token):
manifest_checkper D1 — a passing check’s anchor id is stamped on the FAMILY row (advance(family_key, …, manifest_ref)); the census runs on its own cadence (CHAIN_CENSUS_INTERVAL_SECS), stamping the family row’scycle_completed_atas the cadence marker. Family-scoped findings latch under the family token with shard NULL. -
Per shard, rotating start: acquire the tail key (first-ever acquire passes
init = boundary cursor— genesis or archive-boundary hash viacursor_hash_at; aboundary_unavailablehere latches under the FAMILY token — the shard lease does not exist yet); compare(cursor.seq, cursor.hash)against the captured head: seq beyond head, or EQUAL seq with a different hash → latchhead_regression. Verify bounded batches within the pass’s GLOBAL budget, advancing after every batch. Then the scrub key likewise (D4). -
Every shard visit counts against the budget (X10) — including zero-work lease-refresh advances (the success stamp that clears
errorat read time). The family lease is refreshed by a family-row advance between shard segments — bounded chatter, and the refresh IS the family activity stamp. -
On
Reject: token-guarded latch + run record (outcome 'rejected', the shard attributed); the halt gate stops the family. OnDatabase/Topologyerrors: run record (family token, shard attributed where known), capped backoff (250ms→5s), never park, never green. Runs are recorded for passes that did work, erred, or rejected — never for idle ticks; scheduled runs carrymode = 'scheduled'.
Tick CHAIN_VERIFY_TICK_MS (default 1000); first tick delayed
CHAIN_VERIFY_FIRST_TICK_DELAY_SECS (default 60) — sleep, then interval (the
detection.rs:28-41 delayed-start precedent, generalized to a configurable delay);
tests use paused tokio time (tokio/test-util dev-dep).
D4 — the historical scrub loop (full retained history)
Same task, loop_kind = 'scrub', CHAIN_SCRUB_BATCHES_PER_PASS (default 1) within the
same global pass budget:
-
Coverage: the cycle walks
0 → targetacross ARCHIVE ∪ LIVE in single-statement batches (D1a) — archived rows re-verify under the full check set (a mutated archived payload with an intact stored hash IS detected). A boundary move mid-cycle is benign: the union is by seq range, side-agnostic, and each batch is one snapshot. -
Cycle start (X5 — a cursor-CAS, race-free under the row lock): when the row shows
target IS NULL∨cursor.seq == target.seq, the engine acquires withcycle = ScrubCycleStart{ expected_seq: <the cursor seq it just read>, target: tail cursor snapshot, boundary: (0, genesis hash) }. The fn applies the reset ONLY ifverified_through_seqstill equalsexpected_seq— a delayed worker whose read predates another’s cycle activity LOSES (empty result, no reset). Restart mid-cycle acquires plain and resumes from the stored cursor.cycle_started_atis stamped by the CAS — the first-cycle staleness input (D6). -
Cycle end: reaching the target advances with
cycle_complete = true— and the fn PROVES it (cursor == stored target, seq AND hash) before stampingcycle_completed_at(X5). A >0-rows cycle records a run. -
A scrub
Rejectlatches + halts identically. Empty-range cycles complete immediately.
D5 — census + the manifest trust chain
-
Census: family-lease-serialized, cadenced, whole-history (archive ∪ live — X6), retry-once-before-latch on count/boundary mismatches, explicit genesis arm.
-
Manifest:
manifest_checkper D1 — anchor-first ordering, decoded-bytes vs the SEVEN row-metadata columns vs topology vs chain, one re-fetch on a tip-beyond-head, then latch. The checked anchor id lands on the FAMILY row only (X5) — one unambiguoustrusted_manifest_ref; status and attestation read ONLY the family ref — a newly confirmed but never-verifier-checked anchor authorizes nothing. -
The #1278-facing confirmer contract (pinned for the anchor-authority child): the verifier confirms an anchor (
chain_anchor_transition_confirm) ONLY aftermanifest_checkpasses on THAT anchor — bytes decoded, row metadata matched, topology matched, per-tip prefix consistency proven. #1278’s emitter submits (chain_anchor_transition_emit) and never confirms; the verifier confirms and never emits. The C5 divergence consumer arm rides the same check. -
No confirmed anchor →
manifest: absent→ status caps atverifying. Manifest age (now - checked anchor’s created_at) overCHAIN_MANIFEST_MAX_AGE_SECS→stale(default generous, 7 days; #1278 tightens). Post-cutover the confirmed genesis anchor exists (#1279 gate) —healthyis reachable at reopen; real-event ATTESTATION additionally needs the first periodic tip. -
Residual (Amendment 8): a rewrite strictly newer than the trusted manifest is invisible until the next anchor — #1278’s cadence bounds that window.
D6 — the C6 status machine + backlog inputs + the _app projection
Inputs (assembled per read):
| Input | Source | Availability |
|---|---|---|
Tail/scrub coverage + freshness per shard (incl. |
|
always (0 rows pre-first-acquire) |
Latest SCHEDULED run per (instance, family, epoch, loop, shard-scope) |
|
always |
Unresolved incidents (position + kind + detected loop — never evidence) |
|
always |
Trusted manifest (id, seq, age) |
|
always |
Head seqs for lag |
audit: |
always |
Staging backlog (audit only) |
in-process |
|
Inbox parks + DLQ depth (audit only) |
|
unsampled BLOCKS |
Derivation (derive_status — the domain enum and the pure derivation live in
canopy_common::chain_verify::status; the wire enum maps from it explicitly in
canopy-security, bijection test-pinned — X9). Precedence:
-
breached— any unresolved incident for (instance, family). Latched; only D7 clears. The FIRST unresolved incident’s position populatesbreached_position(D8 — the typed row-banner source). -
error— PER SCOPE (X5): the latest scheduled run for a (loop, shard) hasoutcome = 'error'AND is newer than THAT shard’s checkpointupdated_atsuccess stamp; family-scoped errors (shard NULL) compare against the FAMILY row’supdated_at. A shard’s error clears only against its own zero-work advance — cross-shard masking is structurally gone. Also: any backlog input reportingsampler_untrustworthy. -
stale— read-time: tailupdated_atage >tail_max_ageon any shard; tail lag >tail_max_lagon any shard; scrubcycle_completed_atage >scrub_max_ageon any shard — falling back tocycle_started_atwhen no cycle has EVER completed (X5: a first cycle that never completes goesstale, exactly as the design claims); trusted-manifest age >manifest_max_age(from the FAMILY row). -
unknown— no checkpoint coverage (fresh/dormant) or topology absent/not-active. -
verifying— coverage advancing but incomplete/unanchored: initial catch-up, any backlog count nonzero, any backlog input NOT YET SAMPLED (with the nonzero clause, the exact complement of rule 6’s "SAMPLED and zero"), ormanifest: absent. -
healthy— every shard’s tail at head within freshness + lag; every shard’s scrub cycle fresh; every backlog input SAMPLED and zero; trusted manifest present, checked, fresh; no unresolved incident; no live per-scope error.
HTTP mapping (Amendment 8, unchanged): healthy/verifying → 200;
unknown/stale/error/breached → 503.
There is no separate census-staleness rule by construction: the census rides the family pass, so a family whose census stops has either a stopped pass (tail rows go stale) or a latched census finding (breached).
The _app projection migration (MR-2,
20261015000000_chain_verification_projections.sql + test-lib touch): the four views
above (owner-transferred to canopy_chain_owner_security — a view runs with its owner’s
rights, and only that role reads the C6/anchor bases) + the D8 jobs table + its
guarded fns + the runs job_id FK + the mode ⇔ job CHECK
(ALTER TABLE chain_verification_runs ADD CONSTRAINT …_manual_job CHECK
mode = 'manual') = (job_id IS NOT NULL, ADD CONSTRAINT …_job_fk FOREIGN KEY
(job_id) REFERENCES chain_verify_jobs(id)) + the same-signature
CREATE OR REPLACE FUNCTION chain_run_record adding the job validation (identical
signature — explicitly NOT an overload). chain_incident_resolve is untouched here:
its mode/loop/scope checks are MR-1’s, and MR-2’s mode ⇔ job CHECK makes
manual ⇒ job-linked structurally.
D7 — incidents: latch, evidence, resolution
-
Latch kinds =
VerifyReject::kind_str()= the D2 CHECK vocabulary = the kind→loop mapping — all three test-pinned against each other. Evidence = positions
expected/got hex only, bounded by the_evidence_shapeCHECK. -
The reader split is real (X3): the background verifier reads incidents ONLY through the evidence-free
chain_incidents_verify_v(enough for the halt gate and status);_appreadschain_incidents_app_v(position/kind/detected-loop/state); evidence and resolution text are readable ONLY bycanopy_chain_incident_admin. -
Resolution (runbook in
security-operations.adoc): inspect evidence under the incident-admin credential → ticket → trigger the manual revalidation job (bypasses the halt gate; runs the incident’s DETECTED loop — stored at latch, never inferred) → confirmoutcome 'ok'→chain_incident_resolve(id, reason, ticket, run_id)— the fn recordsactor = session_userand enforces manual mode + detected-loop match
scope coverage + timing in SQL (D2). A clean scheduled pass never clears.
D8 — the unified /v1/security/chain/* surface + durable jobs
Namespace (X9, Amendment 9 — pre-1.0 CHANGELOG Changed, zero compat): ONE chain
namespace replaces the historical scatter. Deleted: GET /v1/security/verify-chain
POST /v1/security/fti/chain-verify (MR-2), GET /v1/security/fti/chain-status
(MR-3 — the interim handler survives untouched until then, serving the #1245 posture).
Path-count assertion (api/mod.rs:781, currently 15): MR-2 → 17 (−2 +4); MR-3 →
16 (−1). Two staged OpenAPI snapshots, one per MR; devstack refresh BEFORE push
(#1267).
-
GET /v1/security/chain/status?family={audit|fti}[&service=]→ChainStatusResponse(200/503 per D6). MR-2:family=fti→ 503verifier_unavailable(routing live, target dormant until MR-3). -
POST /v1/security/chain/verify{family, service?, loop?, incident_id?}→ 202ChainVerifyJobAccepted— or 409verification_in_progress(an active job already exists for the target; body carries itsjob_id), 503 at the queue cap / verifier disabled / family unconfigured (no phantom queue — X4). -
GET /v1/security/chain/verify-jobs/{id}→ChainVerifyJobStatus. Requester-scoped: service callers see only their own jobs; admin sees all; unknown/foreign id → 404. -
GET /v1/security/chain/attest?event_id={uuid}&family={audit|fti}[&service=]→ChainAttestation. Position resolution is VIEW-mediated and INDEXED, across archive ∪ live: audit by the payload-expression predicate, FTI by theidcolumn (both sides explicitly indexed — D2). Attested iff position found AND the position’s(instance, epoch)MATCH the active topology (theChainPositioncarries both — an old-epoch/old-instance row can never attest, X9) ANDseq ⇐ tail.verified_through(shard)ANDseq ⇐ TRUSTED manifest tip.last_seq(shard)AND state ∉ {unknown, stale, error, breached}. Pool discipline: position resolution runs on the VERIFY pools for BOTH families (the preimage views are verify-role-only —_appnever gains them, C8); the status/coverage/trusted-tip inputs ride the_appprojections. Dormant → 503.
Auth unchanged (the existing service-or-admin arms).
The request-error matrix (closed vocabulary; each cell is a named test):
| Condition | Status | error code |
|---|---|---|
|
400 |
|
|
400 |
|
|
400 |
|
|
400 |
|
|
400 |
|
|
400 |
|
|
404 |
|
active job exists for the target (verify) |
409 |
|
queue at |
503 |
|
Replaced DTOs (crates/canopy-contracts-security/src/chain.rs — full serde-attributed
shapes; every closed vocabulary is a real enum, wire strings test-pinned):
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum ChainFamilyWire { Audit, Fti }
#[derive(Serialize, Deserialize)]
pub enum ChainServiceWire {
#[serde(rename = "canopy-tanf")] CanopyTanf,
#[serde(rename = "canopy-medicaid")] CanopyMedicaid }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum ChainVerifyState { Unknown, Verifying, Healthy, Stale, Error, Breached }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum ChainLoopWire { Tail, Scrub, Family }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "kebab-case")]
pub enum RequestedLoopWire { Tail, Scrub, FamilyFull } // matches the DB vocabulary
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum JobStateWire { Queued, Running, Done, Error }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum JobErrorCode { CoverageIncomplete, VerifierError, IntegrityRejected, Crashed }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum AttestRefusal { UnknownEvent, NewerThanCheckpoint, BeyondTrustedManifest,
StateNotAttestable, VerifierUnavailable, ForeignTopology }
// StatusReason: the closed reasons vocabulary (one variant per firing D6 input),
// #[serde(rename_all = "snake_case")], enumerated + test-pinned in the source.
pub struct ChainPosition { pub instance: Uuid, pub family: ChainFamilyWire,
pub epoch: i32, pub shard: u16, pub seq: i64 } // instance+family: X9 —
// attestation and the row banner filter to the EXACT topology
pub struct ChainBacklog { pub staged: i64, pub parked: i64,
pub inbox_parked: i64, pub dlq_depth: i64 }
pub struct ShardCoverage {
pub shard_id: u16,
pub tail_verified_through: i64,
pub head_seq: i64,
pub tail_lag: i64,
pub tail_updated_at: Option<DateTime<Utc>>,
pub scrub_verified_through: i64,
pub scrub_target: Option<i64>,
pub scrub_updated_at: Option<DateTime<Utc>>,
pub scrub_cycle_started_at: Option<DateTime<Utc>>, // first-cycle staleness input
pub scrub_cycle_completed_at: Option<DateTime<Utc>>, // coverage stamp
}
pub struct ChainStatusResponse {
pub family: ChainFamilyWire,
pub service: Option<ChainServiceWire>,
pub state: ChainVerifyState,
pub reasons: Vec<StatusReason>,
pub epoch: Option<i32>,
pub shards: Vec<ShardCoverage>,
pub backlog: Option<ChainBacklog>, // None for fti — typed applicability,
// never zeroed i64s (X5)
pub trusted_manifest: Option<TrustedManifestSummary>,
pub incident_id: Option<Uuid>,
pub breached_position: Option<ChainPosition>, // the typed row-banner source (X9)
}
pub struct TrustedManifestSummary { pub anchor_id: Uuid, pub anchor_seq: i64,
pub age_secs: i64 }
pub struct ChainVerifyJobAccepted { pub job_id: Uuid, pub poll_url: String }
pub struct ChainVerifyJobStatus { pub job_id: Uuid, pub state: JobStateWire,
pub requested_loop: RequestedLoopWire, pub attempts: i32,
pub run: Option<ChainRunSummary>, pub error_code: Option<JobErrorCode> }
#[derive(Serialize, Deserialize)] #[serde(rename_all = "snake_case")]
pub enum RunOutcomeWire { Ok, Rejected, Error } // = the D2 CHECK vocabulary
pub struct ChainRunSummary { pub run_id: Uuid, pub loop_kind: ChainLoopWire,
pub outcome: Option<RunOutcomeWire>, pub rows_verified: i64,
pub started_at: DateTime<Utc>, pub finished_at: Option<DateTime<Utc>> }
pub struct ChainAttestation { pub attested: bool, pub event_id: Uuid,
pub position: Option<ChainPosition>, pub verified_through: Option<i64>,
pub trusted_anchor_seq: Option<i64>, pub state: ChainVerifyState,
pub reason: Option<AttestRefusal> }
Retirement scope (pre-1.0): ChainVerificationResponse dies in MR-2 — producer + all
four consumer sites (three canopy-web files + the test-lib client) migrate in the same
MR. ChainStatusInterim/InterimChainState remain ONLY for run_archive (1208’s
surface); ChainStatusInterim.last_verification + FtiChainVerification(+Row) die in
MR-3 with the reader narrowed per D10 (FtiChainStatusParams survives). No
[deprecated] anywhere; doc-comments carry the retirement notes; the stale references
at api/mod.rs:578 and events.rs:191 are updated (J5).
The durable job store (X4 — MR-2, in the projections migration):
CREATE TABLE chain_verify_jobs (
id UUID PRIMARY KEY CHECK (uuid_extract_version(id) = 7),
chain_family TEXT NOT NULL CHECK (chain_family IN ('audit','fti')),
fti_source TEXT CHECK (fti_source IN ('canopy-tanf','canopy-medicaid')),
CHECK ((chain_family = 'fti') = (fti_source IS NOT NULL)),
requested_loop TEXT NOT NULL CHECK (requested_loop IN ('tail','scrub','family-full')),
incident_id UUID REFERENCES chain_incidents(id), -- revalidation jobs
requested_by TEXT NOT NULL CHECK (requested_by <> ''),
requested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
state TEXT NOT NULL DEFAULT 'queued'
CHECK (state IN ('queued','running','done','error')),
-- The durable work definition, written ONCE at first claim (X4): reclaim
-- resumes the SAME instance/epoch/target vector — a crash never re-captures
-- weaker targets. Shape: {"shards":[{"shard":N,"seq":S,"hash":"<hex>"},…]}.
chain_instance_id UUID, chain_epoch INT, captured_targets JSONB,
CHECK ((captured_targets IS NULL) = (chain_instance_id IS NULL)
AND (captured_targets IS NULL) = (chain_epoch IS NULL)),
-- The claim is DB-minted-token-based (the checkpoint ABA fix, symmetric):
claim_owner TEXT, claim_token UUID, claim_expires_at TIMESTAMPTZ,
heartbeat_at TIMESTAMPTZ,
attempts INT NOT NULL DEFAULT 0 CHECK (attempts >= 0),
run_id UUID REFERENCES chain_verification_runs(id),
error_code TEXT CHECK (error_code IN
('coverage_incomplete','verifier_error','integrity_rejected','crashed')),
finished_at TIMESTAMPTZ,
-- The full state matrix (X4):
CHECK ((state = 'running') = (claim_owner IS NOT NULL)),
CHECK ((claim_owner IS NULL) = (claim_token IS NULL)
AND (claim_owner IS NULL) = (claim_expires_at IS NULL)
AND (claim_owner IS NULL) = (heartbeat_at IS NULL)),
CHECK ((state IN ('done','error')) = (finished_at IS NOT NULL)),
CHECK (state <> 'done' OR run_id IS NOT NULL),
CHECK ((state = 'error') = (error_code IS NOT NULL)),
CHECK (state <> 'queued' OR (run_id IS NULL AND captured_targets IS NULL
AND attempts = 0))
);
CREATE INDEX chain_verify_jobs_queued_idx ON chain_verify_jobs (requested_at)
WHERE state = 'queued';
CREATE INDEX chain_verify_jobs_reclaim_idx ON chain_verify_jobs (claim_expires_at)
WHERE state = 'running'; -- reclaim scan (X4)
CREATE INDEX chain_verify_jobs_reap_idx ON chain_verify_jobs (finished_at)
WHERE state IN ('done','error'); -- reap scan (X4)
CREATE UNIQUE INDEX chain_verify_jobs_active_target_uq ON chain_verify_jobs
(chain_family, fti_source) NULLS NOT DISTINCT
WHERE state IN ('queued','running'); -- ONE active job per target (X4)
Guarded job fns (ALL with the full SECURITY DEFINER discipline — owner-transfer,
SET search_path, REVOKE PUBLIC — plus PUBLIC probes in the grant test; NO raw DML
grants to anyone, _app gets SELECT for polling):
chain_job_enqueue(p_family text, p_source text, p_loop text, p_requested_by text,
p_incident uuid, p_max_queued int) RETURNS TABLE (job_id uuid, created boolean)
-- An existing active job for the target → (its id, false) — the handler maps
-- to 409 verification_in_progress (idempotent: no duplicate work, the caller
-- learns the in-flight id). Global queued count >= p_max_queued → RAISE
-- (handler → 503). p_incident must reference an existing incident. [_app]
chain_job_claim(p_family text, p_source text, p_worker text, p_claim_secs int)
RETURNS SETOF chain_verify_jobs
-- TARGET-SCOPED (X4): each family task claims ONLY its own (family, source).
-- FOR UPDATE SKIP LOCKED, oldest queued OR running-with-expired-claim
-- (reclaim). Mints claim_token := uuidv7(), attempts := attempts + 1,
-- claim_expires_at := now() + p_claim_secs (domain 1..=600 in-fn).
-- captured_targets is PRESERVED on reclaim. [verify]
chain_job_capture(p_id uuid, p_claim_token uuid, p_instance uuid, p_epoch int,
p_targets jsonb) RETURNS void
-- Token-validated; writes the work definition ONLY when captured_targets IS
-- NULL (first capture wins; a reclaimer sees non-NULL and RESUMES). [verify]
chain_job_heartbeat(p_id uuid, p_claim_token uuid, p_claim_secs int) RETURNS void
-- Token-validated; extends claim_expires_at, stamps heartbeat_at. [verify]
chain_job_finalize(p_id uuid, p_claim_token uuid, p_state text, p_run uuid,
p_error_code text) RETURNS void
-- Token-validated; state ∈ {done, error}; nulls the claim fields; called in
-- the SAME transaction as chain_run_record (VerifierDb::record_run_in).
-- [verify]
chain_job_reap(p_older_than_days int) RETURNS bigint
-- Floor enforced IN the fn: p_older_than_days >= 7 (RAISE below).
-- Terminal-only; rows whose incident_id references a still-latched incident
-- are EXEMPT until resolved. [verify]
Job execution (crash-safe, family-lease-integrated — X2/X4): the family-lease holder
claims its target’s jobs FIRST each pass. First claim: capture instance/epoch/targets
from live heads (chain_job_capture); reclaim: RESUME the captured vector. The job
protocol iterates passes under the global budget until every shard of the family
reaches the CAPTURED target for the requested loop (family-full = tail + scrub
census + manifest; it records one run per loop, the job’s run_id pointing at the
family run). Heartbeat per batch (CHAIN_JOB_HEARTBEAT_SECS); claim lease
CHAIN_JOB_CLAIM_SECS (validated ≥ 3× heartbeat). Manual runs carry mode = 'manual'
+ job_id — they NEVER feed the status machine in either direction (X4). Any shard
unreached at claim expiry → the claim lapses and a reclaimer resumes;
attempts > CHAIN_JOB_MAX_ATTEMPTS → error/crashed. Coverage is all-or-nothing:
finalize done only with full self-verified coverage of the captured targets, else
error/coverage_incomplete. Finalize + run write are ONE transaction. Reap after
7 days.
D9 — canopy-web: the typed-terminal-503 client, badge, banner, citation
-
The client path comes first (X9): the generic canopy-web client maps every non-2xx to
Errand RETRIES 503s with discarded bodies (clients.rs:190) — under the D6 mapping,breachedIS a 503 with a body, so the badge could never render it. MR-2 adds an endpoint-specific terminal fetch (clients.rs): no retry on 503, and theChainStatusResponsebody is deserialized on BOTH 200 and 503. -
project_chain_status(stream.rs:148-183) rewritten overChainStatusResponse:healthy→ verified pill;verifying→ neutral "Verification in progress" (+ backlog note);breached→ broken pill + incident id; everything else / transport error → "Unable to verify chain". The dead per-rowbroken_at == event.idmatcher (stream.rs:192-216) is REMOVED; the row banner derives from the TYPEDbreached_positionfield (instance/family-filtered — X9), not a string note. -
case_detail/sections/audit.rs:105-145+api/audit_log.rs:191-217: same swap; the audit sectionPlugin.tomlis touched alongside. -
Citation PDF (
audit_log.rs:472-545): the flow is pinned as fetch-by-PK → attest-by-event_id(the handler resolves the cited row by its primary key, then attests the row’sevent_id— rename-stable, archive-safe).attested: falseor transport error → 502 "citation not issued" (posture preserved); PDF inputs =verified_through/trusted_anchor_seq. The TEMPLATE is in the file list:rulesets/georgia/notices/audit/citation.typ:111renders the verification block — updated with the new fields + its notice render tests. -
Test-lib client
verify_chain()retargets to the unified status endpoint
ChainStatusResponse; web api-docs snapshot refreshed.
CLI parity (ADR-007 — X9): the CLI hardcodes the deleted path
(tools/canopy-cli/src/cmd/security.rs:66). MR-2: canopy security chain-status
--family [--service], canopy security chain-verify --family [--service] [--loop]
[--wait] (trigger + poll), canopy security chain-attest --event-id --family
[--service] — thin wrappers over the unified endpoints; MR-3 activates the FTI
arguments. The ADR-007 parity inventory is updated in the same MRs.
D10 — the FTI arm (#1206) + the preserved legacy breach (X8)
-
Three verify-pool URLs (D11) as
canopy_security_verify(LOGIN at #1279), pool size 2. Startup validation is parse-only; connectivity is per-family and LAZY (X10): a present-but-unreachable FTI DB degrades THAT family toerror/staleat read time and retries on cadence — one program outage never blocks the process or the other families (the fail-fast contradiction resolved in favor of isolation). The (family, source) topology-slot match runs at each family’s FIRST successful connect — a swapped tanf/medicaid URL pair still fails that family loudly, in its own lane. Dormant = unconfigured. -
Per-family tasks share the engine + config; checkpoints/runs/incidents key by instance in the security DB; head/lag reads on the CHAIN pool’s
chain_heads. -
FTI status/attest arms resolve positions via the verify pools on the
idcolumn (both sides indexed — D2). -
The legacy latched breach STAYS VISIBLE (X8 — reversing the v5 cut, which was wrong): #1245’s handler + seeded-breach test (
api/mod.rs:676,security_test.rs:683) deliberately keep a latched v1 FTI breach visible — "a breach is never silently swallowed" is a SAFETY invariant. MR-3’s FTI status arm ORs the legacy latched row intobreached(reasonlegacy_breach_latched) until #1279 drops the table. The store reader is NARROWED to the breach bit (legacy_fti_breach_latched(pool, service) → boolreplacing theFtiChainVerificationDTO chain); the seeded-breach test SURVIVES, retargeted to the unified endpoint. -
Archive-aware in-vivo proof (both families): seed → append → superuser-move a prefix
advancearchived_through→ scrub verifies THROUGH the archived range and the live suffix; mutate an archived payload (coherently: payload + derived columns together) → scrub latcheshash_mismatch; mutate INCOHERENTLY (payload only) →derived_column_mismatch(the honest two-arm split, X10); purge the boundary row →boundary_unavailable. -
No writer-path changes in tanf/medicaid services; their DATABASES gain exactly the MR-1 migration (views +
id+ archive index + grants).
D11 — configuration
Env var (CANOPY_SECURITY__…) |
Default | Domain (out-of-domain = STARTUP ERROR) |
|---|---|---|
|
|
bool |
|
|
10..=60000 |
|
|
0..=3600 |
|
|
1..=10000 |
|
|
4 MiB..=256 MiB — the floor equals the D1a hard row ceiling, so one max-size row ALWAYS fits the budget |
|
|
1..=256 — GLOBAL shard VISITS per pass, every visit counted, zero-work refreshes included (X10) |
|
|
1..=64, validated < |
|
|
30..=86400 |
|
|
5..=300 — shard AND family leases; also the worst-case crash-recovery pause (expiry-only takeover, X1) |
|
|
1000..=10000 — |
|
|
100..=10000 |
|
|
10..=600, validated ≥ 3 × |
|
|
1..=200 |
|
|
1..=10 |
|
|
1..=64 — passed into |
|
|
≥30 |
|
|
≥1 |
|
|
≥300 — against |
|
|
≥300 (#1278 tightens) |
|
— |
String, Debug-redacted (workspace settings pattern); presence + URL-parse validated at boot when enabled; CONNECTIVITY is lazy per family (X10) |
ChainVerifyConfig::from_config (the ChainDrainConfig pattern), unit-tested; every
relationship above validated at startup, never clamped.
D12 — dormancy + the #1279 handshake
-
Flag off (default): no pools, no tasks, no job servicing; the unified status endpoint serves
unknown→ 503 (the #1245 posture by status code); POST verify → 503verifier_unavailable—chain_job_enqueueis never called for an unconfigured target (no phantom queue, X4). TheChainStatusInputssampler runs always. -
#1279 flips: LOGIN carriers (verify + incident-admin + the emitter for #1278) + three URLs + flag. Sequence after genesis-anchor confirmation:
unknown → verifying → healthywith zero backlog; badge in vivo; ATTESTATION of real events additionally waits for #1278’s first confirmed periodic tip. -
v1 retirement at #1279: the
fti_chain_verificationstable drop + the X8 breach-bit reader + thelegacy_breach_latchedreason retire together (tracked on #1279’s list at 0a).
D13 — perf evidence (numeric gates; cargo xtask perf chain-verify)
Audit dimension runs in MR-2 (before Closes #1205); MR-3 adds the FTI dimension.
Release build, devstack PG, seeded via append_chained_rows through the real append
path with the PayloadDist parameter (X10).
-
Corpus: 2,000,000 rows/family; payload mix 1 KiB typical / 64 KiB p99;
shard_count ∈ {2, 8}; 30s warmup, 60s window, 2 reps, mean ± spread. -
Catch-up: tail from genesis — PASS ≥ 5,000 rows/s per task at the 1 KiB mix, and ABSOLUTE process RSS < 256 MiB throughout the run (X10 — an absolute gate, not a delta a bloated baseline can hide).
-
Sustained: producer at 300/s concurrent — PASS: lag slope ≤ 0 over the window.
-
Independence: status GET and attest GET sampled at 0.5M/1M/2M rows — PASS: p95 < 50 ms at each size AND max/min p95 ratio < 1.5 (flat).
-
Plans: EXPLAIN assertions — the attestation lookups (live + archive, both families), the D1a union batch fetch, and
chain_job_claimuse index scans at 2M rows. -
The D14 fence KATs + proptest gate in-battery (correctness, not perf).
D14 — the JSON number fence: #1285 CLOSED, not observed (X7)
The hole (canon.rs:48-70): validate_number range-checks integers but passes
floats, and RFC 8785 renders every number THROUGH f64 — so distinct stored values that
collide in f64 (the canonical example: 2^53 vs 2^53 + 1) canonicalize to identical
bytes. Hash recomputation alone cannot detect that mutation class. A KAT can only
demonstrate the collision; the fence CLOSES it.
Deliberate non-dependency: serde_json’s `arbitrary_precision feature is NOT
enabled — cargo feature unification would change `serde_json::Number’s parse behavior
for EVERY workspace crate. The fence is a self-contained scanner in canopy-chain.
The fence (canopy_chain::canon::raw_number_fence(text: &str) →
Result<(), NumberFenceViolation>, ~80 lines + tests): walk the JSON text once,
skipping string literals (escape-aware); for each number token t (RFC 8259 grammar):
-
Parse
tas f64 (overflow to ±inf → violation). -
Render the f64’s SHORTEST round-trip decimal (std
Display— shortest-digit guaranteed). -
Compare AT VALUE LEVEL: normalize both
tand the rendering to (sign, digit string, decimal exponent) and require equality. Accept iffdecimal_value(t) == decimal_value(shortest_repr(f64(t))).
Value-level (not text-level) comparison is the load-bearing choice: PostgreSQL stores
jsonb numbers as exact numeric and REFORMATS on output (1e30 prints as its
31-digit expansion), so text equality would false-positive on every legitimate
rendering difference, while VALUE equality accepts exactly the values JCS can represent
losslessly and rejects every f64-collision mutation: a stored 2^53 mutated to 2^53 + 1
prints a token whose decimal value differs from its f64 round-trip
(9007199254740993 ≠ 9007199254740992) → violation; notation differences
(1e30 vs the expansion vs 2.3e1 vs 23) normalize equal → pass. -0 normalizes to
0 → pass (JCS renders it "0").
Where it runs:
-
Verify-side (MR-1): step 0 of the D1a per-row checks, over the fetched
canonical_event_payload::text(and the auditmetadatacolumn text) — violation →malformed_rowlatch with the offending token + offset as evidence. This catches every historical or mutated row. -
Intake-side (MR-2): the same fence over the RAW BODY BYTES in
ingest_audit_event(api/mod.rs:236-265; violation → the 422 I-JSON class extends by one code) and over the staging consumer’s delivery bytes before staging (chain_staging/stage.rs; violation → park as poison, its oversize-integer filter precedent) — no new collision-class value can enter.
Pinning: KAT vectors (2^53 − 1 / 2^53 / 2^53 + 1, exponent vs expansion forms, 0.1,
-0, 1e400 overflow, value-preserving reformattings) + a proptest invariant — for
every f64, raw_number_fence accepts its shortest repr AND every tested
value-preserving reformatting; for every integer magnitude above 2^53 not exactly
representable, the literal is rejected. #1285’s adjudication resolves to this fence
(0a notes it; closed at MR-2 merge).
Scope — explicitly OUT
-
Anchor emission, the external authority, submission/confirmation loops, cadence/SLO → #1278 (WITH the emitter role/arm split + the D5 confirmer contract).
-
Epoch closure/rollover + closed-epoch verification → #1280.
-
Purge-boundary manifests + the archive/purge movers → #1208/#1247 (verification of RETAINED rows, archived included, is IN scope here).
-
The incident-resolution UI → the 0a-filed follow-up issue (0a’s FIRST action).
-
ele_grant→ #1248. Cutover execution + table drops → #1279.
Verification (test inventory — both external reviews' cases included)
canopy-chain unit (MR-1): raw_number_fence_kats (the D14 vector set);
raw_number_fence_value_roundtrip_props (proptest — the D14 invariant);
manifest_decode_round_trips_and_rejects_noncanonical (+ KATs).
canopy-common unit: derive_status_precedence_table (exhaustive: every backlog input
singly, unsampled-blocks-healthy, manifest-absent/aged, per-shard error isolation —
shard A’s error never masks/clears via shard B (X5), first-cycle-never-completed goes
stale (X5), dead-worker read-time stale); reject_kinds_are_closed_and_match_sql
(Rust enum = the D2 CHECK vocabulary = the kind→loop mapping);
verify_batch_math_props (proptest); payload_closed_sets_match_the_append_fns;
dto_serde_shapes (every D8 enum’s wire strings pinned; domain→wire status bijection).
canopy-security tests/chain_verify_engine_test.rs (EphemeralSchema, audit family):
confidentiality + fencing —
token_column_unreadable_by_verify_role (SET ROLE probe on the view + the table — X1);
concurrent_cold_acquire_admits_exactly_one;
acquire_requires_init_cursor_on_absent_row;
expiry_only_takeover_same_owner_waits (X1 — no mid-lease self-reacquire);
advance_rejects_unknown_token_and_absent_row;
token_survives_expiry_until_takeover (X1);
fenced_worker_cannot_latch_or_record;
cursor_regression_rejected_equal_seq_hash_divergence_raises;
family_lease_serializes_pass_and_gates_runs (X2 — run record demands the family
token; a shard token cannot record);
family_scoped_finding_latches_under_family_token_pre_shard_lease (X2);
proofs — advance_cycle_complete_requires_target_equality (X5);
advance_manifest_ref_requires_confirmed_anchor_same_identity (X5);
advance_manifest_ref_rejected_on_shard_keys (X5 — family row only);
cycle_start_cas_stale_reset_loses (X5); cycle_start_partial_param_set_raises;
integrity — payload_rewrite_detected; previous_hash_rewrite_detected;
seq_gap_and_duplicate_detected (incl. the cross-side duplicate arriving adjacent —
D1a); derived_column_mutation_detected_per_column; routing_misplacement_detected;
foreign_source_detected; payload_set_violation_detected;
malformed_persisted_row_latches_not_retries (incl. a D14 fence violation);
head_regression_latches; genesis_mismatch_census_arm;
whole_shard_deletion_detected; rows_beyond_head_single_snapshot_no_false_positive;
wrong_side_and_archived_orphan_rows_detected (X6);
coherent_post_capture_rewrite_hits_target_hash_mismatch;
scrub/archive — scrub_detects_mutation_behind_tail;
scrub_covers_archived_rows_and_detects_archived_mutation (coherent arm →
hash_mismatch; incoherent arm → derived_column_mismatch — X10);
scrub_cycle_fixed_target_terminates_under_continuous_writes;
archive_move_during_walk_single_statement_no_tear (X6);
archive_move_concurrent_with_census_no_false_latch (retry arm);
purged_boundary_fails_closed;
oversized_first_row_admitted_then_ceiling_latch (X6 — one row always admitted; a
> 4 MiB row latches malformed_row);
manifest — manifest_row_metadata_mismatch_latches;
newly_confirmed_anchor_mid_check_refetches_not_latches;
attestation_binds_to_trusted_ref_not_newest_anchor;
incidents — latch_is_idempotent_under_race (incl. family-scoped NULL epoch/shard —
the NULLS-NOT-DISTINCT arm, X3);
incident_evidence_unreadable_by_verify_role (X3);
resolve_enforces_manual_mode_detected_loop_scope_and_session_actor (X3/X4 — a
scheduled ok run and a wrong-loop manual run both REJECTED);
resolution_runbook_restores_loops; clean_pass_never_clears_breached.
canopy-tanf + canopy-medicaid tests/chain_v2_verify_test.rs (two-schema topology):
fti_tail_and_scrub_verify_real_carveout_appends;
fti_view_exposes_id_and_id_mismatch_detected; fti_field_coverage_rehash_matches;
fti_two_instances_are_disjoint;
fti_attestation_resolves_on_id_column_indexed (EXPLAIN, live + archive).
canopy-security tests/chain_verifier_host_test.rs:
dormant_flag_off_unknown_503_and_post_refuses; delayed_first_tick_no_boot_walk
(paused time); pass_budget_counts_zero_work_visits_and_rotates (X10);
per_family_task_isolation_broken_pool_stalls_one_family (paused time + a poisoned
pool — the LAZY connectivity arm, X10); halt_gate_rechecked_between_batches;
backlog_inputs_hold_verifying_and_unsampled_blocks_healthy;
status_http_mapping_all_six_states; chain_request_error_matrix (one named
assertion per D8 request-error row); job_lifecycle_queued_running_done;
job_reclaim_preserves_captured_targets (X4 — crash → reclaim resumes the SAME
vector); job_claim_token_fencing (a stale claimant’s heartbeat/finalize rejected);
job_enqueue_409_on_active_target_and_503_at_cap;
job_crash_reclaim_attempts_then_crashed;
job_all_or_nothing_on_lease_contention (coverage_incomplete);
job_poll_requester_scoped; job_finalize_and_run_are_one_tx;
manual_runs_never_feed_status (X4 — a manual ok run clears nothing; a manual error
run degrades nothing);
grants_match_the_matrix (SET ROLE probes: app reads projections + polls jobs,
cannot DML jobs or C6 tables; verify reads ONLY the token-free/evidence-free views,
cannot resolve incidents, append anchors, or confirm-as-emitter; emitter cannot
confirm; PUBLIC executes nothing; to_regprocedure(<each old signature>) IS NULL);
openapi_path_count_17_and_snapshot (MR-2) / …_16… (MR-3);
attestation_refuses_foreign_instance_or_epoch (X9).
canopy-web: six-state pill matrix; typed_terminal_503_renders_breached (X9 — the
generic-client retry path proven bypassed); row banner from breached_position;
citation attest-false → 502 / attest-true → PDF inputs (+ the citation.typ render
tests); dead row-matcher removed.
canopy-security tests/security_test.rs (MR-3): the #1245 seeded-breach test
SURVIVES retargeted — legacy_fti_breach_stays_visible_on_unified_status (X8).
CLI: chain-status/chain-verify/chain-attest argument matrices + a poll-loop test
against a mock (MR-2), FTI args (MR-3).
Gates, all MRs: full battery; SPDX; proptest on the walker + the fence; B3a
STRUCTURAL-VALUE markers + per-MR lock delta; B3b for test imports.
Files touched (by MR)
0b (this commit): this plan + adr-014-fti-audit-hash-chain.adoc (Amendment 9)
parent scale-audit-adr014-chain-v2.adoc (Amendments 8–9 wording)
architecture.adoc + CHANGELOG.adoc. (nav already links this plan.)
MR-1: services/canopy-security/migrations/20261010000000_chain_verification_hardening.sql
(new — the full D2 set); services/canopy-{tanf,medicaid}/migrations/
20261010000000_fti_preimage_id.sql (new — both FTI preimage views gain id
fti_audit_log_archive_v2_id_uq + re-grants); crates/canopy-test-lib/src/db.rs
(touch); crates/canopy-chain/src/{canon.rs (raw_number_fence + NumberFenceViolation),
anchor.rs (from_canonical_bytes, PartialEq)} + KAT vectors under
crates/canopy-chain/tests/vectors/; crates/canopy-common/{src/chain_verify.rs (new),
src/lib.rs, Cargo.toml}; crates/canopy-test-lib/src/chain.rs (append_chained_rows
+ PayloadDist); services/canopy-security/tests/chain_verify_engine_test.rs (new);
services/canopy-{tanf,medicaid}/{tests/chain_v2_verify_test.rs (new), Cargo.toml};
the pre-hardening substrate suites updated to the reshaped surface —
services/canopy-security/tests/chain_v2_substrate_test.rs (the transition-arm
split, the token protocol smoke, token-guarded latch/resolve with the
manual-run + session-actor proofs, the guarded-fns-only write matrix, the 17-object
ownership + dropped-signature ACL matrix) and
services/canopy-{tanf,medicaid}/tests/chain_v2_substrate_test.rs (the pinned
preimage column lists gain the appended id); CHANGELOG.adoc; own Status row.
MR-2: services/canopy-security/migrations/20261015000000_chain_verification_projections.sql
(new: _app views + jobs table + job fns + the runs FK/CHECK + the same-signature
chain_run_record replace) + test-lib touch; services/canopy-security/src/
{chain_verify/{host,status,inputs,jobs}.rs (new), lib.rs, main.rs (queue constant),
api/mod.rs (unified endpoints; `verify-chain GET + FTI verify POST deleted; the D14
intake fence in ingest_audit_event; :578 doc fix), config.rs,
chain_staging/stage.rs (the delivery-bytes fence beside its oversize-integer filter),
chain_staging/health.rs (additive sampler_untrustworthy),
Cargo.toml (tokio test-util dev-dep)}`; crates/canopy-contracts-security/src/
{chain.rs (the D8 DTO set; `ChainVerificationResponse removed), fti.rs (doc-note),
events.rs (:191 doc fix), paths.rs (−2 4)}` + contracts roundtrip tests;
`crates/canopy-test-lib/src/clients/security.rs`; `services/canopy-web/src/{clients.rs
(typed-terminal-503), audit/stream.rs, case_detail/sections/audit.rs,
case_detail/sections/audit/Plugin.toml, api/audit_log.rs}`;
`rulesets/georgia/notices/audit/citation.typ` + notice render tests;
`tools/canopy-cli/src/cmd/security.rs` ( CLI tests); xtask/src/cmd/perf.rs
(chain-verify, audit dimension); OpenAPI snapshots (security + web); docs —
api/canopy-security.adoc, data-models/canopy-security.adoc,
security-operations.adoc (resolution runbook + credential provisioning),
configuration-reference.adoc, rbac-matrix.adoc (:99 names the old FTI paths),
auditor-handbook.adoc, nist-architecture-mapping.adoc, user-testing-guide.adoc,
runbooks/database-backup-restore.adoc (:453 — deleted paths + the old sync
trigger), the ADR-007 parity inventory, CHANGELOG.adoc; tests/e2e/specs/
audit-rail.spec.ts; own Status row.
MR-3: services/canopy-security/src/chain_verify/ (FTI targets/pools) + config.rs
(URLs) + api/mod.rs (FTI arms + the X8 legacy-breach OR;
GET /v1/security/fti/chain-status deleted) + store/{mod,models}.rs
(FtiChainVerification(+Row) + latest_fti_chain_verification deleted; the narrowed
legacy_fti_breach_latched reader added); tests/security_test.rs (the retargeted
seeded-breach test); crates/canopy-contracts-security/src/{fti.rs, paths.rs (−1)};
tools/canopy-cli/src/cmd/security.rs (FTI args); xtask/src/cmd/perf.rs (FTI
dimension); host-test extensions + archive suites; OpenAPI snapshot (16); docs —
api/canopy-{tanf,medicaid}.adoc, api/canopy-security.adoc,
user-testing-guide.adoc (FTI flow), rbac-matrix.adoc (the
GET /v1/security/fti/chain-status row dies here, not in MR-2), CHANGELOG.adoc;
own Status row.
Step 4 (docs close-out): verify auto-closure of #1205 + #1206 + post closing comments; parent Step 4 → Done; plan → Archive + nav; perf cells on #1279.
Sequencing & review-risk notes
0b → MR-1 → MR-2 → MR-3 → Step 4, strictly. Honest scope statement: this plan performs
a substrate-hardening migration set — two reshaped checkpoint fns (with in-fn
proofs), three guarded writer fns, the family loop kind, the detected-loop incident
model, the token/evidence confidentiality split, CHECK matrices, the NULL-safe dedup,
a runs index, archive indexes, the anchor role/arm split, and the FTI view change —
all on the dormant, pre-1.0 chain-owned surface, all following the substrate’s own
idioms, all flagged in §Open decisions. The FTI databases gain a migration (views
index + grants). MR-2 carries the wire-shape replacement + two path deletions + the
unified namespace (pre-1.0 Changed entries; every consumer migrated in-MR — web, CLI,
test-lib, template, docs; the #1245 fail-closed posture preserved by the HTTP mapping
and, for the legacy FTI breach, by the X8 arm in MR-3).
Open decisions for sign-off
-
The substrate-hardening migration set (D2) — ratified for v5, EXTENDED in v7: token confidentiality (the verify role loses raw C6 SELECTs), duration-based DB-computed expiry, expiry-only takeover, in-fn health-stamp proofs, the family loop kind, the detected-loop incident model, the anchor ARM split. Hardening the dormant substrate remains the architecturally correct fix over client discipline.
-
The FAMILY lease (X2) — one replica runs an entire family’s pass; scale-out is BY FAMILY while shard tokens keep write-level fencing. Ratified by the v7 approval; Amendment 9 records it.
-
Backlog →
verifying/ HTTP 200 with enumerated reasons (ratified 2026-07-31). -
Family halt on latch (+ between-batch recheck; the bounded in-flight window is documented).
-
Manual jobs: durable guarded queue, target-scoped, all-or-nothing coverage, 409 on an active target, manual runs quarantined from status (X4).
-
Incident resolution stays runbook-only (guarded fn + session actor + manual-mode
detected-loop enforcement; the UI follow-up is 0a’s first-filed issue). -
Archived-row verification in scope; single-statement union reads (X6).
-
The number fence closes #1285 (D14) — verify-side MR-1, intake-side MR-2; no
arbitrary_precisiondependency. -
Legacy FTI breach visibility preserved until #1279 (X8) — a safety invariant, not compat; the v5 cut is withdrawn.