Plan: ADR-014 chain-v2 — scale-safe, tamper-evident audit hash chains (#1236, epic &73)

On this page
NOTE

Implements the ADR-014 Amendment 5 (chain-v2) contract (C1–C8) across all three audit hash-chain families (audit_events, fti_audit_log, ele_grant_events). The ADR is the contract + invariants + acceptance-suite home; this plan is the rollout + status + issue map — the byte-level design (schemas, KAT vectors, staging transport, DTOs) lives in the child issues. Governed by ADR-004 (data tenancy), ADR-001 (service isolation), and ADR-003 / ADR-011 (ruleset-driven retention).

Status

Step Description Status

0

File prereq issues (#1245/#1246/#1247/#1248) + blocker links; revise implementation children (#1205/#1206/#1207/#1208); commit this plan + nav.

Done (2026-07-27) — #1236 (this MR)

1

P0 containment (fix: MR, #1245, first): constant per-chain FTI lock + concurrency test; fail-closed ALL current full-walk + archive-mutation paths (sync manual endpoints + boot tick); status → unknown.

Done (2026-07-30) — MR !1044, merge 09c10660

2

#1236 ADR-014 Amendment 5 — the chain-v2 contract + invariants + acceptance-suite (C1–C8).

Done (2026-07-27) — this MR

3

substrate child (#1246) — rows/heads/topology/anchors/checkpoint schema, KAT vectors, indexes, restricted roles, empty-genesis install. Byte-level plan: chain-v2 substrate.

Done (2026-07-30) — #1246 closed; four MRs merged (!1045 b40c2ea9, !1046 129a5572, !1047 d2c80eac, !1048 1123a9da); substrate fully landed, dormant until #1279

4

verifier children (#1205 audit, #1206 FTI) — whole-preimage tail + scrub loops over archive ∪ live in single-snapshot reads, family-leased passes over token-confidential fencing, per-shard status with backlog inputs, trusted-manifest binding, the unified /v1/security/chain/* surface with durable async verify jobs + citation attestation, the #1285 number fence — landed COMPLETE and dormant before cutover. Byte-level plan: chain-v2 verifiers; bindings ratified as ADR-014 Amendments 8–9.

Done (2026-08-02) — #1205/#1206 closed; verifier children landed COMPLETE and dormant; byte-level plan archived

5

#1207 append transport — durable staging + structurally-leased per-shard drainer; FTI shard-order batch primitive — landed dormant before cutover (REORDERED per Amendment 6: at cutover direct DML becomes forbidden, so the v2 writers must exist first). Byte-level plan: chain-v2 append transport; bindings ratified as ADR-014 Amendment 7.

Done (2026-07-31) — #1207 closed; !1050 merged (86f04578: shared primitive, classifier, FTI carve-out, dormant tanf/medicaid seams) + !1051 merged (e236648c: staging, one-way park, drainer, health, perf harness); sustained-ingress AC holds in every 300/s cell (slope ≤ 0); all dormant until #1279

6

#1278 external anchor authority (WORM-tier trust model, ADR-014 Amendment 11) — emitter + confirmer + store hardening, cadence/SLO, the behavioral attestation harness + a provisioning-requirements runbook. Production account/bucket/IAM + signing-identity PROVISIONING is a separate deployment concern (out of #1278), provisioned before cutover.

Not started

7

coordinated-downtime cutover (#1279) — quiesce, reset live+archive+integrity, genesis + notarize + activate epochs, identity swap, v1 retirement, reopen — LAST among the go-live steps, gated on Steps 4–6 (go/no-go in #1279).

Not started

8

Post-cutover verifier follow-through — in-vivo activation validation (status reaches healthy on the live genesis chains; badge + attestation checks per #1279’s go/no-go) + the interim-shape retirement rides #1279’s cleanup. (REWRITTEN 2026-08-01: the original "background jobs, citation coverage, status DTOs, singleton scheduling" list predated Amendment 6’s reordering — with #1207 landed first, Step 4 lands the verifier children COMPLETE and dormant, so nothing of that list remains post-cutover.) Epoch closure/rollover executor (#1280) follows.

Not started

9

archive/purge children (#1208 audit + #1247 FTI) — contiguous-prefix, isolation contract, verify→sign→ack→delete, ruleset retention — LAST overall.

Not started

Epic: &73
Issue: #1236 (critical)
Branch: feature/1236-adr014-chain-v2

Context

The 2026-07-25 scale-readiness audit (epic &73) found five ADR-014-rooted findings (three CRITICAL) that make the original chain unable to scale or stay tamper-evident at the 3M-Georgia / 15M-single-deployment horizon: full-walk-from-genesis verification on request/boot paths (C5/C6), a single-writer global append with an unindexed in-lock predecessor lookup (C4/H13), timestamp-boundary archival that breaks the genesis check and can silently drop rows (H8), and an FTI advisory lock that forks on a caller-supplied originating_system. The hash binds no ordinal, so a reorder is invisible.

chain-v2 rebases the whole subsystem on a hash-bound sequence with a durable head, sharded for horizontal write scale, verified incrementally against fixed trusted targets, and anchored to an external notarized authority. Because canopy is pre-1.0 with no production FTI/audit data (devstack + UAT re-seed), the transition is a coordinated-downtime reset to an externally-notarized empty genesis — no legacy formula, dual-format reader, mixed-order verifier, or rebuild-from-history machinery is written.

The chain-v2 contract (summary — full text in the ADR)

The authoritative contract is ADR-014 Amendment 5 C1–C8; a contextless implementer reads that first. In brief:

  • C1 — Identity + position. Canonical tuple (chain_instance_id, chain_family, chain_epoch, shard_id, chain_seq); timestamps are metadata, never position. "FTI" is two instances (tanf, medicaid).

  • C2 — Hash. SHA-256 over a typed RFC-8785/JCS encoding with domain tag "canopy-chain-v2" and hash_formula_version = 2; a per-family field-coverage table; frozen KATs gate the substrate.

  • C3 — Append. Durable chain_heads tip + per-shard SELECT … FOR UPDATE (N per shard; multi-shard locks in shard-id order); direct ingest enqueues into the same staging transport (202 = "durably staged").

  • C4 — Topology + epochs. Fenced installing|active|closing|anchored|closed state machine (as revised by Amendment 6 — installing closes the appendable-before-notarization window); pre-created heads; shard count changes only at an epoch boundary; sharding loses global order (chronology from metadata).

  • C5 — Anchors. External notarized head-manifests on a cadence/SLO (signed, append-only authority, verifier-confirmed) — closes the coherent-privileged-rewrite gap.

  • C6 — Verification. Sharded tail + historical scrub to fixed trusted targets; states unknown|verifying|healthy|stale|error|breached (read-time stale, latched breach); event-specific citation, fail-closed; staging backlog degrades status.

  • C7 — Archive + purge. Contiguous chain_seq prefix per shard; retention = per-jurisdiction ruleset value per family; transactional DELETE … RETURNING → INSERT; verify → sign boundary manifest → ack → delete → advance.

  • C8 — ADR-004 + least-privilege. Per-record Pub 1075 §4 granularity preserved; restricted DB roles; verify reads the hashed-preimage projection (corrects the false "only integrity columns" claim at adr-014:154 and :264).

Issues (finding → issue → ownership)

Finding Issue Owns

P0

#1245

Constant-lock fork hotfix + fail-closed containment (ships first)

substrate

#1246

chain-v2 columns/heads/topology/anchors/checkpoint schema + KAT vectors + indexes + restricted roles + empty-genesis install

C5

#1205 (revised)

audit_events tail + scrub verify + cached badge

C6

#1206 (revised)

FTI tail + scrub verify, archive-aware, delayed tick

C4

#1207 (revised)

append: durable staging + leased per-shard drainer; FTI shard-order batch

H8

#1208 (revised)

audit_events archive/purge (contiguous-prefix)

—

#1247

FTI archive/purge (contiguous-prefix) — #1208 is audit-only

—

#1248

ele_grant_events under chain-v2 (deferred; blocker-linked)

—

#1278

external anchor authority (WORM-tier): impl, cadence/SLO, attestation harness + provisioning runbook — cutover-blocking (production account/bucket/IAM + signing-identity provisioning is a separate deploy concern)

—

#1279

the coordinated-downtime cutover (Step 7) — reset, genesis, notarize, activate, identity swap, v1 retirement, go/no-go gate

—

#1280

epoch closure/rollover — the C4/C5 crash-resumable cross-database state machine

—

#1281

repo-wide serde_jcs RFC 8785 conformance migration (discovered during #1246 D-CANON research)

H13

#1197 (closed)

legacy created_at predecessor indexes — retarget/retire its EXPLAIN tests (predecessor queries are gone)
NOTE
The Finding column uses the 2026-07-25 audit-finding codes (C4/C5/C6/H8/H13) — distinct from the ADR’s contract-clause codes (C1–C8) used in the contract-summary section above and the child-AC revisions below. A single issue can carry both (e.g. #1205 resolves audit-finding C5 by implementing contract-clause C6).

Ground truth (defects resolved — verified file:line)

  • Fork: FTI lock = fti_chain_lock_id(originating_system) over a global chain — crates/canopy-common/src/fti_audit.rs:31-38,334.

  • False genesis / false breach: predecessor read from live only (fti_audit.rs:302-304); archive cut by received_at/accessed_at with no anchor (services/canopy-security/src/store/mod.rs:1164-1199, fti_audit.rs:700-752); INSERT … ON CONFLICT DO NOTHING + blanket DELETE can drop an unproven row (store/mod.rs:1175).

  • OOM / no-lease / stale-green / auto-clear-breach: fetch_all from genesis (fti_audit.rs:629, store/mod.rs:205); every replica verifies, Err→warn;return, latest-row-wins clears a breach (services/canopy-security/src/jobs/fti_chain_verify.rs:60-124, store/mod.rs:427-443).

  • Hash excludes the ordinal: audit uses typed JCS (store/mod.rs:42), FTI uses concat + comma-join (fti_audit.rs:77); neither binds position.

  • Manual sync endpoints still full-walk / archive inline — api/mod.rs:601 (archive), :692 (fti verify sync-then-202), :547 (audit verify), :235 (/audit/ingest sync append).

  • Citation path needs a global verification result today — services/canopy-web/src/api/audit_log.rs:461.

  • False least-privilege claims at adr-014:154 and :264 (verify SELECT * re-hashes access metadata — fti_audit.rs:313,667-678).

  • Retention doc conflict: 3–5 yr (ato-readiness.adoc) vs "indefinite" (auditor-handbook.adoc) — reconciled here to the per-jurisdiction ruleset value.

§0 — P0 containment (#1245, fix: MR, before the redesign)

  • Constant per-chain FTI lock (kill the originating_system fork) + a concurrency test (two distinct originating_system values cannot fork).

  • Fail-closed every current hazard, not just delay it: gate the sync archive endpoint (api/mod.rs:601), the sync FTI/audit full-walk endpoints (:692, :547), and the boot verify tick — no full walk runs on any request/boot path. Status reports unknown (define the minimal interim wire/flag), never stale-green. (Delaying the boot tick alone leaves the sync manual endpoint OOMing.)

Rollout sequence (corrected by ADR-014 Amendment 6, 2026-07-30)

P0 containment (#1245, Done) → this amendment (#1236) → substrate child (#1246: schema + KATs + roles + empty-genesis, dormant) → verifier children (#1205/#1206) landed dormant + #1207 append transport landed dormant + #1278 anchor authority delivered → coordinated-downtime cutover (#1279: quiesce + reset + genesis + notarize + activate + identity swap + reopen) LAST among go-live steps, gated on all of the above → post-cutover verifier follow-through (in-vivo activation validation + interim retirement via #1279’s lists — the children land COMPLETE at Step 4, per Amendments 8–9) → epoch closure/rollover (#1280) → archive/purge children last overall (#1208 + #1247). The original sequence put cutover before #1207 — unworkable: at cutover direct DML becomes forbidden and v1 writers cannot call the v2 functions, so nothing could write. ele_grant_events migration deferred (#1248). #1197 stays closed; retarget/retire its predecessor-query EXPLAIN tests and decide which created_at indexes remain for ordinary queries/retention.

Child-AC revisions (done 2026-07-27; blocker-linked)

  • #1205 — drop "port the FTI pattern" + the timestamp-order index dependency; adopt per-shard tail + scrub + the C6 status model.

  • #1206 — drop "resume from `rows_verified`" + "full-from-genesis never steady state"; require recurring bounded scrubs to a fixed target + a delayed dormant-until-cutover start; make the inline verify truly async.

  • #1207 — per-shard (not global) chain; staging + leased-drainer (not inbox-dedup); N-per-shard; pin batch params (default/min/max N, config key, prefetch-32 relationship, dwell/flush, shutdown/reconnect, poison/DLQ, replay identity, throughput measurement).

  • #1208 — contiguous-prefix (not timestamp chunks); transactional DELETE … RETURNING → INSERT; single-owner scheduled job; endpoint enqueues (not inline). The separate FTI archive/purge twin is filed as #1247.

Files touched (this MR)

File Change

adrs/adr-014-fti-audit-hash-chain.adoc

Append [#amendment-5] — the chain-v2 contract (C1–C8) + invariants + acceptance-suite.

architecture.adoc

ADR-index summary line for ADR-014 notes Amendment 5 (chain-v2).

CHANGELOG.adoc

== Unreleased › === Changed entry (Closes #1236).

ato-readiness.adoc, auditor-handbook.adoc

Retention reconciled to a per-jurisdiction ruleset value (per family, legal-hold aware, federal floor).

plans/scale-audit-adr014-chain-v2.adoc, nav.adoc

This plan + its nav entry (Scale Readiness, epic &73).

Downstream steps (P0, substrate, verifier, cutover, append, archive) ship as their own MRs — each updates its own Status row above; the final MR moves this plan to Archive and fixes nav/xrefs.

Verification

  1. cargo xtask plan-lint + cargo xtask check-docs clean; the Antora build resolves the #amendment-5 xref and every issue reference.

  2. CHANGELOG.adoc == Unreleased carries the Closes #1236 entry; architecture.adoc ADR-index updated.

  3. Fidelity re-read: every C1–C8 clause in the ADR maps to a real table/path or a named-child artifact; no ADR-004 clause is weakened; the acceptance suite is enumerated as the children’s invariant gates.

  4. Docs-only amendment ⇒ no functional battery; the children carry the acceptance tests. docs: MR to main.

Documentation updates

  • ADR-014 Amendment 5; architecture.adoc ADR-index; CHANGELOG.adoc.

  • Retention docs reconciled to a per-jurisdiction ruleset value (ato-readiness.adoc, auditor-handbook.adoc).

  • Prereq issues filed (#1245/#1246/#1247/#1248) + blocker links; implementation children revised (#1205/#1206/#1207/#1208).

  • Plan → Archive on completion (final MR of the sequence).

Open decisions

The stream-level decisions are resolved: reset + coordinated downtime; external notarized head-manifests; retention = per-jurisdiction ruleset value; ADR = contract + invariants, children own byte-level; ELE deferred (#1248). The byte-level design surfaced further contract-scale decisions, resolved and recorded in ADR-014 Amendment 6 + the substrate plan (2026-07-30): the RFC 8785-conforming canonicalizer + I-JSON validation layer, chain_source identity, the installing epoch gate, the genesis anchor kind, the corrected rollout order, the migration/runtime identity split. Still explicitly open, owned by named issues: the final anchor-authority selection (#1278, ratified by a further amendment) and the binding shard counts (#1279’s gate).

Edit this page · default