Plan: ADR-041 — configurable structured logging + jurisdiction-owned field redaction; retire the FTI-special hash chain

On this page

Epic: &74
Contract: ADR-041
Related: &73 (scale readiness)

Status

Unit Description Status

Decision MR

ADR-041 + ADR-014 Amendment 12 + ADR-004 Amendment 2 + this plan + nav/architecture/ato-readiness/auditor-handbook/roadmap/CHANGELOG (#1299). Docs only.

In progress

A — redaction mechanism

Custom stdout FormatEvent/field-visitor + OTEL sanitizer + the jurisdiction policy loader + common policy-root/bootstrap contract + canopy-portal coverage + the executable-service inventory (#1300).

Not started

B — audit-export channel

The unfilterable, versioned, digest+policy-version, commit-coupled channel with dedup/flush/gap-reconciliation/collector-health, keeping the audit DB rows; the production capture+reconciliation conformance gate (#1301).

Not started

§9 detection repair

Fix detection.rs dispatch/seed; define FTI/reporting inputs; keep it in-app + config-driven (#1302).

Not started

D — retention/legal-hold lifecycle

General config-driven audit-table retention/purge + legal hold across all audit families; replace the fixed-503 POST /security/archive (#1303).

Not started

C — retirement (LAST, gated)

chain-v2 teardown + citation redesign + public-contract deprecation + per-family cutover, behind the Child-B conformance gate (#1304).

Not started

Context

This work started as "#1278: build the chain-v2 external anchor" and was reframed, through maintainer review, to its actual shape:

  • The external anchor — and then the hash chain itself — is special-cased tamper-evidence for one log type. Tamper-evidence + retention are general logging-infrastructure properties, satisfied per-deployment by standard logging facilities.

  • Principle: canopy provides the mechanism, the deployment provides the policy. A general configurable structured-logging facility in every service, with per-field redaction. Fully jurisdiction-overridable — the deployment owns all compliance risk (maintainer decision): canopy ships safe defaults + the mechanism; a deployment can override anything (including the FTI/PHI defaults); a malformed policy fails closed (config error), an explicit override to a weaker policy is the deployment’s accountable, documented choice. No canopy-enforced floor.

  • §1075 scope (maintainer-confirmed): a metadata-only FTI access log is not itself FTI, and forwarding it to a general collector does not expand the FTI data boundary. But the collector is relied on as audit evidence, so it needs completeness, integrity, retention, and access controls — see the audit-channel contract.

This is a large, cross-cutting change (all services, live compliance controls, a 100+-file retirement of shipped surfaces). It is decomposed and dependency-gated below; the retirement is LAST and gated behind a proven replacement.

Grounded facts (verified file:line)

Logging facility (with the corrections the mechanism must respect):

  • One JSON→stdout init: canopy_common::telemetry::init (telemetry.rs:71), but fmt and otel are sibling layers (telemetry.rs:158/164/178) — a passive Layer CANNOT rewrite an event before they serialize it. Redaction needs a custom FormatEvent/field visitor for stdout plus separate OTEL span/attribute sanitization.

  • EnvFilter sits at the registry root (telemetry.rs:73) — it can suppress audit records entirely. An audit channel must be UNFILTERABLE.

  • NOT "one seam, every service": canopy-portal runs its own tracing_subscriber::fmt().init() (services/canopy-portal/src/main.rs:65-70, non-JSON, own filter); bootstrap reads secrets BEFORE telemetry init (bootstrap.rs:90-91 vs :94) so the earliest events are uncovered (Child A reorders init ahead of secret reads).

  • Config-as-data template: rulesets/{jurisdiction}/jurisdiction.toml + citations.toml, array-of-records precedent (), selected by CANOPY_<SVC>__JURISDICTION. ServiceSettings has jurisdiction but NO ruleset root (settings.rs:39); Child A needs a common policy-root/bootstrap contract.

The retirement blast radius (bigger than first scoped):

  • FOUR live v1 hash-chain instances across THREE families, all shipped + running: TANF FTI + Medicaid FTI (fti_audit_log, off-bus, appended INSIDE the determination commit, fail-closed hot path — append_determination_chain_entries); general audit (audit_events, on-bus wildcard subscriber); ELE (ele_grant_events, medicaid, last_event_hash WRITTEN by grant/transition/renewal upserts store/ele.rs:582, live GET /v1/ele/chain-status). #1248 was the deferred chain-v2 migration of ELE — NOT this live v1 chain.

  • Chain-v2’s append/verification DATA PLANE is dormant (default-off), but schemas are migrated, HTTP routes registered, and staging/control-plane components run — the live contracts/routes/background components are part of the retirement.

  • Three SEPARATE FTI/PII controls that STAY distinct (only the field vocabulary is shared): scrub_fti_fields (mutates RabbitMQ payloads, fti_audit.rs:784), the publisher fail-closed guard (publisher.rs:296), and data-tenancy-authorisation.toml (service tenancy + CI scan). A tracing formatter cannot sanitize RabbitMQ or enforce tenancy.

  • The in-app Pub 1075 §9 detector is INERT: detection.rs dispatch handles only "event_count" (detection.rs:52-57); the seeded rule is 'failed_auth' → skipped; it reads only shared audit_events. It must be REPAIRED, not "retained" as working.

  • Citation ALREADY fail-closes in production: chain/attest returns verifier_unavailable while chain-v2 is dormant, so "Cite for hearing" 502s today. Re-homing RESTORES a broken feature — but flipping 502→success IS a live API/UI change.

  • Reporting owns its audit rows (ADR-004 Am 1 A6, #1256). Retention: Pub 1075 AU-11 is 7 years (the ADR-004 "5 years" statement is stale and is CORRECTED, not reaffirmed); ato-readiness.adoc carries the per-family envelope.

Decision contract (from ADR-041)

The full contract is ADR-041. The load-bearing points the children implement against:

  • Policy model — mechanism + secure default; fully jurisdiction-overridable; missing→default, malformed→fail-closed, explicit override honored (even weaker), no canopy floor.

  • Redaction mechanism — custom stdout FormatEvent/visitor and a separate OTEL sanitizer (span fields + updates, nested, Display/Debug, error chains, message-string secrets); installed in the common init AND canopy-portal; init reordered ahead of secret reads.

  • Audit-export channel — UNFILTERABLE, versioned+schema’d, stable ID, commit-coupled attempt/completion/crash, complete-row digest + policy version, dedup/flush/gap-reconciliation/ collector-health; audit DB rows stay system-of-record; production conformance gate before any retirement.

  • Three separate controls stayscrub_fti_fields, the publisher guard, the tenancy matrix; only the field vocabulary becomes shared config.

  • §9 detection repaired — dispatch/seed fixed; FTI + reporting inputs; in-app + config-driven.

  • Citation — a canopy-signed rendering of the current system-of-record row (ADR-029), removing the chain-state fail-closed matrix.

  • Reporting A6 preserved — reporting-owned rows + export; A7 chain-family withdrawn; A8 stays (#1256).

  • ADR-014 supersession — orig Decision + Am 1/3/4 + C1–C6 + Am 5–11 chain/anchor bindings; C7/C8 surviving non-chain obligations re-ratified.

  • Threat model — detects a privileged post-hoc DB row edit via off-box digest comparison; does NOT defeat an attacker controlling BOTH the DB AND the deployment logging control plane; no in-app cryptographic anti-privileged-rewrite claim.

Sequenced decomposition (dependency-gated; retirement is LAST)

Order: Decision MR → A → B → [capture+reconciliation CONFORMANCE gate] → D lifecycle readiness → C (last, per-family cutover gates). The old control must never disappear before the replacement is proven in a deployment.

Unit Scope (byte-level design authored in the unit)

Decision MR (#1299)

ADR-041 + ADR-014 Amendment 12 + ADR-004 Amendment 2 + this plan + nav/architecture/ato-readiness/auditor-handbook/roadmap/CHANGELOG. No code.

A — redaction mechanism (#1300)

Custom stdout formatter/field-visitor + OTEL sanitizer (span fields/updates, nested, Display/Debug, errors, message-string secrets); the policy loader (safe defaults, fully overridable, malformed=fail-closed) + the common policy-root/bootstrap contract + canopy-portal coverage + the executable service inventory.

B — audit-export channel (#1301)

The unfilterable versioned schema + stable ID + commit-coupled attempt/completion/crash + complete-row digest + policy version + dedup/flush/gap-reconciliation/collector-health; keep the audit DB rows; the deployment capture+reconciliation CONFORMANCE gate.

§9 repair (#1302)

Fix detection.rs dispatch/seed; define FTI/reporting inputs; keep it in-app + config-driven.

D — retention/legal-hold lifecycle (#1303)

General config-driven audit-table retention/purge across security live/archive, TANF+Medicaid FTI live/archive pairs, ELE, future reporting: per-family applicability, scheduler/ownership, leases, batches, purge-vs-archive, hold placement/release authority, races, recovery, metrics, deletion evidence; off-box witness availability for the reliance period. (Today POST /security/archive is a fixed 503; no legal-hold; no audit-retention ruleset.)

C — retirement (#1304, LAST, gated; split into steps, not "C1–C4", to avoid clashing with ADR-014’s control names)

(i) chain-v2 teardown — BOTH the dormant append/verify data plane AND the still-running staging/control-plane/background components; (ii) citation redesign; (iii) public-contract deprecation (status/verify/verify-jobs/archive endpoints + GET /v1/ele/chain-status, hash fields on audit/fact/ELE DTOs, EleStatus.last_event_hash, CLI, test-lib clients, OpenAPI, UI/E2E, config, tooling); (iv) per-live-family cutover with the migration safety below. ELE: last_event_hash is write-coupled to grant/renewal — define its replacement; removing the general-audit advisory lock needs a deterministic ordering contract for fact_change_history.

Migration safety / historical evidence (cutover boundary)

Pre-B rows cannot retroactively gain off-box evidence; removing their hashes + verifier destroys existing integrity evidence. Define a cutover boundary: retain legacy hashes + a READ-ONLY verifier until expiry, OR verify + export a durable closing checkpoint. Resolve or migrate open chain breaches into the incident system before status surfaces disappear. Schema retirement = new forward-only expand-contract migrations (applied migrations remain).

Reconciliation map

Artifact Disposition

#1278

Closed (superseded by ADR-041).

#1279 / #1280 / #1247 / #1248 / #1289 / #934

Withdrawn (chain-v2 activation/extension; #934 was the FTI ADR-014 hash-chain entry).

#1256

SPLIT: A6 reporting-owned rows + export → the facility; A8 storage controls stay as separate tenancy work (preserved, re-scoped in place).

#1208

Re-scoped: batching/index/schedule stay as scale work; the chain-verify portion drops with the chain.

C7 in-DB lifecycle

Re-homed to the general audit-table retention/purge (#1303).

§9 detector

Repaired (#1302) — not delegated.

ADR-014

Amendment 12: superseded-by-ADR-041 (orig Decision + Am 1/3/4 + C1–C6 incl. C5 + Am 5–11 chain/anchor bindings); C7/C8 surviving obligations re-ratified.

ADR-004

Amendment 2: mechanism → ADR-041; §Decision re-affirmed; retention corrected to 7yr (AU-11); A6 preserved; A7 chain-family withdrawn.

architecture.adoc / nav.adoc

ADR-041 added; the stale ADR-014 Amendment 10/11 omission fixed; chain-v2 plans marked superseded.

ato-readiness.adoc / auditor-handbook.adoc / CHANGELOG

Control mappings + 7yr retention + the retired features.

Verification matrix (what the guarantee actually requires)

stdout AND OTEL redaction; canopy-portal; span fields + updates; nested / Display / Debug / message-string leakage; missing (→safe default) / malformed (→fail-closed) / overbroad policy; audit-channel filter-immunity (EnvFilter cannot suppress it); stable schema/version + public-contract snapshots; attempt/outcome/crash-window semantics; collector outage/backpressure/restart/duplicates; gap reconciliation; retention/hold boundary races; mixed-version deployment; eligibility hot-path performance (the determination-commit append change). Global subscriber init ⇒ injectable writer/formatter + subprocess coverage.

Explicitly out

Building a SIEM / log-shipping / bespoke delivery pipeline (the deployment’s job); any in-app cryptographic anti-privileged-rewrite guarantee; one-shot breakage of a live compliance control (everything is sequenced behind the conformance gate).

Edit this page · default