Plan: ADR-041 — configurable structured logging + jurisdiction-owned field redaction; retire the FTI-special hash chain
On this page
Epic: &74
Contract: ADR-041
Related: &73 (scale readiness)
Status
| Unit | Description | Status |
|---|---|---|
Decision MR |
ADR-041 + ADR-014 Amendment 12 + ADR-004 Amendment 2 + this plan + nav/architecture/ato-readiness/auditor-handbook/roadmap/CHANGELOG (#1299). Docs only. |
In progress |
A — redaction mechanism |
Custom stdout |
Not started |
B — audit-export channel |
The unfilterable, versioned, digest+policy-version, commit-coupled channel with dedup/flush/gap-reconciliation/collector-health, keeping the audit DB rows; the production capture+reconciliation conformance gate (#1301). |
Not started |
§9 detection repair |
Fix |
Not started |
D — retention/legal-hold lifecycle |
General config-driven audit-table retention/purge + legal hold across all audit families; replace the fixed-503 |
Not started |
C — retirement (LAST, gated) |
chain-v2 teardown + citation redesign + public-contract deprecation + per-family cutover, behind the Child-B conformance gate (#1304). |
Not started |
Context
This work started as "#1278: build the chain-v2 external anchor" and was reframed, through maintainer review, to its actual shape:
-
The external anchor — and then the hash chain itself — is special-cased tamper-evidence for one log type. Tamper-evidence + retention are general logging-infrastructure properties, satisfied per-deployment by standard logging facilities.
-
Principle: canopy provides the mechanism, the deployment provides the policy. A general configurable structured-logging facility in every service, with per-field redaction. Fully jurisdiction-overridable — the deployment owns all compliance risk (maintainer decision): canopy ships safe defaults + the mechanism; a deployment can override anything (including the FTI/PHI defaults); a malformed policy fails closed (config error), an explicit override to a weaker policy is the deployment’s accountable, documented choice. No canopy-enforced floor.
-
§1075 scope (maintainer-confirmed): a metadata-only FTI access log is not itself FTI, and forwarding it to a general collector does not expand the FTI data boundary. But the collector is relied on as audit evidence, so it needs completeness, integrity, retention, and access controls — see the audit-channel contract.
This is a large, cross-cutting change (all services, live compliance controls, a 100+-file retirement of shipped surfaces). It is decomposed and dependency-gated below; the retirement is LAST and gated behind a proven replacement.
Grounded facts (verified file:line)
Logging facility (with the corrections the mechanism must respect):
-
One JSON→stdout init:
canopy_common::telemetry::init(telemetry.rs:71), butfmtandotelare sibling layers (telemetry.rs:158/164/178) — a passiveLayerCANNOT rewrite an event before they serialize it. Redaction needs a customFormatEvent/field visitor for stdout plus separate OTEL span/attribute sanitization. -
EnvFiltersits at the registry root (telemetry.rs:73) — it can suppress audit records entirely. An audit channel must be UNFILTERABLE. -
NOT "one seam, every service":
canopy-portalruns its owntracing_subscriber::fmt().init()(services/canopy-portal/src/main.rs:65-70, non-JSON, own filter);bootstrapreads secrets BEFORE telemetry init (bootstrap.rs:90-91vs:94) so the earliest events are uncovered (Child A reorders init ahead of secret reads). -
Config-as-data template:
rulesets/{jurisdiction}/jurisdiction.toml+citations.toml, array-of-records precedent (), selected byCANOPY_<SVC>__JURISDICTION.ServiceSettingshasjurisdictionbut NO ruleset root (settings.rs:39); Child A needs a common policy-root/bootstrap contract.
The retirement blast radius (bigger than first scoped):
-
FOUR live v1 hash-chain instances across THREE families, all shipped + running: TANF FTI + Medicaid FTI (
fti_audit_log, off-bus, appended INSIDE the determination commit, fail-closed hot path —append_determination_chain_entries); general audit (audit_events, on-bus wildcard subscriber); ELE (ele_grant_events, medicaid,last_event_hashWRITTEN by grant/transition/renewal upsertsstore/ele.rs:582, liveGET /v1/ele/chain-status).#1248was the deferred chain-v2 migration of ELE — NOT this live v1 chain. -
Chain-v2’s append/verification DATA PLANE is dormant (default-off), but schemas are migrated, HTTP routes registered, and staging/control-plane components run — the live contracts/routes/background components are part of the retirement.
-
Three SEPARATE FTI/PII controls that STAY distinct (only the field vocabulary is shared):
scrub_fti_fields(mutates RabbitMQ payloads,fti_audit.rs:784), the publisher fail-closed guard (publisher.rs:296), anddata-tenancy-authorisation.toml(service tenancy + CI scan). A tracing formatter cannot sanitize RabbitMQ or enforce tenancy. -
The in-app Pub 1075 §9 detector is INERT:
detection.rsdispatch handles only"event_count"(detection.rs:52-57); the seeded rule is'failed_auth'→ skipped; it reads only sharedaudit_events. It must be REPAIRED, not "retained" as working. -
Citation ALREADY fail-closes in production:
chain/attestreturnsverifier_unavailablewhile chain-v2 is dormant, so "Cite for hearing" 502s today. Re-homing RESTORES a broken feature — but flipping 502→success IS a live API/UI change. -
Reporting owns its audit rows (ADR-004 Am 1 A6,
#1256). Retention: Pub 1075 AU-11 is 7 years (the ADR-004 "5 years" statement is stale and is CORRECTED, not reaffirmed);ato-readiness.adoccarries the per-family envelope.
Decision contract (from ADR-041)
The full contract is ADR-041. The load-bearing points the children implement against:
-
Policy model — mechanism + secure default; fully jurisdiction-overridable; missing→default, malformed→fail-closed, explicit override honored (even weaker), no canopy floor.
-
Redaction mechanism — custom stdout
FormatEvent/visitor and a separate OTEL sanitizer (span fields + updates, nested,Display/Debug, error chains, message-string secrets); installed in the common init AND canopy-portal; init reordered ahead of secret reads. -
Audit-export channel — UNFILTERABLE, versioned+schema’d, stable ID, commit-coupled attempt/completion/crash, complete-row digest + policy version, dedup/flush/gap-reconciliation/ collector-health; audit DB rows stay system-of-record; production conformance gate before any retirement.
-
Three separate controls stay —
scrub_fti_fields, the publisher guard, the tenancy matrix; only the field vocabulary becomes shared config. -
§9 detection repaired — dispatch/seed fixed; FTI + reporting inputs; in-app + config-driven.
-
Citation — a canopy-signed rendering of the current system-of-record row (ADR-029), removing the chain-state fail-closed matrix.
-
Reporting A6 preserved — reporting-owned rows + export; A7 chain-family withdrawn; A8 stays (#1256).
-
ADR-014 supersession — orig Decision + Am 1/3/4 + C1–C6 + Am 5–11 chain/anchor bindings; C7/C8 surviving non-chain obligations re-ratified.
-
Threat model — detects a privileged post-hoc DB row edit via off-box digest comparison; does NOT defeat an attacker controlling BOTH the DB AND the deployment logging control plane; no in-app cryptographic anti-privileged-rewrite claim.
Sequenced decomposition (dependency-gated; retirement is LAST)
Order: Decision MR → A → B → [capture+reconciliation CONFORMANCE gate] → D lifecycle readiness → C (last, per-family cutover gates). The old control must never disappear before the replacement is proven in a deployment.
| Unit | Scope (byte-level design authored in the unit) |
|---|---|
Decision MR (#1299) |
ADR-041 + ADR-014 Amendment 12 + ADR-004 Amendment 2 + this plan + nav/architecture/ato-readiness/auditor-handbook/roadmap/CHANGELOG. No code. |
A — redaction mechanism (#1300) |
Custom stdout formatter/field-visitor + OTEL sanitizer (span fields/updates, nested, Display/Debug, errors, message-string secrets); the policy loader (safe defaults, fully overridable, malformed=fail-closed) + the common policy-root/bootstrap contract + canopy-portal coverage + the executable service inventory. |
B — audit-export channel (#1301) |
The unfilterable versioned schema + stable ID + commit-coupled attempt/completion/crash + complete-row digest + policy version + dedup/flush/gap-reconciliation/collector-health; keep the audit DB rows; the deployment capture+reconciliation CONFORMANCE gate. |
§9 repair (#1302) |
Fix |
D — retention/legal-hold lifecycle (#1303) |
General config-driven audit-table retention/purge across security live/archive, TANF+Medicaid FTI live/archive pairs, ELE, future reporting: per-family applicability, scheduler/ownership, leases, batches, purge-vs-archive, hold placement/release authority, races, recovery, metrics, deletion evidence; off-box witness availability for the reliance period. (Today |
C — retirement (#1304, LAST, gated; split into steps, not "C1–C4", to avoid clashing with ADR-014’s control names) |
(i) chain-v2 teardown — BOTH the dormant append/verify data plane AND the still-running staging/control-plane/background components; (ii) citation redesign; (iii) public-contract deprecation (status/verify/verify-jobs/archive endpoints + |
Migration safety / historical evidence (cutover boundary)
Pre-B rows cannot retroactively gain off-box evidence; removing their hashes + verifier destroys existing integrity evidence. Define a cutover boundary: retain legacy hashes + a READ-ONLY verifier until expiry, OR verify + export a durable closing checkpoint. Resolve or migrate open chain breaches into the incident system before status surfaces disappear. Schema retirement = new forward-only expand-contract migrations (applied migrations remain).
Reconciliation map
| Artifact | Disposition |
|---|---|
#1278 |
Closed (superseded by ADR-041). |
#1279 / #1280 / #1247 / #1248 / #1289 / #934 |
Withdrawn (chain-v2 activation/extension; #934 was the FTI ADR-014 hash-chain entry). |
#1256 |
SPLIT: A6 reporting-owned rows + export → the facility; A8 storage controls stay as separate tenancy work (preserved, re-scoped in place). |
#1208 |
Re-scoped: batching/index/schedule stay as scale work; the chain-verify portion drops with the chain. |
C7 in-DB lifecycle |
Re-homed to the general audit-table retention/purge (#1303). |
§9 detector |
Repaired (#1302) — not delegated. |
ADR-014 |
Amendment 12: superseded-by-ADR-041 (orig Decision + Am 1/3/4 + C1–C6 incl. C5 + Am 5–11 chain/anchor bindings); C7/C8 surviving obligations re-ratified. |
ADR-004 |
Amendment 2: mechanism → ADR-041; §Decision re-affirmed; retention corrected to 7yr (AU-11); A6 preserved; A7 chain-family withdrawn. |
architecture.adoc / nav.adoc |
ADR-041 added; the stale ADR-014 Amendment 10/11 omission fixed; chain-v2 plans marked superseded. |
ato-readiness.adoc / auditor-handbook.adoc / CHANGELOG |
Control mappings + 7yr retention + the retired features. |
Verification matrix (what the guarantee actually requires)
stdout AND OTEL redaction; canopy-portal; span fields + updates; nested / Display / Debug
/ message-string leakage; missing (→safe default) / malformed (→fail-closed) / overbroad
policy; audit-channel filter-immunity (EnvFilter cannot suppress it); stable schema/version
+ public-contract snapshots; attempt/outcome/crash-window semantics; collector
outage/backpressure/restart/duplicates; gap reconciliation; retention/hold boundary races;
mixed-version deployment; eligibility hot-path performance (the determination-commit append
change). Global subscriber init ⇒ injectable writer/formatter + subprocess coverage.
Explicitly out
Building a SIEM / log-shipping / bespoke delivery pipeline (the deployment’s job); any in-app cryptographic anti-privileged-rewrite guarantee; one-shot breakage of a live compliance control (everything is sequenced behind the conformance gate).