T2-6 — Crypto-shred redaction/expungement + JWS signing-key retention (#687)

On this page

Epic &56 / Track 2, T2-6 (#687). ADR-027 §8 names the Track-2 fix for the append-only tension: "a genuine purge via crypto-shredding — per-value encryption where redaction destroys that value’s key, leaving the hash over ciphertext intact so the chain stays verifiable — across facts, events, and snapshots, with an explicit Pub 1075 access-audit story for SSN." #687 also carries an independent second half: JWS verification-key retention — today a rotated verification key is removed 30 days after rotation, so a determination signed with a retired key can never be re-verified. This plan delivers both as one dependency-sliced multi-MR marathon closing #687, on the hash-over-ciphertext model, ripping out the back-compat scaffolding that pre-1.0 (no production data, forward-only ADR-016 migrations, devstack re-seeds) no longer needs.

NOTE
All file.rs:NNN anchors below are pre-implementation (accurate on main as of the plan commit). Pair each with its semantic anchor (the named fn/struct), which is the durable address — line numbers drift as the code evolves.

Scope boundary

In scope:

  • Foundation — a new canopy-crypto-shred crate: the SealedValue AEAD envelope (AAD-bound), per-value DEK generation + KEK-wrap (extending canopy_common::crypto with AAD variants), the RedactionKeyStore + KeyHistoryProvider traits, and the hash-over-ciphertext discipline. Property-tested.

  • Crypto-shred across all three surfaces — (a) canopy-persons fact-value version tables + the persons-table PII columns (ssn_encrypted, date_of_birth); (b) the attributed event / canopy-security change-history before/after payloads; (c) the determination snapshots (all five program services).

  • The redaction/expungement operation — per-service endpoints + canopy CLI parity (ADR-027 §10), role-gated, emitting tamper-evident *.redacted audit events with a cross-service fan-out; the Pub-1075 ssn.accessed audit event on every SSN open.

  • JWS verification-key retention — a persistent signing_key_history store, a stable program-bound kid scheme, an async VerifyingKeyRegistry lazy-load on cache-miss, and a GET …/jwks endpoint.

  • Back-compat rip-outs (per the maintainer directive) — audit-chain v1 + hash_version; env CANOPY_VERIFY_KEY_*_PREV; snapshot_hash: Option + NoInputSnapshot/tri-state read; snapshot schema_version 1/2/3 plaintext formats (collapse to v4-only, with a v4 lower-bound on read).

  • ADRs — new ADR-036 (the crypto-shred + key-retention architecture, incl. an explicit threat model) + amendments to ADR-028/014/017.

Out of scope (each a boundary with a reason; filed as follow-ups, never silently dropped):

  • KEK rotation tooling / Vault backend — the kek_version column + the unwrap-old/wrap-new re-wrap path are designed in; the operator runbook + xtask kek-rotate are a follow-up (ADR-017’s CANOPY_ENCRYPTION_KEY + _PREVIOUS window already covers the in-flight decrypt path).

  • Two-person-integrity enforcement for expungement — the ADR records the requirement; v1 enforces a single privileged role + mandatory reason + actor capture. Dual-control needs an approvals surface that does not exist yet.

  • WAL/backup secure-erase tooling — see Decision O (threat model): shred is application-layer; block-layer-encrypted storage + bounded backup retention are operational prerequisites, with a post-grace secure-overwrite sweep filed as a follow-up.

  • Encrypting non-PII structural columns — sealing buys no redaction value and costs queryability (Decision L).

Status

Step Description Status

(plan)

This execution plan + its nav.adoc entry; iterated through contextless review rounds (implementability, crypto/security, convention) until a fresh reviewer found nothing material.

Done (2026-06-24) — the plan commit.

MR1 — foundation crate + ADR-036

canopy-crypto-shred (SealedValue, AAD-bound seal/open, DEK+KEK-wrap, RedactionKeyStore/KeyHistoryProvider traits) + AAD variants + random_key on canopy_common::crypto + proptests; zeroize workspace dep (hkdf NOT added — unused per Decision C, would fail cargo machete); ADR-036 draft (Proposed).

Done (2026-06-24) — the implementation commit.

MR2a — key-retention store + lazy-load mechanism

canopy-security signing_key_history (append-only) + POST /v1/security/signing-keys (register) + GET …/{program}/jwks; canopy-signing additive async fn verify_with_history (keeps the sync verify for the orphaned DeterminationVerifier path; refines Decision H — no async ripple into that trait/tests) + jwk PEM↔xy helpers; canopy-eligibility HTTP KeyHistoryProvider (JWKS→PEM) + security_url config + orchestrator uses the async variant. Integration-tested via the register/JWKS round-trip (no registry cache yet — retired-kid verify re-fetches; perf follow-up).

Done (2026-06-24) — the implementation commit.

MR2b — key-derived kid + boot self-registration

The 5 program services mint a key-derived kid (canopy-{program}-{sha256(public_pem)[..16]} — refines Decision G to be collision-free vs. the operator-supplied YYYYqN; settled with the user before building, ADR-036 still Proposed) and idempotently register their current public key into signing_key_history on boot via a shared canopy-api helper. Closes the retention half.

Done (2026-06-24) — the implementation commit.

MR3 — rip out _PREV env + runbook

Delete CANOPY_VERIFY_KEY_*_PREV loading from both registry loaders; rewrite the security-operations.adoc rotation runbook (no PREV slot, no 30-day grace). Refinement: the rotation window is now served automatically by the MR2b lazy-load (a determination signed with the old key misses in-memory → lazy-loads the old key from signing_key_history), so no in-memory second key is sourced at startup — _PREV is fully removed, not relocated.

Done (2026-06-24) — the implementation commit.

MR4 — audit-chain v1 rip-out

Collapse to the single (former-v2) formula; drop the hash_version column.

Done (2026-06-24) — the implementation commit.

MR5 — snapshot sealing contract + seal ALL programs (re-sliced)

DeterminationSnapshot value leaves → Sealed*; schema_version 4 (+ a < 4 plaintext floor). Wire sealing + a redaction_keys table + RedactionKeyStore impl into all 5 program services at once (the shared canopy-contracts-eligibility leaf type can’t change per-program without back-compat scaffolding the maintainer ruled out — settled with the maintainer 2026-06-24; see the Re-slice note below). Orchestrator treats the snapshot as opaque for hash/signature verify (the sealed leaves are ct strings). Scope split (2026-06-24): the snapshot_hash-required (Option→String on SignableDetermination + the 5 program DTOs) + drop-NoInputSnapshot/tri-state contract-hardening is separable + mechanical (no crypto), and cascades into 5 DTOs + canopy-signing + canopy-web + OpenAPI — split to a focused follow-up (#911) to keep the crypto-sealing MR reviewable. The schema_version < 4 floor already refuses plaintext snapshots, so sealing is sound without the optionality change.

Done (2026-06-24) — the implementation commits (sealing only; the split-out snapshot_hash-required hardening is follow-up #911).

MR6 — snap redaction op + CLI (reference)

POST …/determinations/{id}/redact (sub-resource form, not :redact — see the as-built note), gated on the new dedicated data_steward role; CLI canopy snap determination redact --id <id> --reason <reason>; the shred + a plaintext-free determination.redacted audit event commit in one TX. Headline shred test (redact → snapshot canonical_hash unchanged + JWS-verifiable + redaction_keys row tombstoned) + role-gate/idempotent/404 tests. Sealing already lands in MR5; MR6 is the redaction operation only.

Done (2026-06-24) — the implementation commit.

MR7 — redaction op + CLI for tanf/medicaid/caps/wic

Mechanically identical per-service redaction op (the /redact endpoint — sub-resource form per the MR6 as-built note + CLI parity). Sealing + the redaction_keys tombstone trigger already landed in MR5. As-built (DRY): the RedactDeterminationRequest/RedactDeterminationResponse DTOs + the REDACT_DETERMINATION path const were promoted from canopy-contracts-snap to the shared canopy-contracts-eligibility crate (used directly by the four programs; snap re-exports for compat) so the wire shape has one definition, not a per-program copy; the four programs share one cmd::program::redact_determination CLI helper.

Done (2026-06-25) — the implementation commit.

MR8 — persons fact-value + persons-PII sealing + redaction + CLI

Seal value columns + ssn/date_of_birth; fact-redact + /redact-ssn endpoints (sub-resource form per the MR6 as-built note) + CLI; SQL-aggregate audit gate.

Done (2026-06-25) — the implementation commit. As-built deviations: (1) the per-fact DEK uses subject_id = fact_id (not version_id): the redaction unit is the fact, so a remnant re-tile copies the sealed envelope verbatim and a redact shreds every version in one shred_with(subject_kind, fact_id) UPDATE. (2) Reads surface a redacted fact with redacted: true + value-leaves None (the row is kept — append-only/auditable erasure), so the shared read DTOs' value fields became Option and all eligibility consumers (orchestrator filter, medicaid ELE, web) were updated in the same MR (J3). (3) canopy-persons now require_kek at boot (no plaintext-PII fallback). (4) The SQL-aggregate audit gate (Decision N) found no SQL-side value math in persons; the new household_member_versions table (added after the plan) was assessed — its only value column is non-PII relationship, nothing to seal. (5) Deferred follow-up: the DEMO-profile seed personas still hand-write plaintext PII (separate from the default sealed random seed).

MR9 (FINAL) — event sealing + shared-DEK redaction + Pub-1075 + close

Sealed before/after (sealed in-store); audit-copy expungement via the shared per-fact DEK; ssn.accessed; docs/status flip. Closes #687.

Done (2026-06-25) — the implementation commit. As-built deviations: (1) No cross-service fan-out (Model B, the architecturally-correct realization). The proposed Decision M had canopy-security re-seal the audit before/after under its own DEK and shred it on a fact.redacted fan-out. That is an antipattern: canopy-security cannot own a second value-key without receiving plaintext (violating ADR-004), and a second key makes redaction a delivery-dependent distributed transaction (PII survives in the audit copy if the event is lost). Instead the persons store seals each event’s PII leaves under the same per-fact DEK as the at-rest value (the envelope copied verbatim, never re-sealed), so the canopy-security audit copy is ciphertext under that one DEK — redacting the fact expunges the at-rest and audit copies atomically, with no security-side redaction_keys, KEK, or subscriber. (2) The event value leaves become typed sealed shapes (IncomeEventValue/AssetEventValue/ExpenseEventValue) carrying SealedDecimal/SealedValue; the store builds + returns the fully-typed *ClaimedEvent so the publisher stages it verbatim (no plaintext on the *_before/publish path; income/asset/expense close no longer take a KEK). (3) MR8’s inline-json! redaction events are formalized as typed FactRedactedEvent/SsnRedactedEvent/SsnAccessedEvent (+ the SsnAccessPurpose enum) in canopy-contracts-persons. (4) ssn.accessed (Pub-1075) fires at the 7 persons_to_wire SSN-open sites, one per genuinely-decrypted person (ssn_last_four.is_some()), staged through a short outbox tx — fail-closed (no disclosure without an audit row); a redacted SSN reads None and fires nothing. (5) The canopy-security change-history renders (sealed) for value leaves (it never holds the DEK); worker-facing value display re-sourced from the system-of-record is filed as #920. (6) The amount-fidelity assertions in the event-emission + finalize tests move to API reads (server-side open) — superseded-window values are not API-readable, so those assert structure + sealing, mirroring the MR8 no-KEK-in-test convention.

Epic: &56
Issue: #687 — a single issue delivered as 9 dependency-sliced MRs (justified per gitlab-issue-mr-standards: each slice is independently reviewable
mergeable + leaves the tree green; bundling would be one unreviewable diff across ~9 crates). Relates to #687 on MR1–8; Closes #687 on MR9.
Branches: feat/fact-authoring-t2-6-{foundation,key-retention,prev-rip,chain-v1-rip,snapshot-seal,snap-redact,program-fanout,persons-seal,events-finalize}, each cut fresh from a main that already has its deps (not stacked); regular merge commits, never squash.
Merge order (mandatory — not stacked): MR1 first. Then three independent chains off MR1: (retention) MR2 → MR3; (chain rip-out) MR4; (sealing) MR5 → MR6 → MR7 and MR5 → MR8. MR9 depends on BOTH MR4 (v2-only metadata hash) AND MR8 (sealed fact events) and is last. MR5/MR6/MR7 do not depend on MR4 (snapshots are not audit_events).

NOTE
Re-slice (2026-06-24): snapshot sealing is all-programs-at-once, redaction stays per-program

The original slicing assumed sealing could roll out per program (MR5 contract → MR6 snap → MR7 others). It can’t: all five program services construct IncomeFactLeaf / AssetFactLeaf / ExpenseFactLeaf directly against the shared canopy-contracts-eligibility::DeterminationSnapshot type, so changing a leaf field (Decimal → SealedDecimal) changes the type all five compile against — it breaks every producer at once. Sealing one program at a time would require a transitional plaintext-or-sealed leaf representation, i.e. exactly the back-compat scaffolding the maintainer ruled out. So MR5 seals all five programs together (contract change + per-program redaction_keys
RedactionKeyStore + builder wiring), and MR6/MR7 carry only the genuinely-per-service redaction operation (the /redact endpoint — sub-resource form per the MR6 as-built note + one-way-tombstone trigger + CLI). The dependency graph and MR count are unchanged; only the MR5↔MR6/MR7 content boundary moved. Maintainer-approved 2026-06-24.

Context

ADR-027 §8 makes a plain DELETE untenable: append-only facts + immutable signed snapshots + a tamper-evident chain mean a delete is either blocked (immutability triggers) or chain-breaking (deleting a hashed value rotates every downstream hash). Crypto-shred resolves it: encrypt the value, hash the ciphertext, redact by destroying the per-value key — the ciphertext + hash stay (chain + signature still verify), only the plaintext becomes unrecoverable.

Today (verified on main):

  • Fact values are plaintext columns — income_versions.amount NUMERIC(10,2), employer_name TEXT, address_versions.line_1/line_2, etc.

  • Only SSN is encrypted at rest — ssn_encrypted, via a single service-wide key applied directly (services/canopy-persons/src/store/persons.rs, encrypt_ssn), no per-value key → not selectively shreddable. date_of_birth is plaintext on the persons row.

  • The snapshot hash is over plaintext — serde_jcs::to_vec(snapshot) (crates/canopy-contracts-eligibility/src/snapshot.rs, canonical_bytes); the ECDSA P-256 detached-JWS signature binds snapshot_hash (crates/canopy-signing/src/envelope.rs, set before signing in the program determine.rs).

  • The audit chain is dual-path — v1 (delimiter-free concat) + v2 (JCS over AuditChainInputsV2, which includes the full metadata JSONB) selected by a hash_version column (services/canopy-security/src/store/mod.rs).

  • The verifier is in-memory only — VerifyingKeyRegistry (crates/canopy-signing/src/verifier.rs) loads CANOPY_VERIFY_KEY_{P} + _PREV env vars; verify_detached is synchronous; the runbook removes _PREV after 30 days → retired-key determinations become unverifiable forever.

Reuse target. canopy_common::crypto already provides AES-256-GCM encrypt/decrypt (output nonce||ct||tag, non-deterministic), EncryptionKeys{current, previous}, decrypt_with_rotation, and the CANOPY_ENV fail-closed loader (ADR-017). aes-gcm 0.10, sha2, p256 (with jwk), rand/getrandom are workspace deps; hkdf/zeroize are transitive-only (MR1 cargo add`s them). `deny.toml bans openssl → pure-Rust only. crypto::encrypt/decrypt take no AAD today — MR1 adds encrypt_with_aad/decrypt_with_aad (the existing fns delegate with empty AAD, so the SSN path is byte-compatible until MR8 migrates it).

Decisions

Decision Resolution

A — new canopy-crypto-shred crate, not a canopy-common extension

canopy-common is the universal leaf dep (recompiling it recompiles the world). Crypto-shred needs new deps (hkdf, zeroize) + stateful key-store traits; a dedicated crate depending on canopy-common (reusing crypto::{encrypt_with_aad,decrypt_with_aad}) is pulled only by the ~7 sealing services. #![forbid(unsafe_code)], typed thiserror errors.

B — hash-over-ciphertext; seal ONCE; the SealedValue is the hashed unit

A value field becomes a SealedValue (ciphertext). serde_jcs hashes the SealedValue → the hash covers ct. The seal happens once at write/assembly time; stored bytes re-serialize verbatim on every read. No read path ever re-seals (AES-GCM’s random nonce would change ct → change the hash → break the signature). Shred mutates only redaction_keys, never the hashed artifact → hash + signature unchanged; only open() returns Ok(None). A correction is a new version row with its own SealedValue (the bitemporal model already appends, never mutates) — so corrections never re-seal an existing value. Load-bearing (Risk 1; proptest-guarded + a "no seal() on a read path" review rule, Risk 7).

C — KEK = the existing CANOPY_ENCRYPTION_KEY; per-value DEK is random + KEK-wrapped; both encryptions bind AAD

Reuse the ADR-017 secret + fail-closed loader + EncryptionKeys rotation window as the per-service KEK (no new env var). Each sealed value gets a fresh random 32-byte DEK (OsRng, in Zeroizing), AEAD-wrapped under the KEK, stored in redaction_keys. Random independent DEKs, not HKDF-derived (destroying one reveals nothing about siblings). AAD binding (hardening): the value-seal binds AAD = "{v}:{alg}:{dek_id}"; the DEK-wrap binds AAD = "{dek_id}:{subject_kind}:{subject_id}". So a swapped redaction_keys row (open(dek_id_A, wrapped_dek_B)) fails the auth tag — no confused-deputy across values.

D — DEK granularity = redaction granularity

One DEK per version-row for facts (a row’s value-tuple redacts together); per PII column per person for persons-table PII (ssn and date_of_birth get separate DEKs so one redacts without the other; subject_id = person_id); per audit-event for events; per determination for snapshots (a frozen legal artifact is expunged wholesale; per-leaf snapshot DEKs would ~20× the rows with no redaction benefit).

E — redaction_keys per-service store (ADR-001); shred = one-way tombstone, DB-enforced

Each sealing service owns its table (ADR-001). Shred = UPDATE … SET wrapped_dek = <32-zero-byte sentinel>, shredded_at = now() WHERE shredded_at IS NULL. A dedicated one-way-tombstone trigger (NOT the snapshot maintenance GUC — redaction is a routine privileged op, not a sweep) permits exactly that transition and the INSERT; it rejects any other UPDATE, any DELETE/TRUNCATE, and un-tombstoning (shredded_at non-NULL → NULL). The redact endpoint runs the UPDATE directly; idempotency is the WHERE shredded_at IS NULL (re-shred = 0 rows).

F — signing_key_history lives in canopy-security (an explicit ADR-001 carve-out)

Public verification-key material is cross-cutting compliance metadata, not program-tenant data; the orchestrator already reaches across services to verify; canopy-security owns tamper-evidence (ADR-014). One store = one retention owner + one lazy-load target. Public keys only — never private material, never FTI. The carve-out is recorded in ADR-036 for reviewer scrutiny.

G — stable program-bound kid (operator-supplied, validated)

Today kids are canopy-{program}-current/-prev — slot names reused every rotation → a kid-keyed history would collide. New scheme: kid = canopy-{program}-{generation}, generation = YYYYqN (e.g. canopy-snap-2026q2), operator-supplied via CANOPY_{PROGRAM}__SIGNING_KID and validated at startup against ^canopy-{program}-\d{4}q[1-4]$ (fail-closed if absent/malformed; rejects a kid whose program segment ≠ the service’s program). On startup each program idempotently INSERTs its current public key into signing_key_history.

H — async VerifyingKeyRegistry lazy-loads a program-bound kid on cache-miss; a JWKS endpoint backs it

verify becomes async fn verify(&self, program, payload, jws). It extracts the kid (jws_kid), rejects a kid whose prefix ≠ canopy-{program}- (defeats cross-program/forged-kid fetches), tries the in-memory map, and on miss calls an injected Option<Arc<dyn KeyHistoryProvider>> querying by (program, kid) (HTTP-backed in the orchestrator; None → today’s in-memory-only behavior, so unit tests need no DB), verifies, caches. GET /v1/signing-keys/{program}/jwks (incl. retired keys; p256 jwk feature) is the interop surface + the provider’s backing. Orchestrator call sites are already async.

I — rip out env _PREV dual-key loading; the rotation window is now served by the lazy-load

Delete CANOPY_VERIFY_KEY_*_PREV loading. As-built refinement: the zero-downtime window needs no in-memory second key at all — the MR2b lazy-load already serves it (a determination signed with the old key misses the in-memory current key, then verify_with_history fetches the old key from signing_key_history, where it was registered while active). So _PREV is fully removed, not relocated; each loader now holds only the current key. RotationState::DualKeyRotation / add_keys survive as a programmatic escape hatch (and a test exercises them) but env never loads two keys. Rewrite security-operations.adoc + runbooks/signing-key-rotation.adoc (delete the PREV slot + the 30-day / dual-key-window steps).

J — rip out audit-chain v1 + hash_version; v2 is the sole formula

No v1 rows exist post-reseed. Delete compute_event_hash_v1 + the hash_version dispatch; drop the column (audit_events + archive). Sealing needs zero v2 formula change — v2 already JCS-hashes the full metadata, so sealed before/after are covered as ct automatically.

K — snapshot: snapshot_hash required; schema_version 4 with a v4 lower-bound; drop legacy read affordances

Make SignableDetermination.snapshot_hash non-Option; delete SnapshotStatus::NoInputSnapshot + the tri-state 404-legacy read; snap_determinations.snapshot_hash → NOT NULL (the migration DELETE`s any legacy NULL-hash rows first — devstack re-seeds, no prod data). `SCHEMA_VERSION_MAX = 4, v4 = "value leaves are SealedValue`"; `verify_schema_version rejects both > 4 and < 4 (no plaintext-downgrade).

L — seal PII-bearing values; leave structural discriminators plaintext

See the two lists below the table.

M — redaction op: role-gated, emits tamper-evident *.redacted; SSN open emits Pub-1075 ssn.accessed

Redaction is privileged + irreversible: a dedicated canopy:redact/data-steward role behind the #632 gate, mandatory reason, actor sub captured. It emits a .redacted audit event (no plaintext) that chains into the ledger; a fact-value .redacted carries the audit linkage so the canopy-security subscriber can shred the matching audit-event value-DEK (the fan-out). Every SSN open emits ssn.accessed ({actor_sub, person_id, purpose, source_service}, no plaintext; purpose validated against an enum, never free-text) per ADR-027 §8, obeying ADR-004 event scrubbing.

N — NUMERIC→BYTEA: drop the value CHECK constraints, audit SQL aggregates; EXCLUDE is unaffected

The non-overlap EXCLUDE keys only on fact_id+daterange → sealing value columns does not touch it. CHECK (amount >= 0) becomes meaningless on ciphertext → drop it (the invariant moves to the application layer pre-seal). Any SQL SUM/WHERE amount > x breaks — MR8 runs the SQL-aggregate audit gate (below) before sealing.

O — honest threat model: shred is application-layer redaction

Tombstoning the DEK destroys it in the live DB, but the wrapped-DEK plaintext can residue in Postgres WAL + base backups (until retention expires), unencrypted storage pages (until overwritten), and the EncryptionKeys.previous KEK during a rolling rotation. ADR-036’s threat model states this explicitly and sets the operational prerequisites: block-layer-encrypted storage (so old pages are unreadable), bounded backup retention, and brief KEK-rotation windows. "The value is gone" is scoped to the application/live-DB layer; a post-grace secure-overwrite WAL/backup sweep is a filed follow-up. (Honest per Kerckhoffs — no overclaim.)

Decision L — seal vs. leave plaintext:

  • Seal (PII-bearing): money (amount, value), employer_name, description, address line_1/line_2, persons-table ssn + date_of_birth, SOLQ dollar amounts, DerivedFactNode.value, the whole program_input (embeds household money) + cross_program_inputs (FTI-derived).

  • Leave plaintext (structural discriminators / non-PII — sealing costs queryability, buys no redaction value): income_type/asset_type/expense_type, frequency, relationship, household_size, all UUIDs (fact_id/person_id/…), corpus_hash, policy_params (jurisdiction thresholds), address_type/city/state/zip/county_fips (already coarse; the street is the PII).

Data model

New types live in canopy-crypto-shred (MR1) unless noted. serde_json::Value appears only where a sealed value is genuinely heterogeneous (the existing program_input/DerivedFactNode.value STRUCTURAL-VALUE pattern, ADR-003); no business logic reads it.

// canopy-crypto-shred (MR1) — the AEAD envelope; serializes JCS-stably and is the hashed unit.
pub struct SealedValue {
    pub v: u8,            // envelope format version (1); refuse unknown on read
    pub alg: String,      // "A256GCM"
    pub dek_id: Uuid,     // -> redaction_keys.dek_id; shred tombstones that row, orphaning this
    pub ct: String,       // base64url(nonce||ct||tag); value-seal AAD = "{v}:{alg}:{dek_id}"
}
// Hand-written Debug redacts `ct`. Send + Sync. Derives Serialize/Deserialize/Clone/PartialEq/Eq.

pub struct SealedDecimal(SealedValue);  // seal: rescale(2) -> canonical string -> seal
pub struct SealedJson(SealedValue);     // seal: serde_jcs canonical bytes -> seal

// canopy_common::crypto (MR1) — AAD-capable variants; existing encrypt/decrypt delegate w/ empty AAD.
pub fn encrypt_with_aad(plaintext: &[u8], key: &[u8;32], aad: &[u8]) -> Result<Vec<u8>, CryptoError>;
pub fn decrypt_with_aad(ciphertext: &[u8], key: &[u8;32], aad: &[u8]) -> Result<Vec<u8>, CryptoError>;

pub trait RedactionKeyStore {              // sqlx-backed per service; the impl holds the service's PgPool
    // mint DEK -> wrap under KEK (AAD = dek_id:subject_kind:subject_id) -> persist -> seal value (AAD = v:alg:dek_id)
    async fn seal(&self, kek: &EncryptionKeys, subject_kind: &str, subject_id: Uuid, plaintext: &[u8])
        -> Result<SealedValue, ShredError>;
    async fn open(&self, kek: &EncryptionKeys, subject_kind: &str, subject_id: Uuid, sealed: &SealedValue)
        -> Result<Option<Vec<u8>>, ShredError>;   // Ok(None) = DEK tombstoned (redacted); plaintext in Zeroizing
    async fn shred(&self, subject_kind: &str, subject_id: Uuid) -> Result<u64, ShredError>; // rows tombstoned
}

pub trait KeyHistoryProvider: Send + Sync {   // the registry calls this on a kid cache-miss (MR2)
    async fn public_key_pem(&self, program: Program, kid: &str) -> Result<Option<String>, KeyHistoryError>;
}
-- redaction_keys: per sealing service (persons, the 5 program services, security). One-way-tombstone trigger (Decision E).
CREATE TABLE redaction_keys (
    dek_id        UUID PRIMARY KEY,
    wrapped_dek   BYTEA NOT NULL,                 -- nonce||ct||tag of the DEK under the KEK (AAD-bound); zero-sentinel after shred
    kek_version   SMALLINT NOT NULL DEFAULT 1,    -- which KEK wrapped it (supports KEK-rotation re-wrap)
    subject_kind  TEXT NOT NULL,                  -- 'income_version' | 'ssn' | 'date_of_birth' | 'audit_event' | 'determination_snapshot' | ...
    subject_id    UUID NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    shredded_at   TIMESTAMPTZ                      -- NULL = live; non-NULL = tombstoned (redacted-at proof)
);
CREATE INDEX idx_redaction_keys_subject ON redaction_keys (subject_kind, subject_id);

-- signing_key_history: canopy-security only (Decision F). STRICTLY INSERT-only (append-only
-- trigger blocks UPDATE/DELETE/TRUNCATE) so a public key can never be silently swapped — a
-- tamper-evidence property. "Current vs. retired" is DERIVED from registration order (latest
-- registered_at per program = the active signer; older rows = retired), so no mutable retired_at
-- is needed. Registration is idempotent: INSERT ... ON CONFLICT (kid) DO NOTHING. MR2a.
CREATE TABLE signing_key_history (
    kid              TEXT PRIMARY KEY,            -- 'canopy-{program}-{YYYYqN}' (Decision G)
    program          TEXT NOT NULL,
    public_key_pem   TEXT NOT NULL,              -- SPKI PEM, PUBLIC key only
    registered_at    TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX idx_signing_key_history_program ON signing_key_history (program, registered_at DESC);

Redaction + access events (as-built: canopy-contracts-persons, MR9 — they are persons-published, so they live with the other persons event payloads, not in canopy-contracts-security), all plaintext-free:

  • FactRedactedEvent { person_id, kind, fact_id, author: Option<Author>, reason, redacted_at } — the as-built shape (the proposed subject_kind/subject_id/dek_ids were for the rejected fan-out; under the single-owner shared-DEK model the persons redaction expunges the audit copy directly, so no dek_ids travel and canopy-security records fact.redacted as an ordinary audit row — no subscriber/shred).

  • SsnRedactedEvent { person_id, author: Option<Author>, reason, redacted_at }.

  • SsnAccessedEvent { person_id, actor_sub, purpose: SsnAccessPurpose, source_service } — purpose is the enum { case_view, search, batch_lookup, foia, portability }, never free-text.

Implementation

Eight MRs + the (plan) commit under #687, sliced by dependency. Each commit builds green; per commit the pre-commit token gate + a fresh Explore subagent answering J1–J8 over the staged diff, reported as text. The (plan) commit (this .adoc + the nav entry) lands first in MR1. The seal/open of a value always happens inside the owning service’s store layer (never in an API handler), so unsealed PII never crosses a service boundary.

MR1 — Foundation crate + ADR-036

Files: crates/canopy-crypto-shred/** (new), crates/canopy-common/src/crypto.rs (AAD variants), root Cargo.toml (workspace member + cargo add hkdf zeroize), docs/…​/adrs/adr-029-crypto-shred-redaction.adoc (new), this plan .adoc + nav.adoc.

  • Add encrypt_with_aad/decrypt_with_aad to canopy_common::crypto; existing encrypt/decrypt delegate with empty AAD (SSN path byte-stable until MR8).

  • Define SealedValue/SealedDecimal/SealedJson, AAD-bound seal_*/open_*, DEK generation (OsRng → Zeroizing<[u8;32]>; open returns plaintext in Zeroizing), KEK-wrap/unwrap, the RedactionKeyStore + KeyHistoryProvider traits, typed ShredError/KeyHistoryError.

  • No service wiring, no migrations (the DDL is defined here but applied per-service later).

  • Proptests (mandatory): seal(plaintext) → hash(h1) → drop-DEK → hash(h2) → assert h1==h2 → open()==Ok(None); re-serialize byte-stability of a stored SealedValue; seal twice on equal plaintext → different ct; AAD-swap rejection (unwrap wrapped_dek_B under dek_id_A’s AAD → `Err); round-trip seal→open for arbitrary Decimal/JSON.

  • ADR-036 draft (Status Proposed): Decisions A–O condensed; an explicit Threat model section (Decision O residue vectors + the block-layer-encryption / bounded-retention prerequisites); cross-ref ADR-027 §8 / 028 / 014 / 017.

MR2 — JWS key-retention store + lazy-load (closes the retention half)

Files: services/canopy-security/migrations/<ts>_create_signing_key_history.sql, services/canopy-security/src/store/signing_keys.rs (new) + store/mod.rs, services/canopy-security/src/api/… (jwks handler + route), crates/canopy-signing/src/{verifier.rs,signer.rs}, services/canopy-eligibility/src/{config.rs,main.rs,orchestrator.rs,api/handlers.rs}.

  • signing_key_history table + append-only trigger + INSERT/by-(program,kid)/by-program reads; GET /v1/signing-keys/{program}/jwks (P-256 → JWK, incl. retired).

  • Stable program-bound kid (Decision G): signer embeds CANOPY_{PROGRAM}__SIGNING_KID, validated at startup; each program idempotently registers its current public key on boot.

  • VerifyingKeyRegistry: add provider: Option<Arc<dyn KeyHistoryProvider>>; make verify async (kid-extract → program-prefix check → in-memory → on-miss provider.public_key_pem(program, kid) → verify → cache). Thread .await through the orchestrator verify call sites (already async). Orchestrator injects an HTTP-backed provider hitting the jwks endpoint; unit tests pass None.

MR3 — Rip out _PREV env dual-key + runbook rewrite

Files: crates/canopy-signing/src/verifier.rs, docs/…​/security-operations.adoc, docs/…​/runbooks/signing-key-rotation.adoc, docs/…​/configuration-reference.adoc, docs/…​/deployment-guide.adoc.

  • Delete CANOPY_VERIFY_KEY_*_PREV loading from both registry loaders (each now holds only the current key). As-built (Decision I refinement): no in-memory second key is sourced from the store — the MR2b lazy-load already serves the rotation window, so _PREV is fully removed. RotationState / add_keys are kept as a programmatic escape hatch. Rewrite the standalone rotation runbook + the security-operations.adoc runbook section (delete the PREV slot + dual-key-window/30-day steps) and drop the stale _PREV rows from configuration-reference.adoc + deployment-guide.adoc. devstack_guard::ensure_signing_keys never set _PREV, so no devstack change. Subtractive — lands after MR2 bakes.

MR4 — Audit-chain v1 rip-out + drop hash_version

Files: services/canopy-security/src/store/{mod.rs,models.rs,fact_history.rs}, services/canopy-security/migrations/20260624130000_drop_audit_hash_version.sql; docs adrs/adr-014-fti-audit-hash-chain.adoc (Amendment 3), data-models/canopy-security.adoc, api/canopy-security.adoc, CHANGELOG.adoc. (As-built: fact_history.rs carried a hash_version: 2 test-fixture literal, and the J5 doc surfaces describe the column as live — both were under-specified in the original file list; added here per the living-spec rule.)

  • Collapse compute_event_hash to the single formula (delete compute_event_hash_v1 + the version dispatch + AuditHashError::UnknownVersion; rename compute_event_hash_v2 → compute_event_hash and AuditChainInputsV2 → AuditChainInputs — the "v2" suffix is vestigial with no v1); drop the hash_version column (audit_events + archive, one lock-step migration so the positional archive INSERT … SELECT keeps aligned ordinals) + the AuditEventRow model field + the test-fixture literal. Rewrite mixed_v1_v2_chain_verifies → multi_row_chain_verifies (pure single-formula), delete unknown_hash_version_breaks_chain (the version concept is gone), and fold the three v1-only unit tests into one sole-formula test. ADR-014 Amendment 3 + the J5 doc flips (the formula is unchanged — it *is the former v2 — so existing event_hash values verify unchanged; DB-only, no wire/OpenAPI delta). Independent rip-out; MR9 depends on it (event-metadata sealing relies on the single formula).

MR5 — Snapshot sealing contract + seal ALL programs (re-sliced)

Files: crates/canopy-crypto-shred/src/store.rs (add mint_dek to RedactionKeyStore); crates/canopy-contracts-eligibility/src/{snapshot.rs,derivation.rs} (+ Cargo.toml dep on canopy-crypto-shred); services/canopy-{snap,tanf,medicaid,caps,wic}/src/{determine.rs,store/…} (per-service RedactionKeyStore impl + redaction_keys migration with the one-way-tombstone trigger + builder wiring); contract
per-service tests. (The snapshot_hash-required + NoInputSnapshot-drop cascade — canopy-signing envelope, the 5 program DTOs, canopy-web, OpenAPI, the snapshot_hash NOT NULL migrations — is split to follow-up #911; see the status note above.)

  • Contract (canopy-contracts-eligibility). Value leaves → Sealed* per Decision L: IncomeFactLeaf.amount, AssetFactLeaf.value, ExpenseFactLeaf.amount → SealedDecimal; IevsReconstruction money (self_reported_monthly_income / verified_monthly_income / variance_monthly) → Option<SealedDecimal>; program_input + DerivedFactNode.value → SealedJson; cross_program_inputs → Option<SealedJson> (seal the whole SOLQ/FTI projection — the ADR-004-conservative choice; covers the SOLQ dollar amounts + the SSA flags/category/dates as one opaque blob). Leave plaintext: _type, frequency, relationship, household_size, MemberLeaf.date_of_birth (the *authoritative DOB is sealed at the persons table in MR8), all UUIDs/person_id`s, `corpus_hash, policy_params. The rescale(2) invariant lives inside SealedDecimal::seal.

  • SCHEMA_VERSION_MAX = 4 + SCHEMA_VERSION_MIN = 4; the builder emits schema_version = 4. verify_schema_version rejects > 4 (unknown-future, existing guard) and < 4 (a new PlaintextSchemaVersionRejected — a v1–v3 plaintext snapshot is a downgrade, refused so unsealed PII is never served/re-verified). This floor makes sealing sound on its own; the snapshot_hash-optionality tightening is the separable #911 follow-up.

  • Sealing (all 5 programs). mint_dek(kek, "determination_snapshot", determination_id) mints+wraps+persists ONE DEK per determination (Decision D), returns (Zeroizing<[u8;32]>, dek_id); the snapshot builder seals each value leaf with the sync SealedDecimal::seal / SealedJson::seal under that DEK, in the store layer (unsealed PII never crosses a service boundary). The KEK is the program’s existing CANOPY_ENCRYPTION_KEY (EncryptionKeys, ADR-017). FTI programs (tanf/medicaid): the program’s own KEK/DEK seals its FTI-bearing snapshot; the orchestrator receives only snapshot_hash + outcome, never ct/keys (ADR-004).

  • Orchestrator. Treats the snapshot as opaque for verification (re-hashes canonical_bytes → compares to the signed snapshot_hash); the sealed leaves are just ct strings, so no open() is needed to verify. No snapshot_hash-optionality change here (that is #911).

  • Tests: v4 round-trip (seal → canonical_hash stable across re-serialize → open() recovers the leaf); verify_schema_version rejects v3 (plaintext floor) and v5 (unknown future); a sealed snapshot’s wire JSON carries ct, never plaintext money; the per-program determine path seals + the signature verifies over the sealed bytes.

MR6 — Snap redaction op + CLI (reference impl)

Files: services/canopy-snap/src/{store/mod.rs,api/…} (the shred call + redact handler), tools/canopy-cli/src/cmd/… (the snap determination redact subcommand). (Sealing + the redaction_keys table/trigger already landed in MR5.)

  • POST /v1/determinations/{id}/redact (gated on the dedicated data_steward role, Decision M) → store.shred("determination_snapshot", id) + emit determination.redacted, both in one TX (ADR-018); request body RedactDeterminationRequest { reason } (blank → 400), response RedactDeterminationResponse { determination_id, redacted_at }; CLI canopy snap determination redact --id <id> --reason <reason>.

  • Headline integration tests (devstack): (1) sign → store sealed snapshot → redact (shred DEK) → re-read JSONB → canonical_hash == signed snapshot_hash → JWS still verifies → the redaction_keys row is tombstoned → a leaf open() → Ok(None). (2) no-blob-leak: the orchestrator’s SignableDetermination response carries only snapshot_hash (hex), never the snapshot blob. (3) role-gate (non-steward → 403), idempotent (re-redact → 0 rows, still 200), and unknown determination → 404.

NOTE
As-built deviations (MR6, 2026-06-24)

Two deviations from the plan above, recorded per the living-spec rule:

  1. :redact → /redact (sub-resource form, not the AIP-136 custom method). The endpoint is POST …/determinations/{id}/redact, mirroring …/{id}/resolve, not …/{id}:redact. axum/matchit 0.8 allows only one parameter per path segment, so a {id}:redact segment is unroutable. The same routing constraint applies to MR7’s /redact and MR8’s /redact//redact-ssn endpoints (updated above).

  2. A dedicated data_steward realm role was added (Claims::require_data_steward), mirroring fti_auditor: admins do not auto-hold it (separation of duties — admins grant/revoke it but do not themselves hold redaction authority). This is the concrete realization of Decision M’s "`canopy:redact`/data-steward role".

MR7 — Redaction op + CLI for tanf/medicaid/caps/wic

Files: services/canopy-{tanf,medicaid,caps,wic}/src/{store/…,api/…} (the shred call + redact handler); tools/canopy-cli parity. (Sealing + the per-service redaction_keys table/trigger already landed in MR5.)

  • Mechanically identical to MR6’s redaction op. FTI programs (tanf/medicaid) respect ADR-004: the redact op shreds the program-local DEK; the orchestrator/canopy-security never see FTI.

MR8 — Persons fact-value + persons-PII sealing + redaction + CLI

Files: services/canopy-persons/src/store/{income_versions,asset_versions,expense_versions,address_versions,persons}.rs, services/canopy-persons/migrations/<ts>_create_redaction_keys.sql (+ trigger)
<ts>_seal_fact_value_columns.sql, services/canopy-persons/src/api/…, tools/canopy-cli/src/cmd/{income,asset,expense,address,person}.rs.

  • SQL-aggregate audit gate (do FIRST): rg -n "SUM\(|WHERE\s+amount|WHERE\s+value|ORDER BY\s+amount" services/canopy-*/src across all five program services + persons; record per-service the result (expected: all value math is in-Rust post-read). Any SQL-side value math must move to Rust before its column is sealed; capture the table in the MR description.

  • Seal amount/value/employer_name/description/line_1/line_2 (column → BYTEA/JSONB SealedValue; DEK per version-row, subject_id = version_id); drop CHECK (amount>=0) (Decision N).

  • Seal persons-table PII: ssn (migrate off the direct-KEK ssn_encrypted to a per-person SealedValue, subject_kind='ssn', subject_id=person_id) + date_of_birth (subject_kind='date_of_birth', separate DEK so it redacts independently of SSN).

  • POST /v1/persons/{id}/facts/{kind}/{fact_id}/redact (shreds the fact’s version-row DEKs)
    …/redact-ssn (sub-resource form per the MR6 as-built note — the matchit-0.8 one-param-per-segment constraint applies); CLI canopy {income,asset,expense,address} redact <fact_id> + canopy person redact-ssn <id>.

MR9 (FINAL) — Event-value sealing + shared-DEK audit expungement + Pub-1075 + close

Files: crates/canopy-contracts-persons/src/events.rs (sealed event-value types + typed redaction/access events), crates/canopy-contracts-security/src/fact_history.rs ((sealed) display), services/canopy-persons/src/{events.rs,store/*_versions.rs,api/{mod,export}.rs} (seal in-store before publish + ssn.accessed emit), the ADR/data-model/CHANGELOG/master-plan/this-plan/status.

  • Persons seals each fact event’s PII before/after leaves in the store layer under the same per-fact DEK as the at-rest value (the envelope copied verbatim, never re-sealed — so the event ct shares the fact DEK), and publishes the fully-typed *ClaimedEvent; canopy-security stores them in metadata (the v2 hash covers ct unchanged — Decision J / ADR-014 Amendment 4).

  • As-built deviation from Decision M — no cross-service fan-out. Because the audit copy is sealed under the persons fact DEK, redacting the fact (one shred_with) expunges the at-rest and audit-ledger copies atomically. canopy-security needs no redaction_keys, no KEK, and no subscriber — a fan-out would require it to own a second value-key (only obtainable by receiving plaintext → violates ADR-004) and would make redaction delivery-dependent. The single-owner shared-DEK model is the architecturally-correct realization; the change-history shows (sealed) for value leaves (worker value display re-sourced from the SoR in follow-up #920).

  • ssn.accessed Pub-1075 event at each of the 7 persons_to_wire SSN-open sites, one per genuinely-decrypted person (enum purpose), fail-closed through the outbox. ADR-036 → Accepted
    ADR-014 Amendment 4 (ADR-028/017’s snapshot/SSN-at-rest surfaces were MR5/MR8, already cross-referenced from ADR-036 §Amends); the data-models/canopy-{persons,security}.adoc + api/ pages + cargo xtask api-docs --update (no OpenAPI delta — events/access are not HTTP shapes); CHANGELOG == Unreleased; master plan T2-6 → Done; this plan → Done + As-built. Closes #687.

Verification

Per MR: cargo build -p <touched>; cargo clippy -p <…> --all-targets — -D warnings; focused tests on the service’s dedicated postgres (set -a; source .ports.env; set +a; cargo nextest run -p <svc>); cargo xtask quality-budgets (the crypto Value`s in `canopy-crypto-shred / canopy-contracts-* are legitimately structural — mark per convention; OFFSET, never raise); cargo xtask check-docs + docs plan-lint; full pre-push battery (validate --skip-docker + Playwright e2e + cargo doc + k6
git-lfs) on every push; cargo xtask dev refresh before integration tests.

Load-bearing assertions:

  • MR1: seal → hash(h1) → drop-DEK → hash(h2) → h1==h2 → open()==Ok(None); re-serialize is byte-stable; double-seal differs; AAD-swap (open(dek_id_A, wrapped_dek_B)) → Err.

  • MR2: sign with kid-A → insert kid-B active + retire kid-A → drop kid-A from memory → verify an old kid-A determination → lazy-load from signing_key_history succeeds; a forged kid (canopy-tanf-… presented to a snap verify, or an unknown kid) is rejected without a fetch; JWKS returns retired keys.

  • MR4: post-rip, a chain over sealed-metadata events verifies; tampering a ct breaks it.

  • MR5/6 (mirrored MR7): headline — redact a determination → snapshot canonical_hash unchanged → JWS still verifies → leaf open()→Ok(None); no-blob-leak — orchestrator response carries only snapshot_hash; verify_schema_version rejects v3 + v5.

  • MR8: seal a fact value → as-of read opens it → redact → read returns redacted; SSN + DOB round-trip through the envelope and redact independently; the SQL-aggregate audit table is recorded.

  • MR9: the fact change-history is plaintext-free — a claim/correction/close renders wages · (sealed) · monthly and NEVER the figure (the integration test asserts the plaintext amount is absent + the marker present); the audit_events chain stays valid after a fact redaction shreds the shared per-fact DEK (the audit ct is byte-identical, only unopenable — no security-side shred); SSN open emits one plaintext-free ssn.accessed per decrypted person, a redacted SSN fires none; the event payload carries a SealedDecimal (ct present), never a plaintext amount.

Risks / sharp edges

  1. Non-deterministic nonce → seal once, never re-seal (Decision B). A correction appends a new version row (its own SealedValue); no read path re-seals.

  2. NUMERIC→BYTEA (Decision N): EXCLUDE safe; drop CHECK (amount>=0); the MR8 SQL-aggregate audit gate must pass before sealing.

  3. FTI / ADR-004 / Pub-1075: seal in the store (not the handler) so unsealed PII never crosses a boundary; orchestrator gets only outcome + snapshot_hash; canopy-security receives only sealed bytes + public keys; ssn.accessed/*.redacted are plaintext-free; purpose is an enum.

  4. Shred is application-layer (Decision O): wrapped-DEK plaintext can residue in WAL/backups/page-reuse/the KEK-previous window — ADR-036 states this + the block-layer-encryption
    bounded-retention prerequisites; a secure-overwrite sweep is a filed follow-up. Not overclaimed.

  5. Key-loss = data-loss (by design): KEK is the ADR-017 secret; KEK rotation re-wraps DEKs (kek_version, unwrap-old/wrap-new via the EncryptionKeys window) without re-sealing values; the role-gate + mandatory reason + the one-way-tombstone trigger guard accidental/malicious shred.

  6. Deploy ordering: MR2 before MR3; MR5 (verifier accepts required+v4) before MR6/7; MR9 after BOTH MR4 + MR8. Pre-1.0 + devstack re-seed keeps the window short.

  7. "No seal() on a read path" is a review-checklist item for MR5–MR9 (the catastrophic-failure trap of Decision B); plus a grep aid in each MR’s J1–J8 subagent prompt.

  8. Sealed serde_json::Value fields (program_input, DerivedFactNode.value) become opaque blobs — future materiality (T2-7) / overpayment (T2-8) readers must open() before comparing (those consumers aren’t built yet; noted in ADR-036).

Follow-ups

File each as a separate GitLab issue and /relate #687 before merging MR9:

  • KEK rotation runbook + xtask kek-rotate (the re-wrap path is designed in; tooling deferred).

  • Two-person-integrity enforcement for expungement (needs an approvals surface).

  • Post-grace secure-overwrite sweep for WAL/backup DEK residue (Decision O Tier-3).

  • Full DeterminationSnapshot ToSchema sweep so OpenAPI documents the sealed fields (the T2-1/T2-2-deferred item, now also covering SealedValue).

Edit this page · default