Plan: A8b — reporting least-privilege restricted DB role + credential cutover (#1456, epic &73)
On this page
- Status
- Context
- Scope
- Design
- Steps
- Step 0: File the fleet prerequisites
- Step 1: #1463 (P1, fleet) — relocate
idempotency_keysDDL to the migration path (both arms) - Step 2: P2 — additive
CANOPY_{SVC}__MIGRATION_DATABASE_URL - Step 3: #1464 (P3, fleet) — KEK preflight in the migrate job
- Step 4: Roles migration
- Step 5: Ownership transfer + grant matrix
- Step 6: SECURITY DEFINER janitor
- Step 7: Runtime boot guard
- Step 8: Devstack cutover
- Step 9: Real-login tests
- Step 10: Docs + closeout
- Files Touched
- Verification
- Documentation Updates
Child of ADR-004 reporting PHI tenancy (Step 2b). Implements the ADR-004 Amendment 1 A8 "least-privilege restricted DB role" storage control. Sibling of #1256 (A8a sealing — Done 2026-08-11). Not fleet-first: the owner/app split, fail-closed role reconcile, and SECURITY-DEFINER-only delete path all ship today in the chain-v2 substrate (canopy-security / canopy-medicaid / canopy-tanf) — this plan adopts that pattern for reporting’s ADR-004 tenancy.
Status
| Step | Description | Status |
|---|---|---|
0 |
File the fleet prerequisites (#1463 idempotency-DDL relocation, #1464 KEK preflight in the migrate job) + blocked-by links; commit this plan + nav; amend the parent Step-2b row; refresh #1456 ACs. |
Done (2026-08-12) — this MR |
1 |
#1463 (P1, fleet) — relocate the |
Done (2026-08-12) — #1463 |
2 |
P2 (rides #1456) — additive |
Done (2026-08-12) — #1456 (this MR) |
3 |
#1464 (P3, fleet) — KEK preflight in |
Done (2026-08-12) — #1464 |
4 |
Roles migration — |
Done (2026-08-12) — #1456 (this MR) |
5 |
Ownership transfer + grant-matrix migration — catalog-driven |
Done (2026-08-12) — #1456 (this MR) |
6 |
|
Done (2026-08-12) — #1456 (this MR) |
7 |
Runtime role-attribute boot guard ( |
Done (2026-08-12) — #1456 (this MR) |
8 |
Devstack cutover — roles in |
Done (2026-08-12) — #1456 (this MR) |
9 |
Real-login tests — grant matrix, immutability, guard rejection/override, battery/e2e as the restricted login. |
Done (2026-08-12) — #1456 (this MR) |
10 |
Docs — cutover/rollback runbook, the per-migration ownership-transfer standing convention, configuration reference, parent-plan closeout. |
Done (2026-08-12) — #1456 (this MR) |
Epic: &73
Issues: #1456 (this plan), #1463 (P1), #1464 (P3)
Branch: feature/1456-reporting-least-privilege-role (implementation; this plan rode feature/1456-a8b-canonical-plan)
Context
ADR-004 Amendment 1 A8 mandates that reporting’s restricted holdings — the sealed T-MSIS extract (#1256), its per-generation DEKs, the run substrate — be owned by a least-privilege restricted DB role, not the shared broad canopy role every devstack service connects as today (docker-compose.yml:919). canopy-reporting currently runs ONE DB role by design (services/canopy-reporting/migrations/20261101000001_report_runs.sql:38); with A8a sealing shipped, this role split is the last unshipped A8 storage control.
The fleet already ships this architecture. The chain-v2 substrate (services/canopy-security/migrations/20260910000000_chain_v2_substrate.sql, mirrored in canopy-medicaid/canopy-tanf) delivers the NOLOGIN owner + app split (:32, :52), a fail-closed reconcile refusing over-privileged roles (:59-73), create-then-transfer ownership (GRANT <owner> TO current_user :119; ALTER … OWNER TO :828-835), and REVOKE … FROM PUBLIC + an explicit per-object grant matrix (:838-876). 20260930000000_chain_append_staging.sql is the exact shape for this plan’s janitor: SELECT/INSERT granted (:72), column-restricted UPDATE (:80), no DELETE grant (:82), the only delete path a SECURITY DEFINER fn (:92-104). And services/canopy-caps/migrations/20260622000000_create_determination_snapshots.sql:35-37 records the load-bearing caveat that shapes everything here: a table-level REVOKE does not bind the table’s owner — immutability-by-grant-omission requires the runtime NOT to own the tables, which is why ownership transfers to a NOLOGIN role the runtime is not a member of.
Two fleet-wide "don’ts" from that precedent bind this plan (and corrected the parent plan’s original wording): no ALTER DEFAULT PRIVILEGES (zero occurrences fleet-wide; chain-v2 substrate — "explicit per-object v1-surface grants, never ALTER DEFAULT PRIVILEGES`"), and no `SET ROLE (it persists and would run sqlx’s _sqlx_migrations bookkeeping as the owner; handoff is ALTER OWNER + membership).
One discovery makes this bigger than reporting: idempotency_keys is the fleet’s only runtime DDL — created by app code on the runtime pool at crates/canopy-api/src/lib.rs:173 (IdempotencyCache::with_pool; DDL at idempotency.rs:337-409), defined in no .sql migration. Any service adopting a no-CREATE runtime role fails startup there first. That relocation (#1463) and the migrate-job KEK preflight (#1464 — the JobOwned path can currently run reporting’s destructive sealing migration with no KEK, stranding an unsealable service, despite 20261110000001:46-48 claiming otherwise) are fleet-shared canopy-api/xtask prerequisites, filed as their own blocking issues.
Scope
In scope (#1456):
-
canopy_reporting_owner(NOLOGIN) +canopy_reporting_app(LOGIN) with fail-closed attribute reconcile. -
One-time ownership transfer (catalog-driven
ALTER … OWNERloop) + the per-object grant matrix (REVOKE ALL FROM PUBLIC). -
SECURITY DEFINER reporting_janitor_reap— published-snapshot immutability by grant omission. -
Runtime role-attribute boot guard with the
CANOPY_REPORTING__ALLOW_BROAD_DB_ROLEaccountable override. -
Devstack real-login cutover; credential-cutover + rollback runbook; real-login grant-matrix/immutability/guard tests.
-
P2: additive
CANOPY_{SVC}__MIGRATION_DATABASE_URLbootstrap support (fleet-shared but additive/harmless-until-used, so it rides this issue).
Out of scope:
-
#1463 (P1)
idempotency_keysDDL relocation and #1464 (P3) migrate-job KEK preflight — separate fleet blocking issues (independently shippable; this plan’s Steps 1/3 describe them for context). -
#1303 orphan per-generation DEK reclamation (the janitor never touches
redaction_keys). -
#1459
report_run_universe.auxresidual; other services' least-privilege adoption (they reuse this pattern later).
Design
Role + credential architecture
-
canopy_reporting_owner—NOLOGIN; owns every reporting schema object after transfer.canopy_reporting_app—LOGIN; the runtime connection identity. Passwords never in source (Kerckhoffs): the dev app password is set in devstackinit.sql; prod passwords are secret-managed. Role creation isDO IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '…')-guarded (Postgres has noCREATE ROLE IF NOT EXISTS). -
Ownership transfer — the pre-existing-object subtlety (differs from chain-v2). chain-v2 only transfers objects it creates in the same migration (it owns them at creation). Reporting’s ~14 tables + 8 functions — the 7
report_run_*fns plus thecanopy_redaction_keys_one_way_guard()trigger fn (20261110000000_reporting_redaction_keys.sql:44) — already exist, owned by whoever historically applied migrations (devstack:canopy). (Corrected 2026-08-12, probe-proven: the originally ratifiedREASSIGN OWNED BY <old-owner>FAILS in the devstack —canopyis the pinned bootstrap superuser, and Postgres refusesREASSIGN OWNEDfor pinned roles: "cannot reassign ownership of objects owned by role canopy because they are required by the database system".) The cutover therefore uses a catalog-drivenDOloop —ALTER … OWNER TO canopy_reporting_ownerfor every relation in the schema (pg_class, relkindsr/p/v/m/S) and every function (pg_proc) — complete by construction (cannot miss the trigger fn), and working for any old owner including pinned superusers. Prereq before any transfer:GRANT USAGE, CREATE ON SCHEMA <schema> TO canopy_reporting_owner(clone20260910000000_chain_v2_substrate.sql:120-123; withoutCREATE ON SCHEMA, ownership transfer and subsequent object creation fail under a non-superuser migrator). -
Migrator identity.
ALTER … OWNERrequires the executor to own the object (or be a superuser) and to hold SET-membership in the new owner role. The one-time cutover migration therefore runs as the current object-owner / a superuser (devstack:canopyviaMIGRATION_DATABASE_URL; prod: the historical migrator identity or a superuser — runbook), afterGRANT canopy_reporting_owner TO current_user. Every subsequent reporting migration runs as the ongoing migrator (aCREATEROLEnon-superuser that is a member ofcanopy_reporting_owner): it creates objects ascurrent_user, thenALTER … OWNER TO canopy_reporting_owner+ grants to app. This per-migration transfer is a standing convention (Step 10) — the first migration that skips it silently regresses to migrator-owned objects with the app missing grants. The NOLOGIN owner never logs in; "owner = migrator identity" in #1456’s prose means objects are owned by the owner role, not that the owner is a login. -
Fail-closed attribute reconcile in the roles migration:
RAISE EXCEPTIONif either role carriesrolsuper/rolcreatedb/rolcreaterole/rolreplication/rolbypassrls(or the owner isrolcanlogin) — clone20260910000000_chain_v2_substrate.sql:59-73.
Grant matrix (enumerated from real runtime SQL — per-object, no ALTER DEFAULT PRIVILEGES)
REVOKE ALL ON ALL TABLES/FUNCTIONS IN SCHEMA … FROM PUBLIC; GRANT CONNECT ON DATABASE; GRANT USAGE, CREATE ON SCHEMA … TO canopy_reporting_owner (transfer prereq) + GRANT USAGE ON SCHEMA … TO canopy_reporting_app; pin ALTER ROLE canopy_reporting_app SET search_path. No sequence grants exist to make — every PK is a UUID (uuidv7()/gen_random_uuid()); no serial/sequence/nextval anywhere in the reporting migrations.
| Table | S | I | U | D | Evidence / note |
|---|---|---|---|---|---|
|
✓ |
✓ |
INS |
||
|
✓ |
✓ |
INS |
||
|
✓ |
✓ |
INS |
||
|
✓ |
✓ |
INS |
||
|
✓ |
✓ |
INS |
||
|
✓ |
✓ |
|
||
|
✓ |
✓ |
|
||
|
✓ |
✓ |
✓ |
UPSERT |
|
|
✓ |
✓ |
✓ |
permanent provenance — never runtime-DELETEd; UPDATEs: pins, supersede, publish, the #1462 abandons |
|
|
✓ |
✓ |
✓ |
✓ |
DML almost entirely via the SECURITY-INVOKER fns; DELETE = |
|
✓ |
✓ |
INS |
||
|
✓ |
✓ |
get-or-create INSERT + live-DEK SELECT ( |
||
|
✓ |
✓ |
✓ |
the shared |
|
|
no grant — no subscriber registered, no |
GRANT EXECUTE on the 7 report_run_* plpgsql fns (enqueue/claim/checkpoint/heartbeat/abandon/finalize/reap). Load-bearing: all seven are SECURITY INVOKER — their bodies run as the caller, so EXECUTE alone is insufficient; the app also needs the underlying report_runs/report_generations DML enumerated above (which it has). Built-ins (uuidv7, gen_random_uuid, pg_advisory_xact_lock) are PUBLIC-executable by default. The canopy_redaction_keys_one_way_guard() trigger fn needs no app EXECUTE (trigger firing bypasses EXECUTE checks) but must transfer ownership with everything else.
Published-snapshot immutability — SECURITY DEFINER reporting_janitor_reap
Today the generation janitor is a Rust loop of unguarded direct DELETEs (worker/mod.rs:324-337 over JANITOR_TABLES — the 5 output tables + report_run_universe) trusting a candidate list from a separate SELECT (janitor_candidates, mod.rs:498-529). Under least privilege the runtime holds no direct DELETE on those six tables; the delete moves behind a SECURITY DEFINER fn owned by canopy_reporting_owner:
-
reporting_janitor_reap(p_generation_id uuid, p_retention_days int) RETURNS (output_rows bigint, universe_rows bigint),SET search_path = <schema>, pg_temp;REVOKE EXECUTE FROM PUBLIC+GRANT EXECUTE TO canopy_reporting_app(thechain_staging_dequeueshape,20260930000000:92-104). -
Re-verifies eligibility inside the function — the immutability guard must not trust the caller:
SELECT state … FOR SHARE; proceed only ifabandonedpastcreated_at + retentionORsupersededwith a qualifying successor past retention (replicatemod.rs:505-512); astaged/publishedgeneration is unreapable by construction (return 0). Then the sixDELETE FROM <t> WHERE generation_id = p_generation_id, tallying viaGET DIAGNOSTICS. -
The FKs to
report_generationshave noON DELETE CASCADEand generation rows are permanent, so the fn deletes each child table explicitly.janitor_candidatesstays as the SELECT-only batch selector; the Rust loop callsSELECT * FROM reporting_janitor_reap($1, $2)per candidate.report_run_reap(already a guarded SQL fn) is kept as-is. PreserveReapSummary.output_rows_deleted/universe_rows_deleted.
Runtime boot guard (the one genuinely-new layer)
No runtime DB-role guard exists anywhere in the fleet (the rejection logic ships only migration-side, in the chain-v2 reconcile). Clone the #1006 guard structure from services/canopy-applications/src/guard.rs:
-
RolePrivilegeVerdict { DevAllowed, OverrideAllowed, Refused }; a pureevaluate(env: &str, is_overbroad: bool, allow_broad: bool) → Verdict(no I/O — the whole decision table unit-tested); fail-closedresolve_env(unsetCANOPY_ENV⇒production). -
A thin
require_least_privilege_role(pool, allow_broad)wrapper runs the SQL probe outside the pure fn —SELECT rolsuper OR rolcreatedb OR rolcreaterole OR rolreplication OR rolbypassrls FROM pg_roles WHERE rolname = current_userplus acanopy_reporting_app-membership check (pg_has_role) — passes the boolean in,tracing::warn!`s the auditable line on `OverrideAllowed(naming the runbook), andErr`s on `Refusednaming the hazard + theCANOPY_REPORTING__ALLOW_BROAD_DB_ROLEoverride. (pg_rolesis world-readable — a non-superuser can probe its own attributes; onlyrolpasswordis masked.) -
Timing: after the runtime pool is built (
bootstrap.rs:121) and before serving; the elevated migration pool has already closed (bootstrap.rs:309), so the probe reads the runtime credential. Config:allow_broad_db_role: boolon the reporting config.
Steps
Step 0: File the fleet prerequisites
Files: none (GitLab + this plan)
Done (2026-08-12, this MR): #1463 (P1) + #1464 (P3) filed with full ACs; blocks links to #1456 set; parent Step-2b row amended; #1456 ACs refreshed.
Step 1: #1463 (P1, fleet) — relocate idempotency_keys DDL to the migration path (both arms)
Files: crates/canopy-api/src/idempotency.rs, crates/canopy-api/src/bootstrap.rs, crates/canopy-api/src/lib.rs, xtask/src/cmd/migrate.rs
Delivered under #1463 (see the issue for full ACs). Extract the DDL (CREATE_TABLE_DDL + EXPAND_MIGRATION_DDL, idempotency.rs:337-409) into a standalone fn; split IdempotencyCache::with_pool (:550-644) into that DDL part + a runtime part (metrics + cleanup task + renewal pool) that assumes the table exists; lib.rs:173 becomes DDL-free. The DDL fn is invoked from both migration arms: (a) bootstrap.rs migration_phase on the migration pool before it closes (:296-311), and (b) xtask/src/cmd/migrate.rs::apply after migrator.run (:200-205) — arm (b) is load-bearing because under SKIP_MIGRATIONS/JobOwned, migration_phase no-ops (bootstrap.rs:289-295); the chain-migration-split compose profile (dormant until the #1279 cutover, docker-compose.yml:1676-1728) models canopy-security/tanf/medicaid exactly that way whenever invoked, and reporting joins them at Step 8. The idempotency.rs:544-549 "no sqlx::migrate!`" constraint is respected — the DDL runs on the migration pool/job, never inside `_sqlx_migrations.
Step 2: P2 — additive CANOPY_{SVC}__MIGRATION_DATABASE_URL
Files: crates/canopy-common/src/settings.rs, crates/canopy-api/src/bootstrap.rs
Add migration_database_url: Option<String> beside database_url (settings.rs:19), read through the secret provider with the optional-secret pattern (settings.rs:222-229). bootstrap.rs:119 runs migration_phase against migration_database_url.as_deref().unwrap_or(&database_url); the runtime pool (:121) stays on database_url. Run validate_database_name + the TLS gate on the effective migration URL as well. Fallback-to-runtime-URL is the correct default for the ~18 services that do not split credentials; a mis-set reporting deployment fails loud (permission denied on the first DDL), not silent.
Step 3: #1464 (P3, fleet) — KEK preflight in the migrate job
Files: xtask/src/cmd/migrate.rs
Delivered under #1464 (see the issue for full ACs). In migrate.rs::apply, before any pool opens: canopy_crypto_shred::require_kek("CANOPY_ENCRYPTION_KEY")? gated on the sealing-service set {reporting, persons, snap, tanf, medicaid, caps, wic}. Closes the JobOwned hole the sealing migration’s 20261110000001:46-48 comment wrongly assumed shut. (Deviation from the original step: that migration comment itself is NOT byte-edited — sqlx checksums applied migrations, so any edit breaks every existing database’s version validation. Post-#1464 the comment’s conclusion — "this reset cannot commit and then strand an unsealable service" — is actually true via both gates; the corrected two-gate claim lives in the preflight’s own comment in migrate.rs.)
Step 4: Roles migration
Files: services/canopy-reporting/migrations/<ts>_least_privilege_roles.sql
DO-guarded CREATE ROLE canopy_reporting_owner NOLOGIN + CREATE ROLE canopy_reporting_app LOGIN (no password in source); the fail-closed attribute reconcile (clone 20260910000000_chain_v2_substrate.sql:59-73); GRANT canopy_reporting_owner TO current_user (the migrator’s SET-membership for the transfer).
Step 5: Ownership transfer + grant matrix
Files: same migration as Step 4, or a sibling <ts>_least_privilege_grants.sql (ordering: roles → schema grants → transfer → object grants)
GRANT USAGE, CREATE ON SCHEMA … TO canopy_reporting_owner; the one-time catalog-driven ownership-transfer loop (pg_class relkinds r/p/v/m/S + pg_proc, schema-scoped, ALTER … OWNER TO canopy_reporting_owner — the ratified REASSIGN OWNED fails on the pinned devstack superuser, see Design); REVOKE ALL … FROM PUBLIC; the per-object grant matrix + fn EXECUTEs from the Design table; GRANT USAGE ON SCHEMA … TO canopy_reporting_app; ALTER ROLE canopy_reporting_app SET search_path.
Step 6: SECURITY DEFINER janitor
Files: services/canopy-reporting/migrations/<ts>_janitor_security_definer.sql, services/canopy-reporting/src/worker/mod.rs
Create reporting_janitor_reap per the Design; refactor reap_and_janitor (worker/mod.rs:310-339) to call it per candidate; the app role receives no direct DELETE on the six tables (grant omission in Step 5 + this fn is the only path).
Step 7: Runtime boot guard
Files: services/canopy-reporting/src/guard.rs (new), services/canopy-reporting/src/config.rs, services/canopy-reporting/src/main.rs
Clone services/canopy-applications/src/guard.rs per the Design. Call site: after bootstrap() returns the runtime pool, before serving.
Step 8: Devstack cutover
Files: devstack/postgres/init.sql, docker-compose.yml
Roles + dev app password in init.sql behind DO-block guards (created as devstack superuser canopy). Init scripts run only on empty PGDATA, so adoption requires a one-time cargo xtask dev destructive volume reset (already blessed pre-1.0 by the parent’s fresh-start reset) — documented; no new post-start provisioning path. Flip docker-compose.yml:919 to postgres://canopy_reporting_app:…@postgres:5432/canopy_reporting; migrations run as owner via CANOPY_REPORTING__MIGRATION_DATABASE_URL (Step 2) or a canopy-reporting-migrate one-shot modeled on the chain-migration-split services (docker-compose.yml:1690-1704; note that profile is dormant until #1279 and its rehearsal carries the #1372 depends_on caveat — verify both when wiring the one-shot).
Step 9: Real-login tests
Files: services/canopy-reporting/tests/least_privilege_role_test.rs (new), e2e config
Grant-matrix proofs over a dedicated canopy_reporting_app connection (no pooled SET ROLE): the app runs every service flow; cannot UPDATE extracts; cannot DELETE outputs directly; cannot DDL; cannot reap a staged/published generation through reporting_janitor_reap. Guard rejection: superuser and broad-membership sessions refused outside development without the override; the override WARNs and proceeds. The full battery + e2e run reporting AS the restricted login (Step 8 makes this the devstack default). Mirror services/canopy-tanf/tests/chain_v2_substrate_test.rs:1220-1242.
Step 10: Docs + closeout
Files: docs/modules/ROOT/pages/security-operations.adoc, configuration-reference.adoc, deployment-guide.adoc, coding-conventions.adoc (overlay), the parent plan, CHANGELOG.adoc
Credential-cutover + rollback runbook (the one-time transfer-as-current-owner/superuser note; the ongoing CREATEROLE non-superuser migrator; app-password provisioning). The standing convention: every future reporting migration must ALTER … OWNER TO canopy_reporting_owner + grant the app its new objects — recorded in the coding-conventions overlay, not just prose, else the first unaware migration silently regresses privileges. Configuration reference: MIGRATION_DATABASE_URL, ALLOW_BROAD_DB_ROLE. Parent Step-2b row → Done; this plan → Archive.
Files Touched
| File | Change |
|---|---|
|
this plan + its nav entry (Scale Readiness, epic &73) — this MR |
|
Step-2b row → Blocked(#1463/#1464) + child pointer; "default ACLs pinned" → enumerate-per-object — this MR |
|
#1463 (P1) + #1464 (P3) + P2 — the fleet prerequisites |
|
roles + reconcile; REASSIGN + grant matrix; SECURITY DEFINER janitor |
|
boot guard; janitor call-path refactor |
|
role provisioning; runtime-URL cutover; migrate identity |
|
real-login grant-matrix / immutability / guard battery |
Verification
-
cargo xtask plan-lint+cargo xtask check-docsclean; the Antora build resolves every xref (this MR). -
Implementation MRs:
cargo nextest run -p canopy-reportinggreen ascanopy_reporting_app(the devstack default after Step 8). -
The Step-9 negative proofs: UPDATE extract →
permission denied; direct DELETE output →permission denied; DDL →permission denied;reporting_janitor_reapon apublishedgeneration → 0 rows / refusal. -
Guard: superuser session outside development → boot refused naming the override; with
CANOPY_REPORTING__ALLOW_BROAD_DB_ROLE=true→ boots with the WARN line. -
Full pre-push battery +
cargo xtask e2egreen with reporting on the restricted login end-to-end.
Documentation Updates
-
Antora canonical docs —
security-operations.adoc(cutover/rollback runbook),configuration-reference.adoc(MIGRATION_DATABASE_URL,ALLOW_BROAD_DB_ROLE),deployment-guide.adoc(role provisioning), the coding-conventions overlay (the per-migration ownership-transfer standing convention). -
CHANGELOG.adoc— entry under== Unreleased(implementation MRs). -
Parent plan Step-2b row kept current at each step; this plan → Archive + nav update on completion.