ADR-022: Composition Override Storage Layering

On this page
NOTE
Amended by ADR-024 (user-layer semantic delta schema).

Context

ADR-021 defines the composition runtime as a five-layer top-wins resolver: user delta → role override → jurisdiction live override → jurisdiction TOML baseline → system defaults. Three of those layers (user, role, jurisdiction live) are DB-backed and need a storage schema; the other two (jurisdiction baseline TOML on disk, system defaults compiled into canopy-web) need none.

This ADR defines the DB schema for the three DB-backed layers, the override merge semantics, the live-override lifecycle (create → edit → promote → archive), and the audit retention policy for override-layer events. It is Stage-2 ADR ratification 2 of 3 for epic &51 (#460).

It does not define the composition loader runtime (ratified in ADR-021) or the promote-live-to-baseline mechanism (originally scoped as ADR-023; deferred 2026-05-20 in favor of #507, a broader unified config-backend ADR across canopy domains).

Options considered

DB schema shape

Option B: Unified composition_documents table (selected)

One table covers all three DB-backed layers. Each row carries (jurisdiction_id, layer, scope_key, surface) as a composite key + a JSONB body containing the RFC 6902 patch ops. layer is a PostgreSQL enum ('user' | 'role' | 'jurisdiction_live'); scope_key is polymorphic by layer (user ID for user, role slug for role, sentinel string "jurisdiction" for jurisdiction_live).

  • Pros: one table, one set of migrations, one set of query patterns; the composition loader does one indexed SQL query per render to fetch ALL DB-backed layers (WHERE jurisdiction_id = $1 AND surface = $2 AND (layer = 'jurisdiction_live' OR (layer = 'role' AND scope_key = $3) OR (layer = 'user' AND scope_key = $4))); adding a future layer (e.g., team-level overrides) is a schema-stable enum addition + a loader update, not a new table.

  • Cons: less constraint expression at the schema level — scope_key cannot FK to users.id for user rows because the column is polymorphic. Mitigation: application-layer validation in the write API enforces the relationship (write API rejects a user write if the scope_key doesn’t resolve to an active user); row count is slightly larger than partitioned tables but well below indexed query cost concerns.

Option A: Per-layer tables (rejected)

Three tables: user_compositions, role_compositions, jurisdiction_live_compositions. Each row keyed by (scope_key, surface, …) for its layer.

  • Rejected. Clearer schema-level constraints (user_compositions.scope_key can FK to users.id) but the composition loader has to issue 3 queries per render (or one UNION ALL query), and adding a new layer is a new migration + new query path. The trade — schema strictness for runtime + migration complexity — doesn’t favor Option A given the application-layer validation already needed at the write API.

Merge semantics

Option C: RFC 6902 JSON Patch (selected)

Each override carries a list of patch operations (add / remove / replace / move / copy / test) at JSON Pointer paths. Studio writes to an override surface accumulate ops; the composition loader replays the patch list against the baseline document (or the previously-merged layer document) to produce the merged document.

  • Pros: Studio "add one panel" is one {"op": "add", "path": "/items/-", "value": {…}} op (not a full document rewrite); diff-friendly storage (smaller rows, clear edit history); JSON Pointer paths debug-readably; supports single-element array removal (which RFC 7396 cannot); the test op lets Studio implement optimistic concurrency (refuse a write if the underlying baseline shape moved out from under it).

  • Cons: more implementation complexity than full-document replace; Studio UI has to model patch ops (likely via a hidden current ops JSON view + the user-facing visual editor); debugging "why did this panel disappear" requires replaying the op list. Mitigation: every write API call persists the post-merge document alongside the patch list in an audit row, so debugging walks the audit history rather than replaying ops manually.

Option B: RFC 7396 JSON Merge Patch (rejected)

Recursive shallow merge: object values merge recursively, null values delete, arrays + scalars replace.

  • Rejected. Familiar but cannot remove a single element from an array without rewriting the entire array. The canonical Studio operation is "add or remove a panel from the dashboard"; RFC 7396 forces full-array rewrites for either op. Loses the diff-friendly storage benefit.

Option A: Full document replace (rejected)

Higher layer wins entirely; lower layers ignored for that surface.

  • Rejected. Every override duplicates the entire baseline composition. Studio "add one panel" becomes "rewrite the full composition with the panel added"; the storage layer becomes a noisy duplicate of baselines.

Override lifecycle

Option B: Explicit archive in Studio (selected)

Live override stays in place after the v1 "promote" affordance completes (originally scoped to ADR-023; deferred to #507's unified config backend). In v1, the "promote" affordance is admin-driven: admin edits the jurisdiction’s TOML baseline directly via their existing workflow (PR / Salt / manual edit) external to canopy. Studio surfaces a "live override matches baseline" hint (computed by comparing the patched composition against the post-refresh baseline). The jurisdiction admin clicks "Archive" in Studio to move the live override row to composition_documents_archive (separate table, same shape + archived_at timestamp).

  • Pros: admin owns the lifecycle moment (no spurious archives if the PR didn’t actually contain what was expected); no canopy-core-repo watcher required (which would be the alternative for auto-archive); the post-merge "live matches baseline" state is a no-op at render time anyway — the patch ops resolve to the same merged document — so leaving it in place until explicit archive is correctness-safe.

  • Cons: admins might forget to archive; live overrides accumulate as no-ops. Mitigation: once a baseline refresh detects "live override matches baseline", Studio surfaces a prompt offering "Archive this live override?" with both an "Archive" and a "Keep" button — no promote-PR mechanism in v1, so the prompt fires on the next composition load after the admin’s external baseline edit lands and the loader picks it up.

Option A: Auto-archive on promote-merge (rejected)

Background watcher on the canopy-core repo. When the PR merges, the watcher archives the corresponding live override row.

  • Rejected for v1. Requires the canopy-core webhook + correlation logic between the PR’s commit subject and the live override row. Complex enough to be its own ADR; not necessary if Option B’s UX nudge handles the lifecycle.

Option C: Leave live overrides in place forever (rejected)

  • Rejected. Same correctness as Option B (the post-merge state is a no-op) but accumulates rows indefinitely. The archive-on-explicit-action mechanism is one click; not having it produces a graveyard.

Audit retention

Option A: Uniform 1-year retention for all override-layer audit events (selected)

Every override-layer write (create / edit / archive / promote) emits a JWS-signed AuditEvent per ADR-014. Retention is uniform 1 year for all override layers.

  • Pros: simple; no per-layer policy to maintain; aligns with the existing auth-events retention policy (also 1 year per CLAUDE.md’s broader retention strategy).

  • Cons: doesn’t distinguish baseline edits (rarer, more consequential) from user-delta edits (frequent, transient). Mitigation: revisit if a jurisdiction asks; the retention is a config value, not schema-encoded.

Option B: Per-layer retention (rejected)

7yr baseline / 90d live / 30d user.

  • Rejected for v1. No external regulation forcing per-layer retention; no use case yet from a jurisdiction; the per-layer policy adds config complexity. Revisit if Georgia or another jurisdiction asks.

Decision

Unified composition_documents table + RFC 6902 JSON Patch merge semantics + explicit Studio archive + uniform 1-year audit retention.

Schema

-- Forward-only migration per ADR-016.
CREATE TYPE composition_layer AS ENUM ('user', 'role', 'jurisdiction_live');

CREATE TYPE composition_surface AS ENUM (
    'worker_dashboard',
    'supervisor_dashboard',
    'analyst_dashboard',
    'case_detail',
    'sign_in'
);

CREATE TABLE composition_documents (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    jurisdiction_id UUID NOT NULL,
    layer        composition_layer NOT NULL,
    scope_key    TEXT NOT NULL,           -- user ID, role slug, or 'jurisdiction' sentinel
    surface      composition_surface NOT NULL,
    patch_ops    JSONB NOT NULL,          -- RFC 6902 operation list
    created_at   TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    created_by   UUID NOT NULL,           -- user who created
    UNIQUE (jurisdiction_id, layer, scope_key, surface)
);

CREATE INDEX composition_documents_lookup_idx
    ON composition_documents (jurisdiction_id, surface, layer, scope_key);

CREATE TABLE composition_documents_archive (
    LIKE composition_documents INCLUDING DEFAULTS INCLUDING IDENTITY,
    archived_at  TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    archived_by  UUID NOT NULL
);

CREATE INDEX composition_documents_archive_lookup_idx
    ON composition_documents_archive (jurisdiction_id, surface, archived_at DESC);
NOTE
The archive table deliberately copies columns + defaults + identity ONLY (not constraints or indexes) via LIKE …​ INCLUDING DEFAULTS INCLUDING IDENTITY. Inheriting INCLUDING ALL would copy the active table’s UNIQUE (jurisdiction_id, layer, scope_key, surface) constraint, which would forbid archiving the same composition tuple more than once over a jurisdiction’s lifetime — that conflicts with audit/history retention (a jurisdiction can create, archive, re-create, archive again any number of times). The archive table is append-only history; its only index is the lookup index above (jurisdiction + surface + archived_at DESC) to support audit-trail queries.

scope_key semantics by layer:

Layer scope_key value Notes

user

UUID of the user as TEXT

App-layer validates the user exists + is active

role

Role slug from idp.toml (e.g., "eligibility_worker")

App-layer validates the role exists in the jurisdiction’s idp.toml

jurisdiction_live

Literal 'jurisdiction'

Only one row per (jurisdiction_id, surface) for this layer

Loader query

The composition loader (ADR-021’s load_composition) fetches all three DB-backed layers for a given (jurisdiction, role, user_id, surface) request in one query:

SELECT layer, scope_key, patch_ops
  FROM composition_documents
 WHERE jurisdiction_id = $1
   AND surface = $2
   AND (
       layer = 'jurisdiction_live'
       OR (layer = 'role' AND scope_key = $3)
       OR (layer = 'user' AND scope_key = $4)
   )
 ORDER BY CASE layer
            WHEN 'jurisdiction_live' THEN 1
            WHEN 'role'              THEN 2
            WHEN 'user'              THEN 3
          END;

The loader composes all five ADR-021 layers (system defaults → jurisdiction TOML baseline → jurisdiction live → role → user delta) into a single merged document.

Merge semantics

The composition uses two merge styles because the layers have different shapes:

  • System defaults → jurisdiction baseline: RFC 7396 JSON Merge Patch semantics. The baseline TOML is a structural overlay — keys it declares replace the defaults; keys it omits fall through to the defaults; null in the baseline removes a defaults key. This is the right shape because baselines are partial documents (a jurisdiction needs to declare only what differs from canopy core’s defaults) and they’re authored as TOML (not as patch ops).

  • Baseline → DB-backed layers (jurisdiction live, role, user): RFC 6902 JSON Patch op lists per the Decision 2 rationale above. The DB-backed layers are precise operations (add/remove/replace/move/copy/test) that Studio authors directly.

Pseudocode:

// 1. System defaults — compiled into canopy core (panel/section registry initial state).
let mut merged: serde_json::Value = system_defaults.clone();

// 2. Jurisdiction baseline — TOML loaded from `rulesets/{juris}/composition/{surface}.toml`,
//    cached per (jurisdiction, surface) via ADR-021's invalidate-on-write cache,
//    deserialized to JSON, applied as RFC 7396 merge patch.
apply_merge_patch_7396(&mut merged, &baseline_document);

// 3-5. DB-backed layers — RFC 6902 op lists, applied in
//      jurisdiction_live → role → user precedence (lowest first).
for (layer, _scope_key, patch_ops) in db_layers {
    json_patch::patch(&mut merged, &patch_ops)
        .map_err(|e| CompositionLoadError::PatchFailed { layer, error: e })?;
}

The json-patch crate (RFC 6902) returns an error if any operation’s test op fails or a referenced path doesn’t exist — surfaced as CompositionLoadError::PatchFailed to Studio with the offending layer + op index. The RFC 7396 baseline overlay cannot fail in this way (merge-patch is total over its inputs).

NOTE
The merge between system defaults and jurisdiction baseline uses RFC 7396 deliberately even though Decision 2 above rejected 7396 for the DB-backed layers — that rejection cited 7396’s inability to remove single array elements, which is a critical operation for Studio-driven editing of overrides but not for baseline authoring (a baseline that needs to remove a specific defaults panel can replace the entire panel array). The two layers carry different semantics because they have different mutation surfaces.

Write API contract

The Stage-3 live override APIs (#491) accept patch ops directly (no document-level diff inference):

PATCH /v1/composition/{surface}/live
PATCH /v1/composition/{surface}/role/{role}
PATCH /v1/composition/{surface}/user/me
Content-Type: application/json-patch+json

[
  { "op": "add", "path": "/items/-", "value": { "slug": "snap-overpayment-summary", "span": 4 } },
  { "op": "test", "path": "/items/0/slug", "value": "household-summary" }
]

Server-side, the API merges the incoming ops with the existing patch_ops row (appending new ops to the list, or — for an existing user/role override — replacing the full list per a If-Match: <etag> header). Returns the resolved merged document + new ETag.

Consequences

Positive

  1. Schema-stable for additional layers. A future "team-level overrides" layer is an enum addition + a loader change — no new table, no new query path.

  2. One indexed query per render to fetch all DB-backed layers for a composition. Composition loader latency is dominated by the patch replay, not the SQL roundtrip.

  3. Diff-friendly storage. Patch ops are small; an override row carries only the delta, not the full baseline.

  4. Studio UX naturally maps to ops. "Add panel", "remove panel", "resize panel span", "reorder panels" each correspond to one or two RFC 6902 ops. The Studio backend doesn’t have to diff documents to produce a write.

  5. Optimistic concurrency via test ops. Studio can refuse a stale write (e.g., if the baseline shape changed mid-edit) by including a test op in the patch list.

  6. Override-layer audit aligns with ADR-014 hash chain. Every write emits a JWS-signed AuditEvent with previous_hash / event_hash per ADR-014 — chain integrity extends across composition mutations without special-casing.

Negative

  1. Polymorphic scope_key cannot FK to a specific table. Application-layer validation enforces user/role existence at write time. Mitigation: the write API’s validation is a single resolve call against the same JurisdictionRegistry the loader uses.

  2. Patch op debugging is replay-based. When a worker reports "the appeals panel disappeared from my dashboard", the debug path is: load the merged document at the time of the report → walk the audit history backward → identify the op that removed the panel. The audit row carries the post-merge document snapshot for each write, so replay is just a git log-style walk.

  3. No automatic archive on promote. Live overrides that became no-ops after a successful promote stay in composition_documents until the admin clicks Archive in Studio. The "live matches baseline" hint surfaces the prompt; mitigation is UX, not runtime.

  4. test ops can produce non-obvious write failures. If a Studio session has been open long enough for the baseline to shift, a write may fail with PatchFailed. The Studio modal exposes the failure with a "refresh baseline and retry" affordance.

Implementation

Tracked under Stage 3 of epic &51 (#460):

  • #489 — DB migrations for composition override layers (this ADR’s schema). Forward-only per ADR-016.

  • #490 — Composition loader (consumes this ADR’s loader query + merge semantics).

  • #491 — Live override APIs (consumes this ADR’s write API contract).

  • ~~#492 — Promote-live-to-baseline~~ closed-deferred 2026-05-20; the promote-live-to-baseline mechanism is deferred to #507 (unified config backend across canopy).

The json-patch Rust crate is the canonical implementation; v1 pins to the latest stable version. The crate handles RFC 6902 semantics including JSON Pointer escaping.

References

Edit this page · default