Plan: Single-flight Idempotency-Key execution across replicas

On this page

Status

MR Description Status

MR1

Actor-scoped, canonical, validated cache key (extract_cache_key rewrite; closes the cross-actor auth-bypass; fingerprint + key-validation). Ships on the current middleware.

Done (2026-07-11) — !807

MR2

Expand phase: additive nullable schema + forward-compatible reader that fails-closed (503) on any non-replayable row; cleanup first-tick stagger; ApiError::ServiceUnavailable. Deploy + fully roll out before MR3.

Done (2026-07-11) — !809

MR3

Core single-flight: claim CTE + renewed/fenced lease + body-hash conflict + follower wait + crash recovery + status matrix + header allowlist + configurable response cap + 5xx failure-cooldown + two metric counters + validated SingleFlightConfig + cleanup race fix.

Done (2026-07-11) — !810

MR4

Contract/observability + docs: global utoipa::Modify responses (409/413/503) applied centrally in ApiServer::router (see the MR4 note below — this supersedes the per-service wiring originally scoped here) + api-docs --update; doc + CHANGELOG updates; archive this plan. Carries Closes #1003.

Done (2026-07-11) — final MR

Epic: &44 (Security, CI/CD & Documentation Remediation)
Umbrella issue: #1003
Child issues: #1027 (MR1), #1028 (MR2), #1029 (MR3), #1030 (MR4) — each Relates to #1003; only MR4 Closes #1003
Branches: feature/1003-idempotency-*

Context

crates/canopy-api/src/idempotency.rs is a middleware every Canopy service mounts (ApiServer::router, lib.rs:117-132) to stop a client retry from double-issuing benefits / notices / audit-chain writes (lib.rs:98-107). Under concurrency it does not. The hot path is check-then-execute: SELECT (miss = no row, :411-430) → run the handler (next.run, :451) → INSERT …​ ON CONFLICT DO NOTHING (:482-497). Two overlapping same-key POSTs (a retry racing a slow first request; across replicas both read the same empty row) both miss and both run the side effects; ON CONFLICT DO NOTHING only picks which response is stored, not which domain writes happen. Observed at commit 0fa79f7a.

Goal: run the handler once per key (happy path), one replica and across replicas sharing the service DB, with defined follower/conflict/crash-recovery behavior and no DB transaction held across the handler.

This is an epic, not one MR (same finding class as #1004). External review established that a correct fix also needs an enforceable liveness mechanism (a fixed 60s lease proves nothing — reporting/eligibility POSTs run minutes), actor-scoped cache identity (a live cross-actor authorization bypass), rolling-deploy safety on a compliance-sensitive shared table, a response-header allowlist, and an RFC 9457 error contract.

Irreducible limit (documented, not fixed here): middleware cannot make a non-transactional handler’s domain writes atomic with the cache finalize without the handler joining the same transaction (AC #5 forbids holding a tx across the handler). Guarantee = exactly-once on the happy path, at-least-once if a winner crashes after its side effect but before finalize, or returns a retryable 5xx whose side effect already landed. Renewal + claim_id fencing bound how often recovery re-runs and make a false-steal at-least-once, never corruption. True exactly-once (handler write + finalize in one tx, outbox-style) is a filed follow-up.

Decisions (settled)

  • Rollout: expand/contract, 2 releases (ADR-016; migrations.adoc expand-contract section) — a forward-compatible reader (MR2) ships and fully rolls out before the single-flight change (MR3), so old + new replicas coexist safely and every step is rollback-safe.

  • Liveness: renewed + fenced lease — a short lease renewed by a background task while the handler runs; claim_id fences finalize/release/renew. A global request-timeout is ruled out (multi-minute reporting/eligibility POSTs); a held DB connection / advisory lock is ruled out (10-connection pool).

  • Metrics: replace (pure) — drop the flat hits/misses/persist_errors counters; emit an outcome counter + an event counter (see Design › Metrics).

Acceptance criteria (from #1003)

  1. Two concurrent same-key POSTs run the handler exactly once on one replica.

  2. Same guarantee across two replicas sharing the DB.

  3. Followers get the winner’s status/body/supported headers, or a documented retryable response while pending.

  4. Reusing a key with a different body → deterministic conflict, no handler run.

  5. Handler failure + process death have bounded, tested recovery; no permanent wedge. (No DB tx across the handler.)

  6. Existing sequential replay, restart-survival, TTL, cleanup stays covered.

  7. A concurrency regression test fails on current code, passes on the fix.

  8. Metrics distinguish claimed, pending-wait, replay, conflict, abandoned, failed.

Ground truth (verified against source)

Fact Where

Postgres-only prod backend; in-memory DashMap is pub(crate) test-only (#434)

idempotency.rs:205-220

Table via raw DDL under pg_advisory_xact_lock(9999), not sqlx::migrate!

idempotency.rs:142-152, 240-269

Deploy is rolling (k8s, 2+ replicas); expand/contract mandated; forward-only migrations; idempotency_keys is compliance-review-gated

deployment-guide.adoc:216-218; migrations.adoc; ADR-016

sqlx 0.8.6 errors decoding SQL NULL → non-Option (UnexpectedNullError→ColumnDecode); the current read decodes response_status:i32,response_body:Vec<u8> non-Option; the error arm turns it into return None → miss → re-run

idempotency.rs:411-429

On-behalf-of actor is Claims.actor: Option<Box<Claims>> (ADR-019, #[serde(skip)]); sub on a service bearer is the shared service account; authz (actor.has_role, per-assignment) + audit (actor.map_or(sub, a.sub)) key off actor.sub; the cache key uses sub only → cross-actor bypass; auth runs before idempotency so actor is available

claims.rs:23,75; assignments.rs:34; enrollment mod.rs:87; idempotency.rs:353

No inbound request timeout anywhere; reporting FNS-388/ACF-199 + eligibility /determine POSTs run seconds→minutes; no streaming responses; pool max 10 conns, no statement_timeout

lib.rs:167-201; reporting/fns388.rs:53; eligibility/orchestrator.rs:1338; canopy-db/lib.rs:90-95

POST /documents/render sets Content-Disposition, Cache-Control: no-store, X-Canopy-Signature (per-request JWS); body may exceed 2 MiB

notices/api/mod.rs:155-173

Error convention = RFC 9457 (coding-conventions.adoc:249); shared canopy_common::error::{ApiError,ProblemDetails} emits the shape over application/json (not application/problem+json); no 413/503 variants

canopy-common/error.rs:7-133

OpenAPI = per-endpoint #[utoipa::path(responses)]; only global Modify precedent is SecurityAddon; regen = cargo xtask api-docs --update

lib.rs:76-91; xtask/api_docs.rs

Retry-After precedent: portal rate-limit, integer seconds, header::RETRY_AFTER

portal/ratelimit.rs:401

Cleanup DELETE matches by cache_key only (no age re-check in the outer DELETE); tokio interval first tick is immediate (herd on rollout)

idempotency.rs:301-337, 276-283

sha2/proptest are workspace deps but NOT in canopy-api/Cargo.toml

canopy-api/Cargo.toml

Concurrency-test idiom (AtomicU32`join!`store(0)); no Barrier/Notify in repo; request buffer/reconstruct idiom

discovery.rs:339-363; csrf.rs:85-111

Design

Schema (evolve CREATE_TABLE_DDL; SET lock_timeout, CHECK NOT VALID)

Base CREATE TABLE for fresh installs + idempotent ALTER`s for deployed tables, all under the existing advisory lock. `SET LOCAL lock_timeout='3s' first (bounded; fail fast rather than block traffic). Add CHECKs NOT VALID (no full-table scan under ACCESS EXCLUSIVE), validate separately. state DEFAULT 'completed' makes every legacy row a replayable completed row with request_hash NULL.

New columns: state TEXT NOT NULL DEFAULT 'completed' (values pending / completed / failed), request_hash BYTEA (NULL or 32 bytes), claim_id UUID, lease_expires_at TIMESTAMPTZ, response_headers JSONB (allowlisted headers), replayable BOOLEAN NOT NULL DEFAULT true (false = a successful-but-over-cap response), response_status/response_body → DROP NOT NULL.

State-dependent invariants as per-state implications (so failed and completed-unreplayable rows are legal), added NOT VALID then validated:

CHECK (state <> 'completed' OR replayable = false
       OR (response_status IS NOT NULL AND response_body IS NOT NULL))
CHECK (state <> 'pending'
       OR (request_hash IS NOT NULL AND claim_id IS NOT NULL AND lease_expires_at IS NOT NULL))
CHECK (state <> 'failed' OR lease_expires_at IS NOT NULL)
CHECK (request_hash IS NULL OR octet_length(request_hash) = 32)

Each idempotent constraint-add DO-block is scoped by conrelid = 'idempotency_keys'::regclass (not just conname).

Cache key — actor-scoped, canonical, validated (MR1)

Replace the collision-prone colon-join (idempotency.rs:363) with a length-delimited canonical tuple, then a Sha256 digest that is what gets stored/logged (never the raw caller-controlled string). Tuple = (effective principal = actor.sub else sub, method, full path + query, idempotency-key). Roles/programs are NOT in the key (a mid-window role change must not turn a retry into a miss → re-run). Key validation: header absent → proceed unguarded (unchanged); header present but empty / non-UTF-8 / > 255 bytes / duplicated → 400.

Atomic claim — one committed statement (MR3)

$1=cache_key, $2=request_hash, $3=Uuid::now_v7(), $4=lease secs (bound i64, $4::bigint). Use clock_timestamp() (advances during the statement), not now() (frozen at tx start), so a lock-wait cannot mint an already-expired lease. Three CTEs distinguish the metric-relevant outcomes:

WITH ins AS (
    INSERT INTO idempotency_keys
        (cache_key, state, request_hash, claim_id, lease_expires_at, created_at)
    VALUES ($1,'pending',$2,$3, clock_timestamp() + ($4::bigint * interval '1 second'), clock_timestamp())
    ON CONFLICT (cache_key) DO NOTHING
    RETURNING 'claimed'::text AS how
),
ttl AS (   -- >24h old: reusable regardless of body
    UPDATE idempotency_keys k SET state='pending', request_hash=$2, claim_id=$3,
        lease_expires_at=clock_timestamp() + ($4::bigint * interval '1 second'),
        response_status=NULL, response_body=NULL, response_content_type=NULL,
        response_headers=NULL, replayable=true, created_at=clock_timestamp()
      WHERE k.cache_key=$1 AND NOT EXISTS (SELECT 1 FROM ins)
        AND k.created_at < clock_timestamp() - interval '24 hours'
    RETURNING 'recovered_ttl'::text AS how
),
steal AS ( -- crashed winner (pending) or elapsed 5xx-cooldown (failed), SAME body only
    UPDATE idempotency_keys k SET state='pending', request_hash=$2, claim_id=$3,
        lease_expires_at=clock_timestamp() + ($4::bigint * interval '1 second'),
        response_status=NULL, response_body=NULL, response_content_type=NULL,
        response_headers=NULL, replayable=true, created_at=clock_timestamp()
      WHERE k.cache_key=$1 AND NOT EXISTS (SELECT 1 FROM ins) AND NOT EXISTS (SELECT 1 FROM ttl)
        AND k.created_at >= clock_timestamp() - interval '24 hours'
        AND k.state IN ('pending','failed') AND k.lease_expires_at < clock_timestamp()
        AND k.request_hash = $2
    RETURNING 'recovered_abandoned'::text AS how
)
SELECT how FROM ins UNION ALL SELECT how FROM ttl UNION ALL SELECT how FROM steal;

claimed/recovered_ttl/recovered_abandoned ⇒ won; no row ⇒ lost (live-completed < 24h, in-flight pending, or an expired-lease pending/failed with a different body → the loser reads it and 409s). ttl/steal predicates are mutually exclusive (<24h vs >=24h) so ≤1 row updates. Verified SOUND by a Postgres-semantics review (fresh-key race loses by snapshot invisibility; steal race by EvalPlanQual re-check; CTE NOT EXISTS(ins) reads the CTE result; no deadlock/livelock). The steal branch requires request_hash=$2 (AC #4: a different body must not steal a crashed winner’s key).

Renewed + fenced lease (MR3)

On winning, spawn a renewal task: every renew_interval (≈lease/3) run UPDATE …​ SET lease_expires_at=clock_timestamp()+lease WHERE cache_key=$1 AND claim_id=$3 AND state='pending'; stop if rows_affected==0 (stolen/finalized). A drop-guard aborts the task when the handler returns or panics, so a dead handler stops renewing and its lease expires within lease, becoming recoverable. finalize/release/renew are all claim_id-scoped, so a stolen-from winner’s writes are no-ops (it returns its response to its own client and never corrupts the new owner).

Control flow, follower, status matrix (MR3)

validate + digest key; buffer + hash body under a permit (cap exceeded → 413; `to_bytes` cannot distinguish a lower-level body error at this layer and such errors are unreachable here, so all buffer failures map to 413)
claim_id = Uuid::now_v7(); deadline = Instant::now() + follower_wait_budget
loop (deadline- + hard-iteration-cap-bounded; each query wrapped in a remaining-deadline timeout):
  claim → Won  => run_winner()          # events: claimed | recovered_ttl | recovered_abandoned
        → Lost => read_existing (state, request_hash, replayable, response_*, response_headers):
             None                        => continue   # vanished (TTL-cleaned) → re-claim wins
             Completed{hash, replayable} => hash != mine (non-legacy) ? 409
                                            : replayable ? replay(resp+hdrs) : 409 completed-not-replayable
             Pending{hash}               => hash != mine ? 409
                                            : now>=deadline ? 503+Retry-After
                                            : { event pending_wait; backoff-sleep; continue }
             Failed{hash}                => hash != mine ? 409
                                            : lease live ? 503+Retry-After (event retryable_cooldown)
                                            : continue   # cooldown elapsed → next claim reclaims via steal

run_winner: rebuild request → next.run (no tx) → buffer response (configurable cap) → status matrix:

Status Action

2xx, and 4xx ∈ {400,404,409,422} (deterministic)

finalize (owner-scoped): store status/body/allowlisted-headers, replayable=true

401/403 (authz), 3xx

do not cache (authz is revocable; redirects need Location) — return, release

5xx / 408 / 429 (matches retry::classify)

release + short failed cooldown (below)

response > cache cap (success)

finalize as completed with replayable=false, empty body; caller gets its response; followers/retries get a defined 409 "completed, response not replayable" (no double-exec)

finalize rows_affected==0 ⇒ lease was stolen mid-handler ⇒ return own response, do not touch the cache (WARN; at-least-once boundary).

Follower-amplification bound (MR3)

Concurrent callers serialize, but on a 5xx release the next waiter re-claims and re-runs → N sequential executions of a persistently-failing handler. Bound it: the 5xx transition is an owner-scoped (WHERE claim_id=mine) UPDATE to a short failed cooldown (state='failed', claim_id=NULL, response_*=NULL, request_hash retained, lease_expires_at=clock_timestamp()+cooldown). Within the cooldown, same-body followers get 503+Retry-After (event retryable_cooldown) instead of promoting; after cooldown the steal branch reclaims (hash-matched). Caps re-exec to ≈1/cooldown.

Response-header allowlist (MR3; AC #3)

Persist an allowlist in response_headers JSONB: content-type, content-disposition, content-language, cache-control, etag, x-canopy-signature. Never persist set-cookie, authorization, or any auth/session header (a replay must not cross sessions). Replay reconstructs status + body + allowlisted headers + x-idempotency-replay: true. This makes POST /documents/render’s signature/`Content-Disposition/no-store replay correctly. Headers outside the allowlist are dropped.

Error contract (RFC 9457)

Add ApiError::ServiceUnavailable(503) (MR2) and ApiError::PayloadTooLarge(413) (MR3) to canopy_common::error; new responses (409/413/503) return ProblemDetails JSON via ApiError (matches the codebase’s application/json shape; the application/problem+json media-type gap is pre-existing → follow-up). 503s carry Retry-After (integer seconds, header::RETRY_AFTER).

Cleanup fix + interval stagger

Outer DELETE re-checks age + only prunes terminal rows:

DELETE FROM idempotency_keys
 WHERE cache_key IN (SELECT cache_key FROM idempotency_keys
                     WHERE created_at < clock_timestamp() - interval '24 hours' LIMIT $1)
   AND created_at < clock_timestamp() - interval '24 hours'
   AND state IN ('completed','failed');

Never deletes a live pending/reclaimed row. First tick staggered via interval_at(Instant::now() + jittered_initial_delay, period) so replicas don’t herd at boot. (Stagger lands in MR2 since it is a boot-behavior change; the DELETE predicate lands in MR3 with the reclaim path.)

Metrics (AC #8) — two counters

Terminal outcome (exactly one per request): canopy_idempotency_requests_total{outcome} ∈ won_completed | replay | conflict | timeout | failed | completed_uncacheable. Lifecycle events (may fire several per request): canopy_idempotency_events_total{event} ∈ claimed | recovered_ttl | recovered_abandoned | pending_wait | retryable_cooldown | lease_stolen | finalize_error | release_error. Keep ttl_deletes. Replaces the flat hits/misses/persist_errors.

SingleFlightConfig (validated, like RetryPolicy)

Private fields + a validated builder (reject zero/negative poll, inverted backoff bounds, zero/huge lease, renew_interval >= lease, unnamed iteration cap): lease (30s), renew_interval (10s), follower_wait_budget (5s), poll (50ms→1s ±25% jitter), retry_after (1s), failure_cooldown (2s), max_request_body_bytes (= router body_limit), max_cacheable_response_bytes (default 2 MiB), max_response_buffer_bytes (hard buffer ceiling, default 32 MiB — an over-cap success still returns its full body to the caller while being stored non-replayable), max_iterations. Concurrent body buffering is bounded by a process-wide BUFFER_SEMAPHORE const, not a per-config field. Stored on Backend::Postgres { pool, config } (touch the 5 match sites: variant :200, with_pool :286 = SingleFlightConfig::default(), check_cache :399, execute_and_cache :477, test matcher :540). with_pool keeps defaults; a pub fn with_config(self, cfg) → Self builder (doc’d — lib.rs:3 missing_docs
-D warnings) lets ApiServer::router set the body cap (opts.body_limit is in scope at lib.rs:117 — no router-signature change) and lets integration tests shorten budgets. Deadline/backoff math uses Instant::checked_add / saturating_* (J7); LEASE/renew bounded via const _ floor asserts.

Steps (MR sequence)

Each MR: own feature/… branch, own child issue (claim @me, workflow::in-progress), own tests, Relates to #1003 (only MR4 Closes #1003; verify #1003 stays open after each non-final merge). Free helpers return Result<_, sqlx::Error> (no anyhow/Box<dyn Error> at pub boundaries); response build reuses the unwrap_or_else(|_| 500) idiom (idempotency.rs:378), no new expect; all functions ≤ 40 lines; new .rs gets the SPDX header; new pub items get doc comments.

MR1 — security: actor-scoped, canonical, validated cache key

Files: crates/canopy-api/src/idempotency.rs

Rewrite extract_cache_key: effective principal actor.sub ‖ sub; full path
query; length-delimited canonical tuple → Sha256 digest (store/log the digest); key validation → 400 for present-but-invalid. Tests: actor isolation, delimiter-collision, query-param distinctness, invalid/dup/oversized key → 400.

MR2 — expand (deploy-safety prep)

Files: crates/canopy-api/src/idempotency.rs, crates/canopy-common/src/error.rs

Additive DDL (nullable columns, CHECK NOT VALID, lock_timeout, conrelid-scoped constraint add). Reader decodes new columns as Option and fails-closed 503 on any non-replayable row (state <> 'completed' OR response IS NULL OR replayable=false) instead of re-running, so it safely defers to any MR3 replica’s pending/failed rows. Add ApiError::ServiceUnavailable. Stagger the cleanup first tick. Keeps old check-then-execute writes (creates no pending rows). Must be deployed + rolled out to all replicas before MR3. Tests: old-reader-defers (503) on pending/failed/over-cap row; fresh-install + legacy-upgrade DDL determinism.

MR3 — core single-flight

Files: crates/canopy-api/src/idempotency.rs, crates/canopy-api/Cargo.toml, crates/canopy-common/src/error.rs, crates/canopy-api/tests/concurrency/…

Claim CTE, renewed+fenced lease, body-hash 409, follower wait/503, crash recovery, status matrix + response-header allowlist + configurable response cap + uncacheable path, 5xx-release + failure cooldown, cleanup race fix, two metric counters, ApiError::PayloadTooLarge, validated SingleFlightConfig, sha2/proptest deps, Memory-backend refactor (helpers take &Arc<DashMap<…>> so no dead Postgres arm / no unreachable!). Full test suite (see Testing).

MR4 — contract/observability + docs (final; Closes #1003)

Files: crates/canopy-api/src/lib.rs, OpenAPI snapshots, docs, CHANGELOG.adoc, nav.adoc, this plan.

Deviation from the original scope (implemented + why): the IdempotencyResponsesAddon utoipa::Modify hook injecting 409/413/503 into every guarded POST is applied centrally in ApiServer::router (immediately before the SwaggerUi merge), not as per-service modifiers(…) on each of the 17 #[openapi] derives. The cargo xtask api-docs snapshots are fetched from each running service’s live /api-doc/openapi.json — which the router serves — so applying the modifier at the router (the same layer that mounts idempotency_middleware) makes it appear in every service’s snapshot from one edit, keeps a new service in sync automatically, and cannot be forgotten per-service. canopy-web (which does not mount the middleware) is correctly excluded, since it serves its own OpenAPI outside ApiServer::router. Then cargo xtask api-docs --update regenerates the snapshots. Docs updated: shared-crates.adoc, migrations.adoc, the idempotency module docs. CHANGELOG === Removed (the three flat counters, each mapped, with the no-1:1 note for misses), === Changed (single-flight behavior + dashboard migration), === Fixed, === Security. This plan moves to plans/archive/ with the nav.adoc xref updated, all Statuses Done.

Testing

Integration tests → a concurrency test target (per testing.adoc — a crates/canopy-api/tests/concurrency/ module or a named target, not a bare flat filename); SPDX; infrastructure_available() guard; reuse the idempotency_persistence_test.rs harness. Unit tests (claim-outcome mapping, ClaimState↔TEXT, key canonicalization, deadline math, metric emission with asserted labels, const _ floors) live in src/idempotency.rs #[cfg(test)] (private helpers aren’t reachable from an external test crate). Property (proptest): key canonicalization is injective / no-panic; hash_request_body deterministic.

Deterministic overlap (no sleep — the repo has timer starvation). Handler signals entered (mpsc) on entry, then awaits a watch::<bool> release. On the fixed code A’s claim commits before A enters the handler, so when A signals entered the pending row is already committed → spawning B then guarantees B loses → handler_count==1. On current code A writes nothing until after the handler, so B (spawned after A’s entered) misses → runs → a second entered fires → handler_count==2. The count assertion is the invariant; no sleep gates it. Staged so the regression runs on current code: MR3’s first commit adds the concurrency test against the pre-fix logic (old API, no with_config) proving count==2; the fix commit flips it to count==1.

Matrix (each row = one property test): AC#1 one-replica exactly-once; AC#2 two-replica exactly-once on two independently-constructed pools; AC#7 fails-on-current; AC#4 same-key-different-body → 409 + racing-different-bodies; expired-lease steal with a DIFFERENT hash → 409, no run; two simultaneous expired-lease stealers → one wins; healthy handler crossing the lease keeps renewing → not stolen; crashed-winner (expired lease, no renewal) → stealable; AC#3 live-lease pending → 503+Retry-After (short budget); stolen-lease finalize/renew is a no-op; AC#6 TTL-expired row re-executes; cleanup-vs-reclaim (reclaim refreshes created_at while cleanup runs → live row survives); 413 (length) vs 400 (other body error); legacy NULL-hash replays (24h transition exception, no 409); response-header allowlist replayed (incl. x-canopy-signature) + set-cookie dropped; over-cap success → completed-uncacheable → follower 409; every error path (claim/read/finalize/release DB error, query-timeout, ownership loss, all release statuses incl 408); actor isolation (two actors, one bearer, same key → no cross-serve); role-change-mid-window still replays same-actor; metric names+labels+values asserted. AC#6 regression: run idempotency_persistence_test.rs unmodified green under the new schema + add the missing cleanup-behavior test; strengthen restart/two-replica to close+recreate pools.

Review-resolution map

Finding Resolution MR

B1 expired-lease steal ignores body

steal CTE requires request_hash=$2; only ttl(>24h) replaces hash

MR3

B2 fixed lease unenforceable

renewed+fenced lease (global-timeout & held-lock ruled out)

MR3

B3 now() frozen → expired lease

clock_timestamp() for all lease math

MR3

B4 rolling deploy unsafe

expand/contract: MR2 forward-compat 503 reader first

MR2→MR3

B5 legacy NULL-hash vs AC#4

documented ≤24h transition exception; ages out

MR3

B6 cache key ignores actor

key = effective principal actor.sub‖sub

MR1

B7 cleanup deletes reclaimed row

outer DELETE re-checks age + state IN(completed,failed)

MR2 stagger / MR3 predicate

B8 header decision breaks render

persisted header allowlist incl. x-canopy-signature

MR3

fingerprint incomplete / colon-collision

canonical length-delimited digest incl. full path+query

MR1

follower latency unbounded

per-query remaining-deadline timeout + lock_timeout + iteration cap

MR3

retryable amplification

5xx release + failed cooldown → 503; contract+metric+test

MR3

first-party retry ignores Retry-After

documented: internal retry uses its own backoff; Retry-After for external clients

MR4 docs

2 MiB response cap unsafe

configurable cap; over-cap success → completed-uncacheable → 409

MR3

request-buffer memory amplification

bounded concurrent-buffer semaphore (config)

MR3

body-read error conflated with 413

length-error → 413, other body error → 400

MR3

key validation unsafe

present-but-invalid key → 400; store/log digest

MR1

cached-response privacy

pre-existing; documented + follow-up

MR4 / follow-up

weak schema invariants

state-dependent CHECKs + 32-byte hash CHECK (NOT VALID)

MR2/MR3

constraint detection not relation-scoped

conrelid='idempotency_keys'::regclass

MR2

boot DDL blocks traffic

lock_timeout + CHECK NOT VALID + separate validate

MR2

cleanup herd on boot

interval_at + jittered initial delay

MR2

status classification contradictory

explicit matrix (don’t cache 401/403/3xx)

MR3

RFC 9457 error bodies

ProblemDetails for 409/413/503; add ApiError variants

MR2 (503) / MR3 (413)

SingleFlightConfig unvalidated

private fields + validated builder

MR3

run_winner omits pool; u64 lease

pool param added; lease bound i64, $4::bigint

MR3

metrics incoherent

split requests_total{outcome} vs events_total{event}; TTL vs abandoned split

MR3

OpenAPI regen unjustified

global Modify responses applied centrally in ApiServer::router (supersedes per-service wiring — snapshots are fetched from the router-served live spec) + api-docs --update

MR4

stale docs

shared-crates/testing/migrations/module docs updated

MR4

CHANGELOG maps 1 of 3 metrics

map all three removed counters + no-1:1 note

MR4

plan archive "after merge"

nav/archive in MR4

MR4

Delivery mechanics

Per MR: cargo fmt --all → cargo clippy -p canopy-api -p canopy-common --all-targets --profile test — -D warnings → cargo xtask quality-budgets → cargo xtask validate → cargo nextest run -p canopy-api --profile integration (source .ports.env) on warm devstack; MR4 also cargo xtask api-docs --update. Commit as the human (signed), body ending Co-Authored-By: the actual implementing-session model. Force-merge per project protocol (CI never passes here — the pre-push battery is the functional gate). Closing comment on each child issue; Closes #1003 only on MR4; epic &44 updated after each merge.

Follow-ups (file & /relate #1003)

  • application/problem+json media type codebase-wide.

  • Cached-response privacy / crypto-shred coordination + storage bound.

  • Global inbound request-timeout as DoS hygiene (17 services).

  • Exactly-once outbox (handler write + finalize in one tx).

  • SingleFlightConfig → per-service ServiceSettings env.

  • response_location for 201-Created idempotency.

Edit this page · default