Plan: #1208 — async, durable, archive-aware v1 audit archival

On this page
NOTE

Rev 2 after external review rejected rev 1 on 20 findings. Rebuilt, not patched: the durable run record resolves the async-contract, cadence, lock-safety, HTTP-safety, idempotency, and accountability findings as ONE mechanism; the DB-clock cutoff + explicit columns resolve the loss-proof/skew findings; the read-contract inventory resolves the silent-truncation findings. The age threshold is NOT retention — policy stays in #1303.

Status

Step Description Status

1

Contracts: ArchiveRequest rename, run DTOs (ArchiveRunAccepted, ArchiveRunStatus, ArchiveRunErrorCode, ArchiveListParams), interim-type deletions, ARCHIVE_RUN path, AuditArchiveRunId, roundtrips.

Done (2026-08-07) — DTOs + ARCHIVE_RUN path + AuditArchiveRunId landed; interim types deleted; roundtrips updated

2

Migration: audit_archive_runs + audit_archive_schedule + four indexes
the definition-verification DO block; test-lib touch-comment companion.

Done (2026-08-07) — 20261101000000 with both tables, the four indexes + the DO verifier; db.rs touch comment cited

3

Store/mover: AUDIT_EVENT_COLUMNS, MOVE_CHUNK_SQL, head-exclusion, preflights, MoveChunkError, runs.rs job/schedule functions.

Done (2026-08-07) — 17-column const, loss-proof chunk move with chain-head exclusion, both preflights, runs.rs job/schedule fns

4

Runner: ArchiveWorker always-spawned tick loop (claim_due → enqueue → claim → preflights → chunk loop → finalize → pull-forward).

Done (2026-08-07) — always-spawned supervised loop, fenced heartbeats, catch-up pull-forward, capped backlog probe

5

Config: 11 serde-defaulted archive_* keys, ArchiveConfigError, domain/relationship validation at boot.

Done (2026-08-07) — 11 keys validated by ArchiveConfig::from_config (error, never clamp; required-iff-enabled)

6

API: admin-only 202/409 enqueue, GET /v1/security/archive-runs/{id} poll, truthful archive-list params, interim-type removal, OpenAPI pins 16→17.

Done (2026-08-07) — 202/409 enqueue + poll + keyset list; interim types gone; OpenAPI pins at 17

7

Read-contract seam: archive ∪ live unions (by-id, FOIA export, fact-history), designated hot-only surfaces, SELECT * retirement on touched paths.

Done (2026-08-07) — by-id/export/fact-history union; list + summary designated hot-only; touched paths on the column const

8

Health/metrics/main: boot wiring, BackgroundWorkerHealth("audit-archive") non-gating readiness, archive/metrics.rs.

Done (2026-08-07) — boot wiring, non-gating /readyz check, archive/metrics.rs gauges/counters

9

Test-lib client: ArchiveRunEnqueued 202/409 decoder, archive_run poll wrapper, list_archived keyset wrapper.

Done (2026-08-07) — ArchiveRunEnqueued decoder + poll + keyset wrappers

10

Docs + CHANGELOG (six 5y→7y surfaces, configuration-reference, ops runbook, api page) + file the partitioning/cold-tier follow-up + the #1303 hand-off comment.

Done (2026-08-07) — docs + CHANGELOG landed; battery green; MR delivered with the storage-architecture follow-up issue filed + the #1303 hand-off comment posted

Status: Done (2026-08-07) — all steps delivered in one MR; plan archived
Epic: &73
Issues: #1208 (priority::high)
Branch: feature/1208-audit-archive-async

TL;DR

  1. Replace the interim-503 POST /v1/security/archive with the chain-verify-jobs pattern applied to archival: a durable audit_archive_runs table (single active run, token-fenced lease, per-chunk committed progress) + an always-spawned in-service runner.

  2. Admin-only POST enqueues → 202 + {run_id, poll_url} + Location; active run ⇒ 409 with that run’s handle (durable ⇒ replay-safe under the idempotency cache); new GET /v1/security/archive-runs/{id} poll (OpenAPI pin 16→17, deliberate).

  3. Mover = explicit-column atomic chunks, cutoff computed by the DB clock (received_at < now() - make_interval(days ⇒ $1)), head-exclusion guard, per-chunk commit; inserted == deleted asserted before every commit.

  4. Scheduling = a transactional due-state row (audit_archive_schedule, Skip semantics — no Burst catch-up, no per-replica interval phase); more=true pulls the next due time forward (default 30s) so backlogs drain immediately and boundedly. Defaults sustain 28.8M rows/day (5.8× the repo’s stated 5M/day ceiling).

  5. Read contracts: by-id, FOIA export, and fact-history become archive ∪ live; the events list, summary, and detection windows are designated hot-only with the designation documented (pushback P2 below).

  6. The knob is archive_after_days — an age threshold, sanity-bounded 1..=36500, required-iff-scheduler-enabled (no default: no policy embedded). Retention floors, legal hold, purge, per-family policy = #1303. The 5y→7y Pub 1075 doc corrections ride along as doc fixes (six files).

Context

1245 deleted the one-transaction mover (unbounded tx / genesis-breaking boundary / ON CONFLICT DO NOTHING silent loss) and left POST /v1/security/archive a fixed 503. #1208’s standing AC: the endpoint triggers/enqueues the scheduled job, never executes inline. Substrate facts (verified at 92f31d76): archive twin = LIKE INCLUDING ALL executed BEFORE the base table’s secondary indexes — so only the PK copied (dup id ⇒ loud 23505; archive has no other indexes today); append-only triggers (UPDATE/DELETE/ TRUNCATE, both tables) bypass = SET LOCAL canopy.audit_maintenance='on'; INSERT unguarded; PREDECESSOR_HASH_SQL is live-only; v1 verify_chain is [cfg(test)]-only and retires with #1304.

Decisions — each review finding → its resolution

# Finding Resolution

1

Cutoff ≠ retention

Knob renamed archive_after_days (age threshold; archive retains indefinitely, so moving early shortens nothing). No AU-11 floor, no policy knobs. Sanity domain 1..=36500 only. 5y→7y doc fixes separate. Policy → #1303

2

Async contract

Durable run enqueue: 202 + handle + poll endpoint; never inline

3

Throughput

Defaults 5,000 × 20 chunks/pass @ 300s = 28.8M rows/day; more=true ⇒ next_due_at = LEAST(now()+30s, …) immediate bounded continuation; partitioning/cold-tier filed as its own issue, not assumed

4

Read contracts

Full FROM audit_events inventory (table below); union where history is promised; hot-only surfaces DESIGNATED in OpenAPI + docs; archive-side indexes per union arm

5

Cadence ≠ mutex

audit_archive_schedule singleton; due-claim = UPDATE … SET next_due_at = now()+interval WHERE next_due_at ⇐ now() RETURNING — transactional, one winner, Skip semantics (a week of downtime = ONE claim); runner first tick delayed

6

Advisory-lock unsafe

Dropped. Token-fenced lease on the run row (lease_token uuidv7, lease_expires_at, heartbeat Ok(false) = fenced ⇒ abandon). No idle-in-tx connection held; pool-size-1 test proves it

7

HTTP-unsafe sync

Gone (async). Per-chunk fenced heartbeat UPDATE persists chunks_committed/rows_archived — partial progress always durable
pollable

8

Idempotency replay

No transient-409 class exists: 409 only for a durable active run WITH its handle; 202 replay returns the same durable handle; endpoint declares its own 409/503 (the addon never overwrites)

9

Index rollout

CREATE INDEX IF NOT EXISTS + a migration DO block that RAISEs unless indisvalid AND indisready AND exact pg_get_indexdef match (name-collision with a wrong/invalid index fails the boot loudly); runbook: out-of-band CONCURRENTLY pre-create for large tables (PG 18 pinned in devstack)

10

Authz/accountability

POST = require_admin() unconditionally — service tokens rejected (test-pinned). The run row IS the accountability record: requested_by (admin:{sub} / scheduler), threshold, config snapshot, committed chunks/rows, more, outcome. Reads stay is_service() || admin

11

Chain honesty

Explicit acceptance section below — no "every #1245 condemnation fixed" claim

12

Duplicate wedge

Bounded first-chunk overlap preflight (≤ chunk_size PK probes) + per-chunk 23505 enforcement ⇒ error/duplicate_overlap on the run row + health degradation + runbook. Full-table overlap query = enablement-time runbook step (O(min(live,archive)) — not per-pass). No in-code auto-reconcile (P4)

13

Loss-proof move

One AUDIT_EVENT_COLUMNS const (17 columns) used on BOTH sides; candidates → INSERT..RETURNING id → DELETE USING (only ids cross CTEs); Rust asserts inserted == deleted pre-commit

14

Archive GET broken

Truthful dedicated ArchiveListParams: validated limit 1..=500 (→400), keyset (before_received_at, before_id) both-or-neither (→400), ORDER BY received_at DESC, id DESC. The advertised-but-ignored filters are REMOVED from this endpoint (P1)

15

Error mapping

Store stays sqlx::Result; handlers ride ApiError::from (preserves PoolTimedOut→503); runner classifies SQLSTATE (23505/57014) before any flattening; OpenAPI documents 500 + 503

16

Hygiene

New ArchiveConfigError (own Display, archive key names); bindable int types; SQL in LazyLock<String>; ceiling 36500 justified (100y > any statute; keeps make_interval in INT range; pure sanity since the DB computes the cutoff); archive_scheduler_enabled named for what it does; more = "pass ended on a full chunk; more movable rows may remain (coexists with the retained head)"

17

Fail-green

BackgroundWorkerHealth("audit-archive") non-gating in /readyz + OTel metrics (last-success age, duration, rows, chunks, capped backlog, run_failures_total{error_code}, scheduler lag)

18/19

Tests/delivery

29-test matrix below (rev-1 arithmetic fixed); nextest group across ALL FIVE profiles; rebuild+restart BEFORE integration tests + OpenAPI regen

20

Docs

Six 5y→7y surfaces; configuration-reference; CHANGELOG Added/Changed/Fixed/Removed; ops runbook

Steps

Step 1: Contracts (crates/canopy-contracts-security + canopy-common)

  • events.rs: ArchiveRequest { archive_after_days: i32 } (honest rename; pre-1.0, no shim). Delete ArchiveResponse. Add ArchiveRunAccepted { run_id, poll_url }, ArchiveRunState (queued|running|done|error), ArchiveRunStatus (run_id, state, requested_by, requested_at, archive_after_days, chunk_size, max_chunks_per_pass, attempts, chunks_committed, rows_archived, more: Option<bool>, error_code: Option<ArchiveRunErrorCode>, started_at, finished_at), ArchiveRunErrorCode (duplicate_overlap|upgrade_state_unrepaired|statement_timeout|db_error|crashed — closed enum), ArchiveListParams { limit, before_received_at, before_id }.

  • chain.rs: delete ChainStatusInterim/InterimChainState (+ their module-doc block; zero other consumers — verified).

  • paths.rs: ARCHIVE_RUN = "/v1/security/archive-runs/{id}".

  • canopy-common/src/id.rs: define_id!(AuditArchiveRunId).

  • Roundtrips: update arb_archive_request; add the three new DTOs; drop removed types.

Step 2: Migration services/canopy-security/migrations/20261101000000_audit_archive_runs.sql

SPDX + rationale comments. Four indexes:

  • idx_audit_events_received_at_id (received_at, id) — the mover’s candidate scan.

  • idx_audit_events_archive_received_at_id (received_at, id) — archive keyset list.

  • idx_audit_events_archive_event_timestamp_id (event_timestamp, id) — FOIA export union arm.

  • idx_audit_events_archive_persons_metadata partial GIN (WHERE source_service='canopy-persons') — fact-history union arm (P3: partial, matching the query predicate exactly).

Then the definition-verification DO block: for each of the four names, RAISE unless pg_index.indisvalid AND indisready and pg_get_indexdef exactly equals the expected definition (regclass-rendered so ephemeral schemas match) — IF NOT EXISTS checks only the name; a wrong same-named index must fail the boot loudly. (Exact plpgsql finalized at implementation; the contract is fixed.) Migration header cites the 20260811000000 precedent (its no-CONCURRENTLY-under-sqlx rationale + out-of-band pre-create posture).

Role note (verified): the dormant chain-v2 role split doesn’t apply — every service connects as canopy, which owns tables its migrations create, so plain SQL on audit_archive_runs works (the reporting report_runs migration is the explicit no-SECURITY-DEFINER precedent). Post-#1279 cutover grants are the same deferred exposure as every v1 table.

audit_archive_runs table (uuidv7 PK; state queued|running|done|error; requested_by; archive_after_days CHECK 1..=36500; chunk_size CHECK 100..=20000; max_chunks_per_pass CHECK 1..=1000; lease_owner/lease_token/lease_expires_at/heartbeat_at; attempts; chunks_committed; rows_archived; more; error_code CHECK in the closed set; error_detail; started/finished_at; state-consistency CHECKs incl. (state='running') = (lease_owner IS NOT NULL), terminal ⇔ finished_at, error ⇔ error_code, queued ⇒ zero progress). One active run total: partial unique index on true WHERE state IN ('queued','running'). Queued + reclaim partial indexes.

audit_archive_schedule singleton (next_due_at, last_claimed_at; seeded now()). Local due-state is shape-compatible with #1211’s future shared fence — converge when it lands. audit_archive_runs is deliberately NOT append-only-guarded (operational state).

Companion: cite the migration in `crates/canopy-test-lib/src/db.rs’s touch comment (#1242 embed staleness).

Step 3: Store/mover (services/canopy-security/src/archive/{mod,runs,mover}.rs)

  • pub const AUDIT_EVENT_COLUMNS — the 17 explicit columns, single source for INSERT target + a-qualified SELECT source (ordinal drift ⇒ compile-visible column-name error, not a silent swap).

  • MOVE_CHUNK_SQL: LazyLock<String> — WITH candidates (SELECT id … WHERE received_at < now() - make_interval(days ⇒ $1) AND id <> (head subquery) ORDER BY received_at, id LIMIT $2), moved (INSERT INTO archive ({cols}) SELECT \{a.cols} FROM audit_events a JOIN candidates USING (id) RETURNING id), deleted (DELETE … USING moved RETURNING id) SELECT counts — Rust asserts inserted == deleted before commit; mismatch ⇒ rollback
    db_error.

  • Head subquery ordering is load-bearing: (SELECT id FROM audit_events ORDER BY created_at DESC, id DESC LIMIT 1) — the CHAIN-HEAD ordering (PREDECESSOR_HASH_SQL, store/mod.rs:80), NOT the candidate ordering. created_at is clock_timestamp() (strictly increasing in append-lock order); received_at is tx-start now() and does NOT follow lock order — the two orderings genuinely diverge, and excluding the wrong "head" would let the next append’s predecessor be moved out from under it. Test 1 gains a divergence case: a row with the newest created_at but an old cutoff-eligible received_at must be the retained one.

  • Per-chunk tx: SET LOCAL statement_timeout = $cfg → SET LOCAL canopy.audit_maintenance = 'on' → move → commit.

  • MoveChunkError (thiserror): DuplicateId (SQLSTATE 23505), StatementTimeout (57014), Db(sqlx::Error).

  • Preflights: upgrade_state_unrepaired (live empty AND archive non-empty — two EXISTS probes) and first_chunk_overlap (first candidate window JOIN archive USING id LIMIT 1 — bounded ≤ chunk_size PK probes).

  • runs.rs — the chain_verify jobs shapes in plain SQL: enqueue (tri-state via the partial-unique 23505 ⇒ AlreadyActive(id)), claim (queued first, else expired-lease reclaim, FOR UPDATE SKIP LOCKED; attempts ladder ⇒ error/crashed with progress intact), heartbeat_progress (token-fenced UPDATE; Ok(false) = fenced), finalize, claim_due (the transactional Skip-semantics due claim), pull_due_forward (LEAST(next_due_at, now()+catchup)), poll.

Step 4: Runner (archive/worker.rs)

ArchiveWorker::spawn(pool, cfg, health) — always spawned (manual runs must be serviced with the scheduler off, else 202 lies). Delayed first tick; per tick: beat → (if scheduler enabled) claim_due → enqueue("scheduler", …) (AlreadyActive = fine, logged) → claim → preflights (refusals finalize with the typed code, recorded on the row) → chunk loop ≤ max_chunks_per_pass (each chunk: move → fenced heartbeat_progress; fenced ⇒ silently drop the run, no second finalize; chunk error ⇒ finalize with code, committed progress stands) → finalize(done, more = last chunk full) → if more, pull_due_forward → health + metrics + capped backlog probe (LIMIT 100001 count). Crash ⇒ lease expiry ⇒ reclaim resumes (the predicate is the cursor — already-moved rows aren’t candidates).

Step 5: Config (config.rs) — 11 keys, all serde-defaulted (absent = dormant)

Key (CANOPY_SECURITY__…) Default Domain

archive_scheduler_enabled

false

—

archive_after_days

None

1..=36500; required iff scheduler enabled

archive_chunk_size

5000

100..=20000

archive_max_chunks_per_pass

20

1..=1000

archive_interval_secs

300

60..=86400

archive_catchup_interval_secs

30

5..=3600, ≤ interval

archive_statement_timeout_ms

30000

1000..=300000

archive_lease_secs

120

10..=600; ≥ 3×(statement_timeout/1000)

archive_max_attempts

3

1..=10

archive_runner_tick_ms

5000

500..=60000

archive_first_tick_delay_secs

60

0..=3600

New ArchiveConfigError (own type + Display naming archive keys). ArchiveConfig::from_config validates domains + relationships at boot (error, never clamp); all fields pub; bindable int types. NOT added to default.yaml (serde defaults + dormancy pin; the chain-verify-keys precedent — stated in the MR).

Step 6: API (api/mod.rs)

  • run_archive: require_admin() only (service tokens 403 — pinned); body domain check → 400; enqueue → Created ⇒ 202 + ArchiveRunAccepted
    Location / AlreadyActive ⇒ 409 + that run’s ArchiveRunAccepted; PoolTimedOut rides ApiError::from ⇒ 503. utoipa: 202/400/403/409/500/503.

  • New get_archive_run (is_service() || admin): poll → ArchiveRunStatus, 404 unknown.

  • list_archived → Query<ArchiveListParams>; validate limit + cursor completeness → 400 (negative paging can never reach PG); keyset store fn.

  • Delete chain_interim_response + interim imports/components; register the four new DTO schemas.

  • BOTH OpenAPI pins update 16→17: openapi_doc_generates (api/mod.rs:1236-1239) AND openapi_snapshot_pins_the_16_paths (tests/chain_verifier_host_test.rs:2383-2393) — the latter also snapshots the path LIST via insta (tests/snapshots/chain_verifier_host_test__chain_v2_unified_namespace_paths.snap), so the new /security/archive-runs/{id} entry needs a blessed snapshot update + a renamed test (…pins_the_17_paths).

  • Interim-type full touch list: contracts chain.rs types + module doc; api/mod.rs imports/components/chain_interim_response; tests/security_test.rs imports (:33-34) + DELETE run_archive_is_gated_to_interim_unknown (:467-490, replaced by tests 20-26) + the :809-812 breadcrumb; test-lib run_archive rewrite (step 9). Do NOT copy chain_verify_enqueue’s authz line (`is_service() || admin) — this POST is admin-only by decision 10.

  • Location header on the 202 only (the reporting precedent sets none on 409).

Step 7: Read-contract seam — every FROM audit_events, classified

Surface Contract Mechanism

PREDECESSOR_HASH_SQL (:80)

Live-only by design

Head-exclusion + upgrade-state preflight protect it

list_audit_events (:203) → GET /v1/security/events

Hot-only, DESIGNATED (P2)

OpenAPI + docs: serves rows younger than the operator threshold; history rides by-id/export/archive GET

list_audit_events_for_export (:241) → FOIA export

Archive ∪ live

UNION ALL, explicit columns, ORDER BY event_timestamp ASC, id ASC (tie-break added); archive (event_timestamp,id) index

get_audit_event (:257) → by-id

Archive ∪ live

Live PK probe, else archive PK probe (two indexed lookups)

AUDIT_SUMMARY_SQL (:358)

Hot-only, designated

Doc note: windows wider than the threshold undercount by design

list_fact_change_history (fact_history.rs:32)

Archive ∪ live

UNION ALL identical predicates, ORDER BY created_at ASC, id ASC; partial GIN arm

detection.rs:79 count

Hot-only by design

Minutes-scale window ≪ any threshold; comment

VERIFY_CHAIN_WALK_SQL + tests

Test-only

Union-walk helper added for new tests

list_archived_events

Archive-only (its purpose)

Keyset (received_at DESC, id DESC)

count_archived_events (:401-405)

DELETE — zero callers, #[allow(dead_code)]

Pre-1.0: no dead code carried; returns when a caller exists

All touched SQL moves to the explicit AUDIT_EVENT_COLUMNS const (retires SELECT * on these paths).

Step 8: Health/metrics/main

ArchiveConfig::from_config(…​).expect at boot; ArchiveWorker::spawn; BackgroundWorkerHealth("audit-archive") non-gating in /readyz (stale_after = max(3×tick, 60s); idle ticks beat, dormant never false-degrades); archive/metrics.rs gauges/counters per decision 17.

Step 9: Test-lib client

ArchiveRunEnqueued { Accepted | AlreadyActive } decoding 202 AND 409 into the typed handle (ReportRunEnqueued mold); archive_run(id) poll wrapper; list_archived(params) keyset wrapper.

Chain-boundary honesty (stated, not claimed away)

  • Interior moves accepted: (received_at, id)-ordered candidates can move non-contiguous chain rows. Acceptable because (a) the only v1 verifier is test-only and retires with #1304, (b) archive ∪ live retains every row and the union walk verifies (test 2), (c) what this plan structurally fixes = #1245’s unbounded tx + silent loss + request-path false breach. NOT claimed: "every #1245 condemnation fixed".

  • Upgrade state (archive non-empty / live empty): per-run preflight REFUSES (upgrade_state_unrepaired); repair = ops runbook (gated re-seed under the maintenance GUC); tooling deferred to #1303.

  • Concurrent append vs move: test 3; appends serialize on pg_advisory_xact_lock(1) independent of chunk txs; the head guard means live never drains empty.

Test matrix (29)

# Name Pins

1

archive_moves_all_but_head_across_passes

25 old ⇒ 24 moved across passes, head retained (rev-1 arithmetic fixed); variant with a fresh head ⇒ all 25 move; divergence case: newest-created_at row with an old cutoff-eligible received_at is the retained one (the head is the CHAIN head, M2)

2

union_chain_verifies_and_next_append_extends_live_head

REAL insert_audit_event rows; union walk verifies; next append chains from the retained head (no re-genesis)

3

concurrent_appends_during_archival_lose_nothing_and_do_not_fork

Racing appenders vs multi-chunk drain; counts conserve; single genesis

4

archive_only_live_empty_state_refuses_upgrade_state_unrepaired

Preflight refusal on the run row

5

duplicate_in_first_chunk_fails_preflight_before_any_move

duplicate_overlap + offending id; zero moved

6

duplicate_in_later_chunk_errors_with_committed_progress

chunk 1 committed; SQLSTATE 23505 asserted at the mover layer

7

cutoff_boundary_is_strictly_less_than

Strict < pinned at the boundary

8

cutoff_is_computed_by_the_database_clock

Static pin: SQL contains now() - make_interval, binds no app timestamp

9

statement_timeout_rolls_back_chunk_atomically

Real mid-statement kill (57014); zero rows moved

10

fenced_worker_heartbeat_returns_false_and_abandons

Stale token ⇒ Ok(false); no double finalize

11

lease_expiry_reclaims_then_crashes_out_at_max_attempts

Attempts ladder ⇒ error/crashed, totals intact

12

due_claim_is_transactional_and_skips_missed_ticks

One winner under concurrency; week-overdue ⇒ ONE claim; next_due_at = now()+interval

13

two_runners_share_one_active_run

Two loops, one schema (staggered replicas/rolling restart);
first_tick_is_delayed unit

14

more_true_pulls_the_schedule_forward

Full-chunk pass ⇒ due ≤ now()+catchup; drain completes across passes

15

pool_of_one_connection_completes_a_run

max_connections(1) — the advisory-lock deadlock class is gone

16

overlap_preflight_plan_is_bounded_pk_probes

EXPLAIN: no full archive scan

17

mover_and_reads_ride_the_new_indexes

EXPLAIN (ANALYZE, BUFFERS, WAL) at ~1KB metadata width; all four indexes; in-test caveat that tiny-fixture seqscan-off proves eligibility only

18

index_verifier_rejects_wrong_same_named_index

Wrong same-named index ⇒ DO block RAISEs

19

archive_config_domains_and_relationships (+ archive_keys_default_when_absent)

Boundaries accepted AND rejected (1000/1001, 86400/86401, 100/99…); enabled-without-days error names the key; new error type’s Display

20

audit_archive_http_post_rejects_service_tokens

Service credential ⇒ 403

21

audit_archive_http_enqueue_poll_roundtrip

202+Location; poll to done; archive_after_days=36500 (NO no-old-rows assumption on the shared devstack); cleans up its run rows

22

audit_archive_http_conflict_returns_active_run_handle

Seeded active run ⇒ 409 carrying THAT id

23

audit_archive_http_validation_rejects_out_of_domain

0 / 36501 ⇒ 400, never PG 500

24

audit_archive_http_unknown_run_is_404

—

25

audit_archive_http_list_validates_bounds_and_keyset_orders

limit 0/501 ⇒ 400; lone cursor half ⇒ 400; page 2 strictly older; DESC tie-break

26

audit_archive_http_idempotency_replays_the_durable_handle

Same-key replay ⇒ same handle; fresh-key continuation documented; no transient-409 class exists

27

historical_reads_cross_the_archive_seam

by-id finds archived; export spans the seam ordered; fact-history complete; list + summary live-only AS DESIGNATED

28

openapi_doc_generates

17 paths, amended message

29

contracts roundtrips

New DTOs roundtrip; removed DTOs gone

Isolation: store/runner tests on EphemeralSchema; HTTP tests own + clean their run rows; nextest group security-audit-archive = { max-threads = 1 } filtering package(canopy-security) & test(audit_archive_http_), replicated under all five profiles (default, integration, validate, ci, ci-integration).

Docs

  • data-models/canopy-security (retention ≠ threshold narrative, new tables/indexes)

  • fti_audit.rs:10 doc comment

  • data-models/canopy-tanf (:58,:346)

  • data-models/canopy-medicaid (:79,~:458)

  • nist-architecture-mapping (~:87) — all 5y→7y

  • configuration-reference (11 new vars)

  • security-operations runbook (enablement: full overlap query + CONCURRENTLY pre-create + indisvalid verify; duplicate-wedge recovery: all-17-columns IS NOT DISTINCT FROM manual reconcile under the GUC; upgrade-state repair; more/backlog interpretation)

  • api/canopy-security.adoc (async POST contract, poll endpoint, truthful GET params, hot-only designations, background task list)

  • CHANGELOG Added/Changed/Fixed/Removed (restored endpoint = Changed contract; renamed field; dropped GET filters; admin-only; union reads; removed DTOs)

Delivery

  1. Repo plan .adoc (this plan) committed + nav-linked (Active) as the branch’s first commit; flipped Done → Archive in the final commit. Branch feature/1208-audit-archive-async. Single MR, Closes #1208; commits signed, subjects ≤72, Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>.

  2. Order: contracts + id newtype → migration + service code (store/worker/config/api/health) → nextest.toml groups → rebuild + restart canopy-security (boot applies the migration) → cargo xtask test --integration (new binary — never the old 503 handler) → cargo xtask api-docs --update (17-path snapshot) → docs/CHANGELOG → full pre-push battery.

  3. File the storage-architecture follow-up issue (partitioning of audit_events_archive on received_at vs external cold tier; relates #1208/#1303/#1247). Post the #1303 hand-off comment. Epic &74 note post-merge.

Pushbacks / judgment calls (decide-or-accept at approval)

  • P1: archive GET gets truthful keyset params, NOT the advertised AuditListParams filters — filters over a billions-row archive without per-filter indexes recreate the O(n) trap; windowed FOIA needs ride the export union. Implementing filters = per-filter archive index decisions, beyond "narrowly mechanical".

  • P2: GET /v1/security/events designated hot-only rather than unioned — offset pagination over a union is the same O(n) hazard; its consumers are operational feeds; history contracts ride by-id/export/fact-history/archive GET. The review’s "explicitly designate" arm — but a stricter reading of finding 4 could demand union here.

  • P3: fact-history’s archive index is a partial GIN on metadata (WHERE source_service='canopy-persons' — the query’s constant predicate; resource_type = $1 stays a heap recheck on both arms, same as the live side today). Still the most expensive new index; the CONCURRENTLY runbook applies foremost to it.

  • P4: no in-code duplicate auto-reconcile — automation deleting audit rows on equality heuristics is worse than a loud wedge; runbook-manual only.

  • P5: one active slot total (queued counts), no depth-N queue — the mover’s work is defined by table state; a queue adds nothing.

Deferred / filed

#1303: retention policy, floors, legal hold, purge, per-family, upgrade-state repair tooling · new issue: partitioning/cold-tier evaluation · #1304: verifier retirement (the interior-move acceptance rests on it) · #1247: FTI twin · #1211: converge the local due-state row onto the shared fence when it lands.

Conventions checklist

SPDX everywhere · uuidv7 PKs + AuditArchiveRunId newtype · thiserror at module boundaries, sqlx::Result in store, no anyhow at pub boundaries · no serde_json::Value · typed validated config (error never clamp), dormant-by-default · ≤40-line fns · no unwrap/expect outside boot+tests · proptests on all new DTOs · plan committed + nav-linked before implementation · CHANGELOG four-section coverage · api-docs blessed in-MR.

Review provenance

Rev 1 rejected by external review (20 findings). Rev 2: fresh design resolving each finding (decision table maps finding → mechanism); one contextless verification round over the rev-2 plan — 3 material findings folded (the second OpenAPI pin + insta snapshot; the head-subquery ordering made explicit with a divergence test; the full interim-type touch list) plus nits (count_archived_events deleted, Location-on-202-only, GIN phrasing, migration-header precedent cite, role-model verification note). Reviewer verdict on substrate fit: the plain-SQL runs table is viable under the single canopy role (dormant NOLOGIN chain roles don’t bind), with the reporting report_runs migration as the explicit precedent.

Edit this page · default