Plan: canopy-store upload validation — full port (Issue #435)

On this page

Status

Step Description Status

1

Workspace + canopy-store dependency additions: add infer = "0.19", sha2 = "0.10", unicode-normalization = "0.1.24", async-trait = "0.1" to [workspace.dependencies] in root Cargo.toml. Add each as { workspace = true } to crates/canopy-store/Cargo.toml [dependencies].

Not started

2

crates/canopy-store/src/scanner.rs (new): Scanner trait (#[async_trait], Send + Sync), ScanResult enum (Clean / Infected / Skipped), ScanError (thiserror), NoopScanner struct (always returns Clean). 2 unit tests (NoopScanner returns Clean; trait object instantiable). Re-export from lib.rs.

Not started

3

crates/canopy-store/src/error.rs: extend StoreError enum with 5 new variants: Empty, UnknownContentType, ContentTypeMismatch { claimed, actual }, Infected { signature, scanner }, ScannerError(#[from] ScanError). Extend From<StoreError> for ApiError impl with matching arms (Empty / UnknownContentType / ContentTypeMismatch / InfectedBadRequest; ScannerErrorinternal).

Not started

4

crates/canopy-store/src/validation.rs validate_upload rewritten: becomes async, takes bytes: &[u8], claimed_content_type: &str, filename: Option<&str>, &UploadValidation, &dyn Scanner. Returns ValidatedUpload { sha256: [u8; 32], content_type, size, sanitized_filename }. Order of checks: size → magic-byte sniff (infer::get) → claimed-vs-actual match → allowlist check → sha256 (Sha256::digest) → scanner.scan().await → filename sanitization.

Not started

5

crates/canopy-store/src/validation.rs sanitize_filename rewritten: NFC normalize via unicode_normalization::UnicodeNormalization::nfc, reject C0 (\x00..=\x1F), C1 (\x7F..=\x9F), bidi overrides (\u{202A}..=\u{202E}, \u{2066}..=\u{2069}), strip / and \, trim whitespace + dots, byte-bounded 255 truncation at char boundary. Signature Result<String, StoreError> preserved.

Not started

6

Existing 6 tests in validation.rs migrated to #[tokio::test] since validate_upload is now async. New tests added: magic-mismatch err, magic-match-extracted, sha256-deterministic, scanner-infected-rejected, NFC-equivalence, C0-control-rejected, C1-control-rejected, RTL-override-rejected, multi-byte-truncation-at-char-boundary, empty-bytes-err. ~16 tests total.

Not started

7

crates/canopy-store/src/disposition.rs (new): Disposition enum (Inline, Attachment), content_disposition_header(filename, disposition) → String per RFC 6266 §4.1 + RFC 5987 §3.2. Internal percent_encode for non-ASCII. 6 tests: ASCII inline, ASCII attachment, Cyrillic UTF-8, Chinese UTF-8, embedded-quote-escapes, embedded-backslash-escapes. Re-export from lib.rs.

Not started

8

crates/canopy-store/src/reconcile.rs (new): ReconcileReport { orphans, leaks }, compare(db_paths, live_keys) → ReconcileReport. Pure function over HashSet diffs. 4 tests: empty/empty; all-match; some-orphan-some-leak; prefix-substring not treated as match. Re-export from lib.rs.

Not started

9

crates/canopy-store/src/store.rs: add Store::put_validated(path, bytes, claimed_content_type, &validation, &scanner) → Result<ValidatedUpload, StoreError> (calls validate_upload then put). Add Store::list_all_keys(prefix: Option<&str>) → Result<Vec<String>, StoreError> paginating the object_store::list API. Mark existing Store::put with #[deprecated(note = "use put_validated; #435 requires upload validation for all writes")]. 4 tests against object_store::memory::InMemory backend.

Not started

10

canopy-notices migration services/canopy-notices/migrations/20260515000000_add_notice_content_integrity.sql: ALTER TABLE notices ADD COLUMN content_sha256 BYTEA NOT NULL DEFAULT '\\x', ADD COLUMN content_type TEXT NOT NULL DEFAULT 'application/pdf', ADD COLUMN scan_status TEXT NOT NULL DEFAULT 'noop'. COMMENTs reference #435 + Pub 1075 §9. Forward-only per ADR-016; DEFAULTs preserve pre-migration rows.

Not started

11

canopy-notices generator.rs:124-127: replace self.object_store.put(…​) with self.object_store.put_validated(…​). Generator struct gains scanner: Arc<dyn canopy_store::Scanner> field constructed at Generator::new with Arc::new(NoopScanner). The ValidatedUpload.sha256 + content_type flow into the notices row via the existing insert_notice path (which gains 2 new column args).

Not started

12

canopy-notices src/reconcile.rs (new): run_reconciliation_loop(store, db, interval) infinite loop with tokio::time::interval (hardcoded 24h). reconcile_once selects pdf_storage_path from notices, calls store.list_all_keys(Some("notices/")), runs canopy_store::reconcile::compare, logs orphans + leaks via tracing::warn! (count + first-10 sample). Wired into main.rs via tokio::spawn(…​).

Not started

13

CHANGELOG entry under === Changed. docs/modules/ROOT/pages/data-models/canopy-notices.adoc updated for the 3 new notices columns. Precommit Q1-Q8 answered via subagent verification per .githooks/pre-commit rule from ae8251f.

Not started

Issue: #435
Branch: feat/canopy-store-upload-validation
Labels: compliance::pub-1075, priority::medium, service::shared-crates, type::security, workflow::ready

Context

Today, canopy-notices renders PDFs and pushes them directly to object storage with no validation of any kind:

// services/canopy-notices/src/generator.rs:124-127
self.object_store.put(&path, Bytes::from(rendered.pdf_bytes)).await

The notices DB row records pdf_storage_path, pdf_size_bytes, page_count — nothing about content integrity. validate_upload() exists at crates/canopy-store/src/validation.rs:40-55 but is never called. Store::put() doesn’t accept a content_type.

The 2026-05-09 external review flagged this as a Pub 1075 §9 ATO gap. User direction (2026-05-14): adopt zero-trust default, harden the object store NOW before user-upload endpoints (applicant portal per ADR-008, FFE attachments per #189–https://gitlab.com/gadhs/application/eligibility/canopy/-/issues/195[#195]) land. Immediate behavioral impact is small (Typst output is trusted; NoopScanner is permissive; canopy-notices is the only uploader) but posture impact is large.

What this plan ports from CRAIG

  1. Magic-byte verification via infer — catches content/claimed-type mismatch.

  2. sha256 hashing — integrity chain enabling future verify-on-download tamper detection.

  3. Unicode filename validation — NFC normalization, C0/C1 control-char + bidi-override rejection.

  4. Content-Disposition normalization — RFC 6266 / RFC 5987 helper for future download endpoints.

  5. AV scanner integration hookScanner trait + NoopScanner. Real impl deferred until needed.

  6. Periodic reconciliation — orphan + leak detection. Logged via tracing; metrics-emission deferred.

Code references

  • crates/canopy-store/src/lib.rs:24-26 — current public re-exports.

  • crates/canopy-store/src/validation.rs:40-55 — current validate_upload (inert).

  • crates/canopy-store/src/validation.rs:61-84 — current sanitize_filename (no NFC, no control-char check).

  • crates/canopy-store/src/error.rs:7-34 — current StoreError enum (5 variants).

  • crates/canopy-store/src/store.rs:62Store::put(path, bytes) signature.

  • services/canopy-notices/src/generator.rs:124-127 — only Store::put caller; bypasses validation.

  • services/canopy-notices/migrations/20260401000000_create_notices_tables.sql:23-25 — current notices columns.

  • ADR-013 — Status vocabulary.

  • ADR-016 — migration discipline.

Scope

In scope (single MR feat/canopy-store-upload-validation):

  • All 13 Status-table steps land together. ~800 LOC new code + ~500 LOC tests.

  • DB migration backward-compatible (DEFAULTs preserve pre-migration rows).

  • Store::put retained but #[deprecated]-flagged with attribute (not just rustdoc); compile-time deprecation warning at any future direct caller.

Out of scope:

  • Real (non-Noop) Scanner implementation. Adding a ClamAV-over-TCP scanner requires deployment wiring. Trait surface stabilises now; wiring comes when needed.

  • Verify-on-download. Future MR; sha256 stored at upload is the prerequisite.

  • Metrics export of orphan/leak counts. Reconciliation logs via tracing; OTLP gauge export comes with the broader observability pass.

  • Migration of any future HTTP upload endpoints to use the new pipeline. None today.

  • Hardening pdf_size_bytes with a non-zero CHECK constraint at the DB level. Size validation lives in put_validated now; DB constraint is future tightening.

Dependencies

  • No upstream code or plan dependencies. #435 is independent of the other Tier 1 issues (#438 ✅, #437, #433, #436).

  • Convention dependencies: ADR-013, ADR-016, ADR-001, pre-commit Q1-Q8 (.githooks/pre-commit from ae8251f).

  • New direct deps: infer = "0.19" (workspace lock already has v0.19.0), sha2 = "0.10", unicode-normalization = "0.1.24", async-trait = "0.1".

Design

Scanner trait

// crates/canopy-store/src/scanner.rs

use async_trait::async_trait;

#[async_trait]
pub trait Scanner: Send + Sync {
    async fn scan(&self, bytes: &[u8]) -> Result<ScanResult, ScanError>;
    fn name(&self) -> &'static str;
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ScanResult {
    Clean,
    Infected { signature: String },
    Skipped { reason: String },
}

#[derive(Debug, thiserror::Error)]
pub enum ScanError {
    #[error("scanner backend error: {0}")]
    Backend(String),
    #[error("scan timed out after {0:?}")]
    Timeout(std::time::Duration),
}

pub struct NoopScanner;

#[async_trait]
impl Scanner for NoopScanner {
    async fn scan(&self, _bytes: &[u8]) -> Result<ScanResult, ScanError> {
        Ok(ScanResult::Clean)
    }
    fn name(&self) -> &'static str { "noop" }
}

Extended StoreError variants

// Additions to crates/canopy-store/src/error.rs

/// Upload contained zero bytes.
Empty,

/// Magic-byte sniffing could not identify the content type.
UnknownContentType,

/// Caller-claimed content type does not match the magic-byte-detected
/// type. Catches client-lies and internal rendering bugs.
ContentTypeMismatch { claimed: String, actual: String },

/// AV scanner reported a positive detection.
Infected { signature: String, scanner: String },

/// Scanner backend failed (transport-level, not a positive detection).
ScannerError(#[from] crate::scanner::ScanError),

validate_upload rewrite

pub struct ValidatedUpload {
    pub sha256: [u8; 32],
    pub content_type: String,
    pub size: usize,
    pub sanitized_filename: Option<String>,
}

pub async fn validate_upload(
    bytes: &[u8],
    claimed_content_type: &str,
    filename: Option<&str>,
    validation: &UploadValidation<'_>,
    scanner: &dyn crate::scanner::Scanner,
) -> Result<ValidatedUpload, StoreError> {
    if bytes.is_empty() { return Err(StoreError::Empty); }
    if bytes.len() > validation.max_bytes {
        return Err(StoreError::TooLarge { size: bytes.len(), limit: validation.max_bytes });
    }
    let kind = infer::get(bytes).ok_or(StoreError::UnknownContentType)?;
    let actual = kind.mime_type();
    if actual != claimed_content_type {
        return Err(StoreError::ContentTypeMismatch {
            claimed: claimed_content_type.to_string(),
            actual: actual.to_string(),
        });
    }
    if !validation.allowed_mime_types.contains(&actual) {
        return Err(StoreError::DisallowedContentType(actual.to_string()));
    }
    use sha2::{Sha256, Digest};
    let sha256: [u8; 32] = Sha256::digest(bytes).into();
    use crate::scanner::ScanResult;
    match scanner.scan(bytes).await? {
        ScanResult::Clean | ScanResult::Skipped { .. } => {}
        ScanResult::Infected { signature } => {
            return Err(StoreError::Infected {
                signature,
                scanner: scanner.name().to_string(),
            });
        }
    }
    let sanitized_filename = filename.map(sanitize_filename).transpose()?;
    Ok(ValidatedUpload { sha256, content_type: actual.to_string(), size: bytes.len(), sanitized_filename })
}

sanitize_filename rewrite

pub fn sanitize_filename(name: &str) -> Result<String, StoreError> {
    use unicode_normalization::UnicodeNormalization;
    let normalized: String = name.nfc().collect();
    for c in normalized.chars() {
        let cp = c as u32;
        let is_c0 = cp <= 0x1F;
        let is_c1 = (0x7F..=0x9F).contains(&cp);
        let is_bidi = matches!(cp, 0x202A..=0x202E | 0x2066..=0x2069);
        if is_c0 || is_c1 || is_bidi {
            return Err(StoreError::InvalidFilename(
                format!("disallowed character U+{cp:04X}")
            ));
        }
    }
    let stripped: String = normalized.chars()
        .filter(|c| !matches!(*c, '/' | '\\'))
        .collect();
    let trimmed = stripped.trim().trim_matches('.').to_string();
    if trimmed.is_empty() {
        return Err(StoreError::InvalidFilename("empty after sanitization".into()));
    }
    let mut out = trimmed;
    if out.len() > 255 {
        let mut cut = 255;
        while !out.is_char_boundary(cut) { cut -= 1; }
        out.truncate(cut);
    }
    Ok(out)
}

Content-Disposition helper

// crates/canopy-store/src/disposition.rs

pub fn content_disposition_header(filename: &str, disposition: Disposition) -> String {
    let kind = match disposition {
        Disposition::Inline => "inline",
        Disposition::Attachment => "attachment",
    };
    let is_ascii_safe = filename.is_ascii()
        && !filename.chars().any(|c| c == '"' || c == '\\');
    if is_ascii_safe {
        format!(r#"{kind}; filename="{filename}""#)
    } else {
        format!("{kind}; filename*=UTF-8''{}", percent_encode(filename))
    }
}

pub enum Disposition { Inline, Attachment }

fn percent_encode(s: &str) -> String {
    let mut out = String::with_capacity(s.len() * 3);
    for b in s.bytes() {
        let unreserved = b.is_ascii_alphanumeric() || matches!(b, b'-' | b'_' | b'.' | b'~');
        if unreserved { out.push(b as char); }
        else { use std::fmt::Write; write!(out, "%{b:02X}").unwrap(); }
    }
    out
}

Reconciliation utility

// crates/canopy-store/src/reconcile.rs

use std::collections::HashSet;

pub struct ReconcileReport {
    pub orphans: Vec<String>,  // in DB, not in storage
    pub leaks: Vec<String>,    // in storage, not in DB
}

pub fn compare(db_paths: &[String], live_keys: &[String]) -> ReconcileReport {
    let db: HashSet<&str> = db_paths.iter().map(String::as_str).collect();
    let live: HashSet<&str> = live_keys.iter().map(String::as_str).collect();
    ReconcileReport {
        orphans: db.difference(&live).map(|s| s.to_string()).collect(),
        leaks: live.difference(&db).map(|s| s.to_string()).collect(),
    }
}

Store extensions

impl Store {
    #[deprecated(note = "use put_validated; #435 requires upload validation for all writes")]
    pub async fn put(&self, path: &str, data: Bytes) -> Result<(), StoreError> { /* unchanged */ }

    pub async fn put_validated(
        &self,
        path: &str,
        data: Bytes,
        claimed_content_type: &str,
        validation: &UploadValidation<'_>,
        scanner: &dyn Scanner,
    ) -> Result<ValidatedUpload, StoreError> {
        let validated = validate_upload(&data, claimed_content_type, None, validation, scanner).await?;
        #[allow(deprecated)]
        self.put(path, data).await?;
        Ok(validated)
    }

    pub async fn list_all_keys(&self, prefix: Option<&str>) -> Result<Vec<String>, StoreError> {
        // Wraps object_store::list with full pagination.
    }
}

canopy-notices wire-up

services/canopy-notices/src/generator.rs:124-127: replace direct put with:

let validation = UploadValidation::default();
let validated = self.object_store
    .put_validated(&path, Bytes::from(rendered.pdf_bytes), "application/pdf",
                   &validation, &*self.scanner)
    .await?;
// validated.sha256 + content_type flow into the notices row via
// insert_notice (which gains 2 new column args).

Generator struct gains scanner: Arc<dyn canopy_store::Scanner>; Generator::new constructs with Arc::new(NoopScanner).

canopy-notices reconciliation task

// services/canopy-notices/src/reconcile.rs

pub async fn run_reconciliation_loop(store: Arc<Store>, db: PgPool, interval: Duration) {
    let mut ticker = tokio::time::interval(interval);
    ticker.tick().await; // skip immediate first tick
    loop {
        ticker.tick().await;
        if let Err(e) = reconcile_once(&store, &db).await {
            tracing::warn!(error = %e, "notice reconciliation failed");
        }
    }
}

Wired in main.rs via tokio::spawn(reconcile::run_reconciliation_loop(…​, Duration::from_secs(24 * 3600))) after store + db boot.

Migration

-- services/canopy-notices/migrations/20260515000000_add_notice_content_integrity.sql

ALTER TABLE notices
    ADD COLUMN content_sha256 BYTEA NOT NULL DEFAULT '\x',
    ADD COLUMN content_type TEXT NOT NULL DEFAULT 'application/pdf',
    ADD COLUMN scan_status TEXT NOT NULL DEFAULT 'noop';

COMMENT ON COLUMN notices.content_sha256 IS 'SHA-256 of the stored PDF bytes (#435 / Pub 1075 §9 integrity chain).';
COMMENT ON COLUMN notices.content_type IS 'Magic-byte-verified content type at upload time (#435).';
COMMENT ON COLUMN notices.scan_status IS 'AV scanner result at upload: clean | infected | skipped | noop (#435).';

Files Touched

File Change

Cargo.toml (root)

Add infer = "0.19", sha2 = "0.10", unicode-normalization = "0.1.24", async-trait = "0.1" to [workspace.dependencies].

Cargo.lock

Updated by cargo automatically; stage with commit.

crates/canopy-store/Cargo.toml

Add infer, sha2, unicode-normalization, async-trait as { workspace = true } [dependencies].

crates/canopy-store/src/lib.rs

Add pub mod scanner;, pub mod disposition;, pub mod reconcile;. Re-export new types.

crates/canopy-store/src/scanner.rs

New file (~80 LOC + 2 tests).

crates/canopy-store/src/error.rs

Add 5 new StoreError variants + matching From<StoreError> for ApiError arms.

crates/canopy-store/src/validation.rs

Rewrite validate_upload (async, takes bytes + scanner, returns ValidatedUpload). Rewrite sanitize_filename (NFC, control-char + bidi reject). Migrate existing 6 tests to #[tokio::test]. Add ~10 new tests.

crates/canopy-store/src/disposition.rs

New file (~60 LOC + 6 tests).

crates/canopy-store/src/reconcile.rs

New file (~30 LOC + 4 tests).

crates/canopy-store/src/store.rs

Add put_validated(…​), list_all_keys(…​). #[deprecated] attribute on put. ~4 new tests.

services/canopy-notices/migrations/20260515000000_add_notice_content_integrity.sql

New migration (3 columns + DEFAULTs + COMMENTs).

services/canopy-notices/src/generator.rs

Replace store.put(…​) with store.put_validated(…​). Plumb scanner: Arc<dyn Scanner> through Generator::new. Persist sha256 + content_type into notices row.

services/canopy-notices/src/store.rs (or wherever insert_notice lives)

Add content_sha256: &[u8] + content_type: &str params; bind into the SQL.

services/canopy-notices/src/reconcile.rs

New file (~50 LOC; loop + reconcile_once).

services/canopy-notices/src/lib.rs (or main module declarations)

Add pub mod reconcile;.

services/canopy-notices/src/main.rs

Wire tokio::spawn(reconcile::run_reconciliation_loop(…​)) after store + db boot.

CHANGELOG.adoc

Entry under == Unreleased / === Changed.

docs/modules/ROOT/pages/data-models/canopy-notices.adoc

notices table gains 3 columns; update Mermaid ERD + per-column descriptions.

OpenAPI snapshot regeneration: not required (no API surface changes).

Verification

  1. cargo nextest run -p canopy-store --lib — all new tests pass + existing 8 tests still pass.

  2. cargo nextest run -p canopy-notices — full canopy-notices suite passes.

  3. cargo build -p canopy-notices — compiles cleanly.

  4. cargo fmt --check --all + cargo clippy --all-targets — -D warnings — zero warnings.

  5. cargo xtask api-docs — confirm OpenAPI snapshots unchanged.

  6. cargo xtask validate — full battery green.

  7. Manual smoke: cargo xtask dev refresh, generate a notice, verify notices row has non-empty content_sha256 + content_type='application/pdf'.

  8. Adversarial smoke: test that feeds non-PDF bytes with claimed="application/pdf" must Err(StoreError::ContentTypeMismatch).

Documentation Updates

  • Plan filed at docs/modules/ROOT/pages/plans/canopy-store-upload-validation.adoc.

  • CHANGELOG.adoc — entry under == Unreleased / === Changed.

  • docs/modules/ROOT/pages/data-models/canopy-notices.adoc — 3 new columns documented.

  • Service Catalog — canopy-notices section gains reconciliation loop + new notice columns.

  • Security — if it documents upload-validation posture, update.

  • Plan moves to plans/archive/ post-merge per ADR-013.

Why this approach (vs alternatives)

  • Don’t trim the scope. User explicitly chose zero-trust default; cost of dormant validation code is small, cost of bolting it on after user-uploads land is large.

  • Don’t add a real (non-Noop) Scanner. Requires deployment wiring; trait surface stabilises now, wiring comes when needed.

  • Don’t defer the reconciliation job. Small, additive, 24h interval. Detecting bypass attempts when scanners or user-upload endpoints land needs the job already running.

  • Don’t remove Store::put entirely. #[deprecated] is the clippy nudge without breakage.

  • Don’t make the migration NOT NULL without DEFAULTs. Forward-only per ADR-016.

  • Don’t make the reconciliation interval configurable. Hardcoded 24h is fine today; configurability is a separate concern.

Risk + Rollback

  • Risk: infer magic-byte detection rejects a legitimate PDF with nonstandard byte ordering. Mitigation: tests include Typst-rendered PDF fixtures from test-results/rendered-pdfs/.

  • Risk: reconciliation log samples leak UUIDs. Mitigation: paths include UUIDs, not PII.

  • Risk: Store::put deprecation trips clippy on transitive callers. Mitigation: canopy-notices is the only known caller (migrated in this MR).

  • Risk: Generator::new signature change is breaking for direct constructors in tests. Mitigation: grep + update each callsite.

  • Rollback: revert the MR; canopy-notices returns to direct Store::put. Migration columns stay (ADR-016 forward-only).

Edit this page · default