Plan: canopy-store upload validation — full port (Issue #435)
On this page
Status
| Step | Description | Status |
|---|---|---|
1 |
Workspace + canopy-store dependency additions: add |
Not started |
2 |
|
Not started |
3 |
|
Not started |
4 |
|
Not started |
5 |
|
Not started |
6 |
Existing 6 tests in |
Not started |
7 |
|
Not started |
8 |
|
Not started |
9 |
|
Not started |
10 |
canopy-notices migration |
Not started |
11 |
canopy-notices |
Not started |
12 |
canopy-notices |
Not started |
13 |
CHANGELOG entry under |
Not started |
Issue: #435
Branch: feat/canopy-store-upload-validation
Labels: compliance::pub-1075, priority::medium, service::shared-crates, type::security, workflow::ready
Context
Today, canopy-notices renders PDFs and pushes them directly to object storage with no validation of any kind:
// services/canopy-notices/src/generator.rs:124-127
self.object_store.put(&path, Bytes::from(rendered.pdf_bytes)).await
The notices DB row records pdf_storage_path, pdf_size_bytes, page_count — nothing about content integrity. validate_upload() exists at crates/canopy-store/src/validation.rs:40-55 but is never called. Store::put() doesn’t accept a content_type.
The 2026-05-09 external review flagged this as a Pub 1075 §9 ATO gap. User direction (2026-05-14): adopt zero-trust default, harden the object store NOW before user-upload endpoints (applicant portal per ADR-008, FFE attachments per #189–https://gitlab.com/gadhs/application/eligibility/canopy/-/issues/195[#195]) land. Immediate behavioral impact is small (Typst output is trusted; NoopScanner is permissive; canopy-notices is the only uploader) but posture impact is large.
What this plan ports from CRAIG
-
Magic-byte verification via
infer— catches content/claimed-type mismatch. -
sha256 hashing — integrity chain enabling future verify-on-download tamper detection.
-
Unicode filename validation — NFC normalization, C0/C1 control-char + bidi-override rejection.
-
Content-Disposition normalization — RFC 6266 / RFC 5987 helper for future download endpoints.
-
AV scanner integration hook —
Scannertrait +NoopScanner. Real impl deferred until needed. -
Periodic reconciliation — orphan + leak detection. Logged via tracing; metrics-emission deferred.
Code references
-
crates/canopy-store/src/lib.rs:24-26— current public re-exports. -
crates/canopy-store/src/validation.rs:40-55— currentvalidate_upload(inert). -
crates/canopy-store/src/validation.rs:61-84— currentsanitize_filename(no NFC, no control-char check). -
crates/canopy-store/src/error.rs:7-34— currentStoreErrorenum (5 variants). -
crates/canopy-store/src/store.rs:62—Store::put(path, bytes)signature. -
services/canopy-notices/src/generator.rs:124-127— onlyStore::putcaller; bypasses validation. -
services/canopy-notices/migrations/20260401000000_create_notices_tables.sql:23-25— currentnoticescolumns. -
ADR-013 — Status vocabulary.
-
ADR-016 — migration discipline.
Scope
In scope (single MR feat/canopy-store-upload-validation):
-
All 13 Status-table steps land together. ~800 LOC new code + ~500 LOC tests.
-
DB migration backward-compatible (DEFAULTs preserve pre-migration rows).
-
Store::putretained but#[deprecated]-flagged with attribute (not just rustdoc); compile-time deprecation warning at any future direct caller.
Out of scope:
-
Real (non-Noop) Scanner implementation. Adding a ClamAV-over-TCP scanner requires deployment wiring. Trait surface stabilises now; wiring comes when needed.
-
Verify-on-download. Future MR; sha256 stored at upload is the prerequisite.
-
Metrics export of orphan/leak counts. Reconciliation logs via tracing; OTLP gauge export comes with the broader observability pass.
-
Migration of any future HTTP upload endpoints to use the new pipeline. None today.
-
Hardening
pdf_size_byteswith a non-zero CHECK constraint at the DB level. Size validation lives input_validatednow; DB constraint is future tightening.
Dependencies
-
No upstream code or plan dependencies. #435 is independent of the other Tier 1 issues (#438 ✅, #437, #433, #436).
-
Convention dependencies: ADR-013, ADR-016, ADR-001, pre-commit Q1-Q8 (
.githooks/pre-commitfromae8251f). -
New direct deps:
infer = "0.19"(workspace lock already has v0.19.0),sha2 = "0.10",unicode-normalization = "0.1.24",async-trait = "0.1".
Design
Scanner trait
// crates/canopy-store/src/scanner.rs
use async_trait::async_trait;
#[async_trait]
pub trait Scanner: Send + Sync {
async fn scan(&self, bytes: &[u8]) -> Result<ScanResult, ScanError>;
fn name(&self) -> &'static str;
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ScanResult {
Clean,
Infected { signature: String },
Skipped { reason: String },
}
#[derive(Debug, thiserror::Error)]
pub enum ScanError {
#[error("scanner backend error: {0}")]
Backend(String),
#[error("scan timed out after {0:?}")]
Timeout(std::time::Duration),
}
pub struct NoopScanner;
#[async_trait]
impl Scanner for NoopScanner {
async fn scan(&self, _bytes: &[u8]) -> Result<ScanResult, ScanError> {
Ok(ScanResult::Clean)
}
fn name(&self) -> &'static str { "noop" }
}
Extended StoreError variants
// Additions to crates/canopy-store/src/error.rs
/// Upload contained zero bytes.
Empty,
/// Magic-byte sniffing could not identify the content type.
UnknownContentType,
/// Caller-claimed content type does not match the magic-byte-detected
/// type. Catches client-lies and internal rendering bugs.
ContentTypeMismatch { claimed: String, actual: String },
/// AV scanner reported a positive detection.
Infected { signature: String, scanner: String },
/// Scanner backend failed (transport-level, not a positive detection).
ScannerError(#[from] crate::scanner::ScanError),
validate_upload rewrite
pub struct ValidatedUpload {
pub sha256: [u8; 32],
pub content_type: String,
pub size: usize,
pub sanitized_filename: Option<String>,
}
pub async fn validate_upload(
bytes: &[u8],
claimed_content_type: &str,
filename: Option<&str>,
validation: &UploadValidation<'_>,
scanner: &dyn crate::scanner::Scanner,
) -> Result<ValidatedUpload, StoreError> {
if bytes.is_empty() { return Err(StoreError::Empty); }
if bytes.len() > validation.max_bytes {
return Err(StoreError::TooLarge { size: bytes.len(), limit: validation.max_bytes });
}
let kind = infer::get(bytes).ok_or(StoreError::UnknownContentType)?;
let actual = kind.mime_type();
if actual != claimed_content_type {
return Err(StoreError::ContentTypeMismatch {
claimed: claimed_content_type.to_string(),
actual: actual.to_string(),
});
}
if !validation.allowed_mime_types.contains(&actual) {
return Err(StoreError::DisallowedContentType(actual.to_string()));
}
use sha2::{Sha256, Digest};
let sha256: [u8; 32] = Sha256::digest(bytes).into();
use crate::scanner::ScanResult;
match scanner.scan(bytes).await? {
ScanResult::Clean | ScanResult::Skipped { .. } => {}
ScanResult::Infected { signature } => {
return Err(StoreError::Infected {
signature,
scanner: scanner.name().to_string(),
});
}
}
let sanitized_filename = filename.map(sanitize_filename).transpose()?;
Ok(ValidatedUpload { sha256, content_type: actual.to_string(), size: bytes.len(), sanitized_filename })
}
sanitize_filename rewrite
pub fn sanitize_filename(name: &str) -> Result<String, StoreError> {
use unicode_normalization::UnicodeNormalization;
let normalized: String = name.nfc().collect();
for c in normalized.chars() {
let cp = c as u32;
let is_c0 = cp <= 0x1F;
let is_c1 = (0x7F..=0x9F).contains(&cp);
let is_bidi = matches!(cp, 0x202A..=0x202E | 0x2066..=0x2069);
if is_c0 || is_c1 || is_bidi {
return Err(StoreError::InvalidFilename(
format!("disallowed character U+{cp:04X}")
));
}
}
let stripped: String = normalized.chars()
.filter(|c| !matches!(*c, '/' | '\\'))
.collect();
let trimmed = stripped.trim().trim_matches('.').to_string();
if trimmed.is_empty() {
return Err(StoreError::InvalidFilename("empty after sanitization".into()));
}
let mut out = trimmed;
if out.len() > 255 {
let mut cut = 255;
while !out.is_char_boundary(cut) { cut -= 1; }
out.truncate(cut);
}
Ok(out)
}
Content-Disposition helper
// crates/canopy-store/src/disposition.rs
pub fn content_disposition_header(filename: &str, disposition: Disposition) -> String {
let kind = match disposition {
Disposition::Inline => "inline",
Disposition::Attachment => "attachment",
};
let is_ascii_safe = filename.is_ascii()
&& !filename.chars().any(|c| c == '"' || c == '\\');
if is_ascii_safe {
format!(r#"{kind}; filename="{filename}""#)
} else {
format!("{kind}; filename*=UTF-8''{}", percent_encode(filename))
}
}
pub enum Disposition { Inline, Attachment }
fn percent_encode(s: &str) -> String {
let mut out = String::with_capacity(s.len() * 3);
for b in s.bytes() {
let unreserved = b.is_ascii_alphanumeric() || matches!(b, b'-' | b'_' | b'.' | b'~');
if unreserved { out.push(b as char); }
else { use std::fmt::Write; write!(out, "%{b:02X}").unwrap(); }
}
out
}
Reconciliation utility
// crates/canopy-store/src/reconcile.rs
use std::collections::HashSet;
pub struct ReconcileReport {
pub orphans: Vec<String>, // in DB, not in storage
pub leaks: Vec<String>, // in storage, not in DB
}
pub fn compare(db_paths: &[String], live_keys: &[String]) -> ReconcileReport {
let db: HashSet<&str> = db_paths.iter().map(String::as_str).collect();
let live: HashSet<&str> = live_keys.iter().map(String::as_str).collect();
ReconcileReport {
orphans: db.difference(&live).map(|s| s.to_string()).collect(),
leaks: live.difference(&db).map(|s| s.to_string()).collect(),
}
}
Store extensions
impl Store {
#[deprecated(note = "use put_validated; #435 requires upload validation for all writes")]
pub async fn put(&self, path: &str, data: Bytes) -> Result<(), StoreError> { /* unchanged */ }
pub async fn put_validated(
&self,
path: &str,
data: Bytes,
claimed_content_type: &str,
validation: &UploadValidation<'_>,
scanner: &dyn Scanner,
) -> Result<ValidatedUpload, StoreError> {
let validated = validate_upload(&data, claimed_content_type, None, validation, scanner).await?;
#[allow(deprecated)]
self.put(path, data).await?;
Ok(validated)
}
pub async fn list_all_keys(&self, prefix: Option<&str>) -> Result<Vec<String>, StoreError> {
// Wraps object_store::list with full pagination.
}
}
canopy-notices wire-up
services/canopy-notices/src/generator.rs:124-127: replace direct put with:
let validation = UploadValidation::default();
let validated = self.object_store
.put_validated(&path, Bytes::from(rendered.pdf_bytes), "application/pdf",
&validation, &*self.scanner)
.await?;
// validated.sha256 + content_type flow into the notices row via
// insert_notice (which gains 2 new column args).
Generator struct gains scanner: Arc<dyn canopy_store::Scanner>; Generator::new constructs with Arc::new(NoopScanner).
canopy-notices reconciliation task
// services/canopy-notices/src/reconcile.rs
pub async fn run_reconciliation_loop(store: Arc<Store>, db: PgPool, interval: Duration) {
let mut ticker = tokio::time::interval(interval);
ticker.tick().await; // skip immediate first tick
loop {
ticker.tick().await;
if let Err(e) = reconcile_once(&store, &db).await {
tracing::warn!(error = %e, "notice reconciliation failed");
}
}
}
Wired in main.rs via tokio::spawn(reconcile::run_reconciliation_loop(…, Duration::from_secs(24 * 3600))) after store + db boot.
Migration
-- services/canopy-notices/migrations/20260515000000_add_notice_content_integrity.sql
ALTER TABLE notices
ADD COLUMN content_sha256 BYTEA NOT NULL DEFAULT '\x',
ADD COLUMN content_type TEXT NOT NULL DEFAULT 'application/pdf',
ADD COLUMN scan_status TEXT NOT NULL DEFAULT 'noop';
COMMENT ON COLUMN notices.content_sha256 IS 'SHA-256 of the stored PDF bytes (#435 / Pub 1075 §9 integrity chain).';
COMMENT ON COLUMN notices.content_type IS 'Magic-byte-verified content type at upload time (#435).';
COMMENT ON COLUMN notices.scan_status IS 'AV scanner result at upload: clean | infected | skipped | noop (#435).';
Files Touched
| File | Change |
|---|---|
|
Add |
|
Updated by cargo automatically; stage with commit. |
|
Add |
|
Add |
|
New file (~80 LOC + 2 tests). |
|
Add 5 new |
|
Rewrite |
|
New file (~60 LOC + 6 tests). |
|
New file (~30 LOC + 4 tests). |
|
Add |
|
New migration (3 columns + DEFAULTs + COMMENTs). |
|
Replace |
|
Add |
|
New file (~50 LOC; loop + reconcile_once). |
|
Add |
|
Wire |
|
Entry under |
|
|
OpenAPI snapshot regeneration: not required (no API surface changes).
Verification
-
cargo nextest run -p canopy-store --lib— all new tests pass + existing 8 tests still pass. -
cargo nextest run -p canopy-notices— full canopy-notices suite passes. -
cargo build -p canopy-notices— compiles cleanly. -
cargo fmt --check --all+cargo clippy --all-targets — -D warnings— zero warnings. -
cargo xtask api-docs— confirm OpenAPI snapshots unchanged. -
cargo xtask validate— full battery green. -
Manual smoke:
cargo xtask dev refresh, generate a notice, verifynoticesrow has non-emptycontent_sha256+content_type='application/pdf'. -
Adversarial smoke: test that feeds non-PDF bytes with
claimed="application/pdf"mustErr(StoreError::ContentTypeMismatch).
Documentation Updates
-
Plan filed at
docs/modules/ROOT/pages/plans/canopy-store-upload-validation.adoc. -
CHANGELOG.adoc — entry under
== Unreleased/=== Changed. -
docs/modules/ROOT/pages/data-models/canopy-notices.adoc— 3 new columns documented. -
Service Catalog — canopy-notices section gains reconciliation loop + new notice columns.
-
Security — if it documents upload-validation posture, update.
-
Plan moves to
plans/archive/post-merge per ADR-013.
Why this approach (vs alternatives)
-
Don’t trim the scope. User explicitly chose zero-trust default; cost of dormant validation code is small, cost of bolting it on after user-uploads land is large.
-
Don’t add a real (non-Noop) Scanner. Requires deployment wiring; trait surface stabilises now, wiring comes when needed.
-
Don’t defer the reconciliation job. Small, additive, 24h interval. Detecting bypass attempts when scanners or user-upload endpoints land needs the job already running.
-
Don’t remove
Store::putentirely.#[deprecated]is the clippy nudge without breakage. -
Don’t make the migration NOT NULL without DEFAULTs. Forward-only per ADR-016.
-
Don’t make the reconciliation interval configurable. Hardcoded 24h is fine today; configurability is a separate concern.
Risk + Rollback
-
Risk:
infermagic-byte detection rejects a legitimate PDF with nonstandard byte ordering. Mitigation: tests include Typst-rendered PDF fixtures fromtest-results/rendered-pdfs/. -
Risk: reconciliation log samples leak UUIDs. Mitigation: paths include UUIDs, not PII.
-
Risk:
Store::putdeprecation trips clippy on transitive callers. Mitigation: canopy-notices is the only known caller (migrated in this MR). -
Risk:
Generator::newsignature change is breaking for direct constructors in tests. Mitigation: grep + update each callsite. -
Rollback: revert the MR; canopy-notices returns to direct
Store::put. Migration columns stay (ADR-016 forward-only).