Runbook: clamav (clamd) Sidecar Operations

The devstack image (devstack/clamav/Dockerfile, digest-pinned clamav/clamav:1.4) ships a build-time definition snapshot; the canopy_clamav_db volume receives it on first mount and the in-container freshclam daemon keeps it fresh incrementally. After editing the baked test.ndb or clamd.conf, docker volume rm canopy_clamav_db so the next boot re-initializes.

Freshness is enforced service-side, fail-closed: ClamdScanner parses the definition date from clamd’s VERSION line; older than CANOPY_APPLICATIONS__SCANNER_MAX_DEFINITION_AGE_DAYS (default 7) fails the scan — uploads stay pending, the backlog gauges climb. Recovery: restore freshclam’s egress (or a private mirror), wait for the daemon’s next check to update the volume; the workers drain the backlog automatically once clamd serves fresh definitions.

Boot ordering (#1522): the image’s stock init starts freshclam and clamd concurrently, and a definition swap that lands while clamd is mid-load leaves clamd pinned to the OLD inode — its SelfCheck baseline stats the new file already in place, reports "Database status OK" forever, and the reload never comes (waiting, the previous recovery advice here, does not work for this mode). The canopy entrypoint (devstack/clamav/entrypoint.sh) closes it: one bounded freshclam pre-pass completes BEFORE clamd loads, so there is nothing left to swap; the daemon’s later checks notify through the then-present socket normally. If a container somehow still serves definitions older than the volume’s (VERSION date vs ls /var/lib/clamav/daily.cld), the fix is a container restart — clamd reloads from disk on boot.

The pre-pass is bounded (FRESHCLAM_PREPASS_TIMEOUT, default 120 s) and fails OPEN on boot — an airgapped or slow-mirror stack still starts on its on-disk definitions; the upload path stays fail-closed regardless via the service-side gate above (the control that matters).

Production must decide freshclam’s egress path explicitly (direct database.clamav.net, or a private mirror for air-gapped deployments) and size start_period accordingly — the pre-pass rides the same egress.

Symptoms: canopy_applications_scan_pending_count and canopy_applications_scan_oldest_pending_age_seconds climb (WARN past 1h); worker logs show scan deferred with backoff.

clamd down → restore it; nothing else is needed. Deferred rows retry with exponential backoff (60s×2ⁿ, cap 6h) and the claim budget (SCAN_MAX_ATTEMPTS, default 8 claims ⇒ 7 real scans) converges permanently-failing rows to terminal error.

Terminal error rows: recover with canopy application document-rescan --application-id … --document-id … (or the service-token POST …/rescan) — never direct SQL. Rescan resets the row pending under a bumped scan generation and REVOKES any standing acceptance (the verdict is in doubt).

CANOPY_APPLICATIONS__SCAN_WORKER_CONCURRENCY=0 is the operator kill switch: boot logs ERROR, canopy_applications_scan_worker_disabled=1, every upload strands pending until re-enabled.

Compose pins mem_limit: 4g (official guidance 3–4 GiB; reloads spike), cpus: 2, pids_limit: 256, MaxThreads 4 / MaxQueue 16 in clamd.conf (≥ replicas × SCAN_WORKER_CONCURRENCY with headroom). OOM kills during reload → raise mem_limit before anything else.

clamd TCP is unauthenticated and unencrypted: never expose it beyond the service network. The devstack publishes loopback-only (host-lane tests); production should not publish at all. The container runs unprivileged (/init-unprivileged, USER clamav).