Plan: BFF Edge Security — Working Per-IP Rate Limit, HSTS, and Session-Fixation Defense

On this page
NOTE

Authored from a code-grounded investigation. The headline finding is bigger than #625 describes: the per-IP rate limiter that #625 wants to "add to canopy-web" is itself broken for every service. ApiServer::serve binds with axum::serve(listener, router) and never calls into_make_service_with_connect_info, so ConnectInfo<SocketAddr> is absent from request extensions, and rate_limit_middleware falls back to keying every request as 127.0.0.1 — one global bucket for the whole service since inception (crates/canopy-api/src/lib.rs). Copying that layer into canopy-web without fixing the ConnectInfo wiring would faithfully reproduce the global-bucket bug. The fix is in the shared crate and touches all 18 services + canopy-web — see Blast radius + the verification that matters.

#550 is grouped here because it hardens the same canopy-web authentication edge (session fixation), shares the verification surface (auth-flow E2E), and is small.

Status

Step Description Status

#625 — working per-IP rate limit + HSTS

1

canopy-api: wire ConnectInfo<SocketAddr> into ApiServer::serve so the per-IP limiter actually keys per IP (shared-crate fix — all 18 services + canopy-web).

Done (2026-06-04) — into_make_service_with_connect_info; integration test through real serve() (fails if ConnectInfo regresses).

2

canopy-web: add the HSTS response header layer to the hand-built router.

Done (2026-06-04) — static Strict-Transport-Security layer (matches canopy-api).

3

canopy-web: add a per-IP rate-limit layer (reuse the canopy-api machinery, now ConnectInfo-correct).

Done (2026-06-04) — new canopy_api::apply_rate_limit; settings.rate_limit_rpm (default 6000).

#550 — session-fixation defense

4

canopy-web: rotate the session ID (cycle_id) + regenerate the CSRF token on authentication success.

Done (2026-06-04) — cycle_id + csrf::rotate_csrf_token in /auth/callback success path.

5

Tests: per-IP limiter keys distinct IPs to distinct buckets; HSTS header present; auth rotates session id + CSRF token.

Done (2026-06-04) — #625a serve/per-IP test + rotate_csrf_token unit test; HSTS static layer; cycle_id via E2E auth-setup.

6

Docs + CHANGELOG + GitLab issue updates.

Done (2026-06-04) — CHANGELOG entries (#625a/#625b/#550); #550 auto-closed via !494, #625 closed with resolution note after !492/!493.

Issues: #625, #550
Branches: fix/625a-connectinfo-rate-limit (Step 1 — shared crate, ships + validates first), fix/625b-canopy-web-hsts-ratelimit (Steps 2-3, depends on Step 1), fix/550-session-fixation (Step 4). Three MRs.

Context

#625 — canopy-web (the worker-portal BFF) is missing two edge protections the JSON API services nominally have: HSTS (Strict-Transport-Security) and per-IP rate limiting. canopy-web builds its own axum router (it does not go through ApiServer::build, which is what applies those layers to the JSON services) and only borrows ApiServer::serve to bind the socket (services/canopy-web/src/main.rs:316). So it ships neither layer. The investigation into "add the limiter to canopy-web" uncovered that the limiter is non-functional everywhere: ApiServer::serve never wires ConnectInfo, so the per-IP key is always the loopback fallback. This is a latent ATO finding (the rate limit exists in config and code but does not actually limit per-IP).

#550 — On authentication success, canopy-web reuses the same session record (and its CSRF token) that existed before login. An attacker who can fix a victim’s pre-auth session identifier (session fixation) would have a valid identifier into the now-privileged session. The standard defense is to rotate the session ID at the privilege boundary and regenerate the CSRF token; canopy-web does neither today (no cycle_id call exists anywhere in the tree).

Both are canopy-web edge-security hardening, share the auth-flow E2E verification surface, and are individually small — grouped to land as a coherent security pass.

Scope

In scope:

  • #625: ConnectInfo wiring in ApiServer::serve (the real fix); HSTS + per-IP rate-limit layers on canopy-web; verification that the now-working per-IP limiter does not throttle legitimate internal/E2E traffic.

  • #550: cycle_id + CSRF-token regeneration in the canopy-web /auth/callback success path; a reusable csrf::rotate_csrf_token helper.

Out of scope:

  • canopy-portal (applicant BFF) — it already wires ConnectInfo correctly (services/canopy-portal/src/main.rs:212-214) and runs its own working ratelimit module; its sessions are Redis-primary opaque tokens minted fresh per login (ADR-026), so the tower-sessions fixation vector does not apply. No change.

  • Per-route or per-user rate-limit tiers, distributed/Redis-backed rate limiting, or making the rate limit configurable per service beyond the existing rate_limit_rpm. (The distributed/Redis-backed deferral was the scale audit’s M8 finding — a process-local edge limit silently multiplies by replica count; delivered 2026-07-29 as #1227: canopy_api::apply_rate_limit_redis, the ADR-026 fixed-window pattern behind canopy-api’s rate-limit-redis feature, consumed by canopy-web.)

  • Changing the CSRF validation model (SameSite=Strict + _csrf/X-CSRF-Token) — only the rotation of the token on login.

Design

Part A (#625) — make the limiter real, then extend to canopy-web

A1 — ConnectInfo in ApiServer::serve (the root fix)

Current (verified on main):

  • ApiServer::serve — crates/canopy-api/src/lib.rs:190-201: axum::serve(listener, router).with_graceful_shutdown(shutdown).await. No into_make_service_with_connect_info.

  • rate_limit_middleware — crates/canopy-api/src/lib.rs:402-445: reads ConnectInfo<SocketAddr> from extensions (line 410-413); when absent, socket_ip = None; with no trusted proxies, ip = socket_ip.unwrap_or(127.0.0.1) (line 429) → every request keys to 127.0.0.1 → one shared governor bucket.

  • The rate-limit layer is applied in ApiServer::build (lib.rs:131-134), so all 18 JSON services + canopy-web (which serves through ApiServer::serve) are affected by the serve-level wiring.

  • canopy-portal already does the correct thing (main.rs:212-214): router.into_make_service_with_connect_info::<SocketAddr>().

Fix: change serve to bind with connect-info:

axum::serve(
    listener,
    router.into_make_service_with_connect_info::<SocketAddr>(),
)
.with_graceful_shutdown(shutdown)
.await

That single change makes ConnectInfo<SocketAddr> present for every request, so rate_limit_middleware keys on the real peer IP (or the x-forwarded-for client when the peer is a configured trusted proxy — that logic at lib.rs:415-430 already exists and only ever worked on paper).

A2 — HSTS on canopy-web

canopy-web’s hand-built router (services/canopy-web/src/main.rs:280-313) already stacks several SetResponseHeaderLayer`s (lines 302-313) and the strict CSP. Add the same HSTS layer the JSON services get (`crates/canopy-api/src/lib.rs:175-180):

.layer(SetResponseHeaderLayer::if_not_present(
    axum::http::header::STRICT_TRANSPORT_SECURITY,
    axum::http::HeaderValue::from_static("max-age=31536000; includeSubDomains; preload"),
))

Use if_not_present (matches canopy-api; lets a fronting proxy override). TLS terminates at the proxy, so the static one-year preload value is correct by deployment contract (same rationale as the canopy-api comment).

A3 — per-IP rate limit on canopy-web

canopy-web serves through ApiServer::serve but builds its own router, so it does not carry the rate_limit_middleware layer. Two options:

  • Preferred — reuse the canopy-api machinery (DRY; canopy-web already depends on canopy-api). Promote the currently-private items to pub: build_rate_limiter, rate_limit_middleware, the KeyedLimiter type alias, and ensure TrustedProxies is pub (it is consumed as an Extension). Then in canopy-web’s router add, near the other `route_layer`s:

    .layer(axum::Extension(canopy_api::build_rate_limiter(settings.rate_limit_rpm)... ))
.layer(axum::Extension(trusted_proxies))
.layer(axum::middleware::from_fn(canopy_api::rate_limit_middleware))

    (mirror the wiring shape canopy-api uses at lib.rs:131-134; build_rate_limiter returns Option, so guard the same way — no layer when rpm is 0.)

  • Alternative — mirror canopy-portal’s ratelimit module (services/canopy-portal/src/ratelimit.rs). Avoids changing canopy-api’s visibility but duplicates the limiter. Choose this only if promoting canopy-api internals proves to pull in unwanted coupling.

Recommend the preferred option and record the decision in the MR. Either way the limiter is now ConnectInfo-correct because of A1.

Blast radius + the verification that matters

A1 flips every service’s limiter from "one global bucket" to "true per-IP". The risk is internal traffic: orchestrator fan-out, E2E bursts, and service-to-service calls share a small set of docker-network source IPs, so they now share per-IP buckets they previously didn’t (everything was one bucket before, so this is not strictly worse — but a single internal IP doing 6000+ rpm during E2E could now 429 where the global bucket’s headroom previously absorbed it). Required checks before merge:

  1. Run the full E2E suite (cargo xtask e2e) against the rebuilt stack — it is the realistic burst test. A 429 regression shows up as fl‐aky/failed specs.

  2. Confirm the default rate_limit_rpm (6000, lib.rs:65) is comfortably above peak per-IP internal rate during E2E; if not, raise the internal default or exempt /healthz//metrics and internal callers (they already may bypass — verify against `ApiServer::build’s layering order).

  3. Verify trusted-proxy handling end-to-end in the devstack (the proxy’s IP must be in CANOPY_TRUSTED_PROXIES for x-forwarded-for to be honored; otherwise the limiter keys on the proxy IP and re-creates a near-global bucket for proxied traffic — call this out in the deployment docs).

This is the part the issue does not mention and the part most likely to bite; treat the E2E run as a gate, not a formality.

Part B (#550) — session-fixation defense

Current (verified on main):

  • CSRF token lives in the session under the private key csrf_token (services/canopy-web/src/csrf.rs:20); get_or_create_csrf_token mints one lazily.

  • /auth/callback (services/canopy-web/src/auth/mod.rs:170-354) exchanges the code, builds SessionData (line 309-323), store_session(&session, &data) (line 325), cleans up OAuth scratch keys (line 336), redirects. It reuses the existing session record — no ID rotation, no CSRF regeneration.

  • tower-sessions 0.14 (Cargo.toml:219) provides Session::cycle_id(); no current call site in the tree.

Fix: in the callback success path, at the privilege boundary, rotate the ID and the CSRF token. Add a helper to the csrf module so the key stays encapsulated:

// services/canopy-web/src/csrf.rs
/// Invalidate the current CSRF token so the next `get_or_create_csrf_token`
/// mints a fresh one. Call at any privilege transition (login) to ensure a
/// pre-auth token cannot carry into an authenticated session (#550).
pub async fn rotate_csrf_token(session: &Session) {
    session.remove::<String>(CSRF_SESSION_KEY).await.ok();
}

Then in callback, immediately after the tokens validate and before (or right after) store_session:

// #550: defeat session fixation — rotate the session identifier at the
// authentication boundary so a pre-auth (attacker-fixed) session id cannot
// be reused once the session becomes privileged.
if let Err(e) = session.cycle_id().await {
    tracing::error!(error = %e, "failed to cycle session id on auth");
    cleanup_oauth_flow_state(&session).await;
    return Redirect::to("/login").into_response();
}
crate::csrf::rotate_csrf_token(&session).await;

Placement: after the SessionData is built and validated, so the privileged data and the new ID are persisted together by the session layer at response time. The existing double-redirect for SameSite=Strict (/auth/landing, line 351-353) sets the new-ID cookie correctly — no change needed there. The OAuth scratch-key cleanup (line 336) stays.

NOTE
cycle_id changes the record ID but preserves the session’s data map, so values written via store_session survive the cycle. Regenerating the CSRF token is a separate, deliberate step (a fixed pre-auth CSRF token would otherwise remain valid).

Steps

Step 1: ConnectInfo in ApiServer::serve (#625 root)

Files: crates/canopy-api/src/lib.rs

Change serve to router.into_make_service_with_connect_info::<SocketAddr>(). Add/extend a unit or integration test that asserts two distinct peer IPs map to distinct governor buckets (or, at minimum, that ConnectInfo is present in a served request). Run the full E2E gate (see Blast radius + the verification that matters).

Step 2: HSTS on canopy-web (#625)

Files: services/canopy-web/src/main.rs

Add the STRICT_TRANSPORT_SECURITY SetResponseHeaderLayer::if_not_present alongside the existing header layers (~lines 302-313).

Step 3: per-IP rate limit on canopy-web (#625)

Files: crates/canopy-api/src/lib.rs (visibility), services/canopy-web/src/main.rs, services/canopy-web/src/config.rs (if a rate_limit_rpm setting is needed)

Promote the canopy-api rate-limit machinery to pub; wire the limiter + TrustedProxies extensions + rate_limit_middleware into canopy-web’s router. Add a rate_limit_rpm to canopy-web settings (default 6000) if not already present.

Step 4: session-fixation defense (#550)

Files: services/canopy-web/src/csrf.rs, services/canopy-web/src/auth/mod.rs

Add rotate_csrf_token. Call session.cycle_id() + rotate_csrf_token in callback success path per Part A (#625) — make the limiter real, then extend to canopy-web.

Step 5: tests

Files: crates/canopy-api/ tests, services/canopy-web/ tests (+ E2E)

  • canopy-api: two distinct ConnectInfo IPs → independent rate-limit buckets (one IP exhausts its quota without 429-ing the other).

  • canopy-web: response carries Strict-Transport-Security; rate-limit layer present (smoke).

  • #550: a session id present before /auth/callback differs after success; the post-auth CSRF token differs from a pre-auth-planted one. If full OIDC callback is hard to unit-test, assert the helper behavior (rotate_csrf_token clears the key) + an E2E that logs in and checks the session cookie changed.

Step 6: docs + issue updates

  • CHANGELOG.adoc — entries under == Unreleased.

  • Antora security-operations page (Security Operations) and the Security cheat-sheet: document that per-IP rate limiting now actually keys per-IP, the trusted-proxy requirement, HSTS on canopy-web, and session-id rotation on login.

  • Update GitLab #625 (correct framing: the limiter was globally bucketed; the fix is the ConnectInfo wiring, not just "add to canopy-web") and #550 (link plan).

Files Touched

File Change

crates/canopy-api/src/lib.rs

into_make_service_with_connect_info in serve; promote rate-limit machinery to pub.

services/canopy-web/src/main.rs

HSTS layer + rate-limit layer/extensions on the BFF router.

services/canopy-web/src/config.rs

rate_limit_rpm setting (if absent).

services/canopy-web/src/csrf.rs

rotate_csrf_token helper.

services/canopy-web/src/auth/mod.rs

cycle_id + CSRF rotation in /auth/callback success.

tests (canopy-api, canopy-web, E2E)

Per-IP bucket isolation; HSTS header; session/CSRF rotation.

CHANGELOG.adoc, Security Operations, Security

Document the corrected posture.

Verification

  1. cargo nextest run -p canopy-api -p canopy-web --lib — unit tests pass.

  2. cargo xtask validate — fmt + clippy + docker build clean.

  3. cargo xtask e2e — gate for the rate-limit blast radius (no new 429-driven flakes); auth-flow E2E exercises the session rotation path.

  4. Manual: curl -I a canopy-web page → Strict-Transport-Security present. Hammer an endpoint from one IP past the quota → 429 for that IP while a second IP still succeeds (proves per-IP, not global). Log in → browser session cookie value changes across /auth/callback.

Documentation Updates

  • CHANGELOG.adoc — Unreleased entries.

  • Antora security-operations page (Security Operations) — rate-limit (now per-IP), trusted-proxy requirement, HSTS on canopy-web, login session rotation.

  • Security cheat-sheet — posture quick-reference if it covers rate limiting.

  • GitLab #625 / #550 — link plan; correct #625 framing.

Edit this page · default