UAT Facilitator Guide

On this page

Audience

UAT facilitators — typically eligibility specialists or QA leads at the deploying jurisdiction — running end-to-end test sessions with caseworkers, supervisors, and applicants ahead of go-live. SNAP UAT (September 2026) is the immediate target; the same patterns apply to TANF / Medicaid / CAPS / WIC UAT post-launch.

For developer-facing test infrastructure see testing.md and known-issues.adoc. For the underlying eligibility logic see federal-requirements.adoc.

Test environment setup

Provisioning

UAT runs against a deployed devstack. The deploying jurisdiction provisions its own environment per deployment-guide.adoc. For local dry-runs:

cargo xtask dev start --profile snap-only      # SNAP UAT only
cargo xtask dev start --profile full            # all programs (post-UAT)
cargo xtask dev refresh                         # auto-detect changes, minimum rebuild

Wait for All services healthy before starting any test session. The troubleshooting page lists common cold-start gotchas.

Seed data

tools/canopy-seed/ populates a deterministic test dataset spanning the eligibility surface:

  • Demographic variety: single-adult, two-parent, three-generation households; ages spanning the ABAWD (18-59) and elderly (60+) thresholds; mixed citizenship statuses where the SAVE adapter is exercised.

  • Income variety: zero-income, partial-FPL, near-130%-FPL (gross income test), categorically eligible (TANF/SSI), self-employed, mixed earned/unearned.

  • Verification states: IEVS-matched, IEVS-mismatched (income discrepancy), pending, expired.

  • Application states: open, in-progress, determined-approved, determined-denied, appealed, renewed.

The seed fixture is reproducible — running cargo xtask seed reset returns the database to a known state. UAT facilitators can re-seed between scripted scenarios.

Test user credentials

Worker/admin test users are seeded into the Keycloak canopy realm by devstack/keycloak/canopy-realm.json (imported at container start via start-dev --import-realm). Roles map to the RBAC matrix. Every human user shares the password password (a dev fixture committed to source — rotate before any non-dev deployment). Applicants do not use a seeded role/password (see the Applicant scenarios below).

Username Realm role(s) Use for

jane.caseworker

caseworker

Intake, read-side flows, appeal filing on an applicant’s behalf. The identity the E2E journey project authenticates as (tests/e2e/auth/setup.ts).

jane.doe

caseworker, eligibility_specialist, supervisor

Full-privilege worker (resolves to Supervisor precedence in the portal) — determination, signing, and approval flows.

jane.supervisor

supervisor

Approval workflows; report generation; sanctions.

jane.snap-worker / jane.tanf-worker

caseworker

Program-team caseworkers (SNAP / TANF queues).

jane.qc

analyst

Audit-event search; FNS-7176 QC universe; read-only across cases.

fti.auditor

fti_auditor

FTI (federal tax info) auditor endpoints (Pub-1075).

data.steward

data_steward

Redaction / crypto-shred expungement ops (ADR-036).

jane.admin (or admin)

admin

System administration; rare in normal UAT but exercise for system-config tests.

applicant.test

applicant

A seeded applicant role — but applicant-portal login is by reference number + passcode, not this user (see the Applicant scenarios).

All human users have password password. Rotate before any non-dev deployment — the realm export (with its dev password) is committed to source.

Role-based test scenarios

Each scenario is a self-contained workflow facilitators run with a participant. Time estimates assume the participant is already familiar with the role; first-time exposure adds 50-100%.

Caseworker (15-25 min per scenario)

  1. Intake — single-adult SNAP applicant: log in → search for applicant by SSN → start new application → enter household + income → confirm expedited screening → save as draft → return tomorrow → resume → submit.

  2. Verification triage: log in → open queue of IEVS discrepancies → review one mismatch → call applicant to clarify → record outcome → escalate to supervisor if unresolved.

  3. Notice retrieval: log in → search applicant → open notices tab → download most recent NOA → read PDF in viewer → confirm content matches expected determination.

  4. Appeal filing: log in as caseworker on behalf of applicant who walked into the office → file fair-hearing appeal → record continued-benefits decision → confirm 90-day clock starts.

Eligibility specialist (20-40 min per scenario)

  1. Determination — happy path: log in → open submitted application → run determination → review SNAP allotment → review benefit calculation breakdown (max allotment, deductions, 30% net contribution) → verify signed determination persists → confirm determination.completed.snap event fired.

  2. Determination — denial: log in → open application with gross income at 135% FPL → run determination → confirm denial reason (gross_income_exceeds_130pct_fpl) → verify NOA generated with denial reason + appeal rights.

  3. Cross-program orchestration: log in → open household with children → run eligibility for SNAP + TANF + Medicaid → confirm Medicaid assigned_coa populates correctly via EE15 hierarchy → review combined result.

  4. ABAWD time-limit edge case: log in → open ABAWD household → review work-activity ledger → confirm month 3 with no qualifying work triggers AbawdExceeded status → verify continued-benefits if appeal pending.

Supervisor (15-30 min per scenario)

  1. Approval workflow: log in → review subordinate’s queued determinations → override one denial with documented reason → sign override → confirm audit-trail entry.

  2. Personal-responsibility sanction (TANF): log in → review caseworker’s sanction recommendation → approve → confirm sanction notice generated → verify cross-program disqualification (if applicable to SNAP).

  3. Chain-verification audit (unified /v1/security/chain/* namespace, #1205 — dormant until #1279, so every step here verifies the FAIL-CLOSED posture): with an admin token (or the CLI: canopy security chain-status --family audit) → confirm GET /v1/security/chain/status?family=audit returns 503 with a typed body — state: "unknown", reason verifier_disabled (never a 200 "intact" while the verifier is dormant) → trigger a manual verify (canopy security chain-verify --family audit, i.e. POST /v1/security/chain/verify) → confirm 503 verifier_unavailable (no job is queued for an unconfigured target) → attest one audit event (canopy security chain-attest --event-id <uuid> --family audit) → confirm attested: false, reason verifier_unavailable → finally confirm the unified FTI arm GET /v1/security/chain/status?family=fti&service=canopy-tanf (#1206 MR-3; the legacy fti/chain-status path is deleted) returns the typed 503 — state: "unknown", reason verifier_disabled — pre-cutover, and that with a seeded legacy v1 breach row it instead reports state: "breached", reason legacy_breach_latched (the latched #1245 posture is never silently swallowed, even while dormant). Post-#1279 UAT re-runs this scenario expecting 200 healthy + a 202 job + attested: true.

  4. Federal report generation: the report surfaces are user-only under the OIDC receiver contract (#1438) — a raw supervisor bearer is rejected; the caller must hold an exchanged aud=canopy-reporting token (no worker-portal UI drives these surfaces yet, so UAT uses the devstack test client, which performs the exchange): run the reporting suite’s generate flow (or ReportingClient with an exchanged token) → generate FNS-388 for the prior month → confirm CSV downloads → spot-check one row against case detail.

Quality control reviewer (10-20 min per scenario)

  1. QC universe pull: user-only surface (#1438 — same exchanged-token requirement as the FNS-388 scenario above): trigger the FNS-7176 QC universe extract through the devstack test client → confirm CSV format matches FNS spec → spot-check 3 rows against canonical eligibility logic.

  2. Audit-event search: log in → search recent determination.completed events → filter by program → export to CSV → confirm hash chain integrity (no audit.chain.breach_detected).

  3. Cross-program review: log in → search a household with both SNAP + TANF determinations → confirm TSNAP transition created if employment closure.

Applicant (10-20 min per scenario)

The constituent-facing canopy-portal (Dioxus fullstack, ADR-008) is shipped. Applicants do not use a seeded role/password — they authenticate with a reference number (HH-…) + 12-digit passcode. Use the default-seed cast from the applicant portal seed credentials runbook — HH-ca570002 / 4821-0073-9156 (submitted-with-verifications) for the action-needed walk, and HH-ca570004 / 7300-2914-8856 (the ELE case) for the cross-portal verification-gated thread (full stack).

NOTE
The applicant portal is a shipped surface; the SNAP program remains the in-scope determination program for September 2026 UAT (worker-side determinations cover SNAP). These applicant scenarios exercise the portal itself, independent of which program is UAT-gated.

  1. Reference-number login: open canopy-portal → /lookup → enter the submitted-with-verifications applicant’s HH-ca570002 + passcode 4821-0073-9156 → confirm landing on the authed Home with a personalised greeting and the correct case-state hero (this persona lands on the "Action needed" / active hero with a live open-verification count).

  2. New application (apply flow): from the unauthenticated portal → start a new application → walk the Apply steps (household + contact + program request) → confirm client-side-encrypted autosave persists across steps → finalize → confirm the credential reveal screen shows a real HH-… reference number + passcode.

  3. Document upload: log in as the ELE applicant (HH-ca570004 / 7300-2914-8856) → open /documents → upload an identity document for the pending request → confirm the upload appears as submitted/pending.

  4. Verification response: as the same ELE applicant → open /verifications → respond to the open identity verification → confirm the response is recorded (note: the response leaves the verification pending until a worker accepts it — that worker-side acceptance is what unblocks Run Determination).

  5. Letter / NOA view: after a determination, log in as the relevant persona → open /letters → confirm the Notice of Action (NOA) is listed and the PDF opens, with the case number rendered as HH-<last8>.

Data collection

What to observe

Observation What it tells you

Task completion time

Workflow efficiency. Compare across roles — caseworker intake should be <5 min once familiar; >10 min flags UI friction.

Error recovery

When the participant clicks the wrong thing, can they back out cleanly? Or do they have to start over?

Confusion points

Verbal cues ("I’m not sure what this does", reading aloud, hesitation). These point at unclear labelling, missing context, or wrong information architecture.

Workarounds

Notes-on-paper, screenshots saved offline, copy-paste between tabs — all signs the system isn’t doing the job for them.

Authentication friction

Token expiry, role mismatches, "forbidden" messages. These often surface RBAC misconfiguration that the dev environment didn’t catch.

How to record

Each session needs at minimum:

  • Screen recording (with explicit consent) — OBS Studio or Zoom record works. Capture audio so verbal cues are preserved.

  • Notes template — see below.

  • Severity classification — every issue gets a severity:

    • S1 Critical: blocks task completion; data loss; compliance violation (e.g. PII in URL).

    • S2 Major: completable with workaround; significant time penalty; documented inconsistency.

    • S3 Minor: cosmetic; copy/wording; non-essential UI polish.

Issues file as GitLab issues with the program::<X> and service::<Y> labels matching the affected surface, plus priority::<critical|high|medium> mapping to S1/S2/S3.

Notes template

# UAT session notes — <date> — <participant ID> — <role>

## Scenario: <name>
- Started: <hh:mm>
- Completed: <hh:mm | abandoned at <step>>

### Observations
- <step 1>: <what happened>; <verbal cue if any>
- <step 2>: ...

### Issues
| Severity | Description | URL/screen at moment of issue |
|---|---|---|
| S1 | ... | ... |

### Participant feedback (post-task)
- What worked well:
- What was confusing:
- What's missing:
- One thing they'd change:

Accessibility testing

Canopy targets WCAG 2.1 AA compliance enforced by axe-core in the E2E suite (tests/e2e/specs/accessibility.spec.ts). UAT facilitators verify that the automated coverage matches real-world use:

Screen reader protocol

  • Tools: NVDA (Windows, free), VoiceOver (macOS, built-in), Orca (Linux, free).

  • Sessions: at least one full caseworker scenario completed using only the screen reader. Record narration.

  • Watch for: unlabelled form fields, focus traps, dynamic content (modal dialogs, htmx swaps) not announced, image-only links.

Keyboard-only navigation

  • Constraint: cover one full eligibility-specialist scenario without touching the mouse.

  • Watch for: focus order matching visual order, focus visibility (clear outline), tab traps, modals closeable with Escape, action confirmations reachable.

Color contrast

axe-core covers AA contrast (4.5:1 normal text, 3:1 large text). Spot-check the dark theme — automated tests run light-mode by default; the accessibility-dark.spec.ts suite covers dark but real-eye verification catches things axe-core misses.

Feedback collection

Structured interview questions (5-10 min, post-session)

  1. On a scale of 1-5, how confident were you that you completed the task correctly?

  2. What was the single most frustrating moment?

  3. What was the single most surprising moment (positive or negative)?

  4. If you had to do this task 50 times today, what would slow you down?

  5. Is there anything missing that you’d expect a system like this to have?

Satisfaction rating scale (per scenario)

  • 1 — I would refuse to use this in production

  • 2 — I could do my job, but I’d be frustrated daily

  • 3 — Acceptable; I’d want some improvements but I could live with it

  • 4 — Good; I’d be productive and minor friction wouldn’t compound

  • 5 — Excellent; this makes my job easier than the current process

A scenario averaging <3 across 5 participants is a release blocker.

Reporting

After each UAT week:

  • Per-scenario success rate (completed / attempted) by role.

  • Issue counts by severity (S1/S2/S3) and by service (canopy-snap, canopy-eligibility, etc.).

  • Top-3 recurring confusion points — these are the highest-leverage UX fixes.

  • Accessibility gaps found by manual testing that automated suite missed.

Report goes to: project owner, security/compliance lead (for any S1 PII / RBAC issues), engineering lead.

S1 issues block the next UAT week. S2 issues are budgeted for the remaining sprints. S3 issues land on the post-UAT backlog.

Cross-references

Edit this page · default