UX testing strategy — dimension-derived grids, human benchmarks, automated expansion

Task QA-UXGRID1 · drafted 2026-07-16 from the user's brief (post BOARD-PHYSICS1 plateau 3115272) · builds on the QA-TESTDIM1 dimension registry (dimensions.html) and the QA-OVERVIEW machinery (kflow / katrun / retest grid). Plain language first, jargon in brackets — same convention as QA-OVERVIEW.

1 · The idea in one paragraph

We stop writing test cases one by one. Instead we define, per behaviour we care about, a test class (what's being proven, and how "OK" is judged) and a set of dimensions (the axes that could change the outcome: player count, aspect ratio, device, what's being dropped where…). Individual test instances are then derived from dimension combinations — never hand-enumerated. Because full combination counts explode ("a stupid crazy number"), we use the industry's standard trick (pairwise testing, §3) to shrink thousands of combinations to dozens that still catch almost everything, plus the known turning points (boundary values) that history says matter — e.g. player count at exactly the ring cap, and square-vs-portrait aspect (a square screen is what hid the circular-walls bug for weeks). A human (you) runs a cherry-picked handful of instances per class and blesses them — the recorded kdiag trace + screenshots become the class benchmark ("this is expected behaviour"). The expanded set runs automated, replays the same steps across the other combinations, checks itself against hard assertions + the benchmark, and reports pass/fail into kflow — the same retest grid the backend robots already report to.

2 · Where we stand — the tooling we already have

The striking gap: everything below the browser is systematised; everything inside it is artisanal. The pieces exist — they've just never been joined into a repeatable, reported loop.

3 · The industry frame — and where we deliberately deviate

The parts of standard practice worth adopting wholesale

• Equivalence partitioning (ISTQB staple): don't test every value, test one per class of values that behave the same. Our "dimensions with a handful of values each" is exactly this — the registry in qa-test-dimensions.html already thinks this way.
• Boundary-value analysis: bugs cluster at edges. Ours provably do: players at exactly maxSegments vs one over (wedge dividers appear), obtuse vs acute segment angle (the ±80° clamp), square vs non-square aspect (hid the circular-walls bug), 0-ghost vs 1-ghost stores (band sizing). Every dimension below marks its boundaries — they're always in the picked set.
• Pairwise / combinatorial selection: the empirical result behind it (NIST's field-failure studies) — the large majority of real-world failures are triggered by an interaction of just two parameters, almost all by three. So instead of the full cartesian product you generate the smallest instance set where every pair of values across every two dimensions appears at least once. Typical shrinkage: our worked example below goes from 7,776 combinations to ~25 pairwise instances to 4 manual picks. Established tools exist (Microsoft PICT, NIST ACTS), but the algorithm is simple enough that we generate ours in a few lines — with constraints (some combos are invalid: "drop on a hidden segment" requires players > cap; the generator must know that).
• Risk-based prioritisation: what you called cherry-picking. The manual set is chosen where judgement is irreplaceable (animation feel, visual balance) or risk is highest (mainnet-adjacent flows).
• Regression-to-canary habit: every past incident becomes a permanently-running named test. The backend already lives this (the zombie canary); §6 extends it to the frontend's actual bug history.

Where we break from the classical mould (the AI-era adjustments)

Classical UX automation assumes tests are expensive to write and brittle to keep — whole industries (page-object patterns, selector-maintenance teams) exist to manage that cost. That assumption is now false here: this week, every bug fix shipped with a purpose-built headless test written in minutes by the agent that wrote the fix (overlap audit, oval invariants, trail tracking…). Three consequences we lean into:

• Tests are generated, not authored. The dimension file is the source of truth; instance scripts are derived from it. When a dimension changes (a new layout mode), the grid regenerates — no stale hand-written case library.
• The human's scarce contribution is the benchmark, not the script. You play the cherry-picked instance once, with kdiag recording and screenshots at defined moments; you say "this is right". That blessed bundle — steps, trace, images — is the oracle the machine replays against. (Classically called golden master / approval testing; we add the diagnostic trace, which classical UI testing never had.)
• Fuzzy visual judgement can now be automated — carefully. A vision model comparing "screenshot now" vs "blessed screenshot, same instance" answers "has this materially changed?" better than pixel-diffing (which cries wolf on antialiasing) and worse than you (which is why the benchmark is yours). We propose it only as a change detector that escalates to a human, never as the judge of first instance. This replaces the classical visual-regression services (Percy/Chromatic) with something we control and that understands "the chips may sit anywhere inside the walls" — a semantic tolerance no pixel-differ has.

4 · The model: classes → dimensions → instances → benchmarks → canaries

The lifecycle: a class is defined → the generator derives instances → you run + bless the picks (benchmarks) → the automated expansion replays the rest → all verdicts land on the kflow grid → any incident adds a canary instance to the class. "Something surprises you → pin it down → leave a robot watching" — QA-OVERVIEW's loop, now for the UI.

5 · Worked example — the CHIPDROP class

5.1 Oracle (what "OK" means, checkable)

• Hard assertions (machine-checkable, no judgement): the chip lands inside the target's basket walls (the same boardRimAt/boardWallClamp functions the physics uses — the walls overlay made these an oracle); a ghost sits on the giver's rim band; pile→ghost→basket endpoints stay within 4px of their chips through and after all motion; giver's store balance decrements; solid-chip invariant (0 overlapping pairs at settle); jitter 0 after settle; no console errors; board:paint:moves goes quiet.
• Trace shape (kdiag): the expected checkpoint sequence for the gesture (drag events → offer sent → render → settle), no unexpected re-solves.
• Benchmark comparison (for the visual/feel part): screenshots at gesture-start / mid-flight / settled vs the blessed run.

5.2 Dimensions (with boundaries and constraints)

Full cartesian: 4·2·3·3·4·3·3·3 = 7,776. Pairwise with constraints: ≈25. Picked for hand-running: 4.

5.3 A derived instance set (illustrative pairwise sample — the generator produces the full ~25)

Note how the manual picks are exactly where machine judgement is weakest: feel of the everyday case, the two signature animations, sticky's game-feel, and the one requiring a mid-flow human gesture sequence. Everything else asserts fine mechanically. Device rows automate as viewport+touch emulation first (Playwright device profiles); real-device passes stay manual until a device farm is ever justified.

6 · The regression catalogue — history as canaries

Mined from the last month of tasks/incidents. Pattern-named so future bugs slot into a category (and inherit its canary shape):

Immediate cheap win: six of ten categories already have working checks sitting in a session scratchpad. Promoting them into a versioned qa/ folder with a one-command runner banks this week's work before it evaporates.

7 · Harness & reporting — proposal and choices

The shape (recommended)

• Source of truth: one machine-readable file per class (qa/cases/chipdrop.json: oracle spec, dimensions, values, boundaries, constraints, picked-manual list). The human registry (qa-test-dimensions.html) stays the readable overview and links to these; the runner and the docs derive from the same file — no drift.
• Runner: Playwright (chromium already on this box, proven against lab1 all week). Fixtures by class: ?demo=1 for pure-frontend classes; katrun-driven live backend for e2e classes; XPROOF headless-WC bot for signing legs (QA-WCTEST1). Device dimension via Playwright's device profiles (touch, DPR, viewport).
• Reporting: every run emits kflow test:start / test:verdict with {class, instance, commit, seat} — UX results land on the same retest grid as katrun's, no new dashboard. Red verdicts follow the same alert path QA-OVERVIEW's next-steps call for.
• Benchmark recorder: a tiny "record mode": you play a picked instance on lab1, it captures the kdiag stream + screenshots at the oracle's checkpoints into qa/benchmarks/<instance>/; blessing = committing it. Automated runs attach their equivalents to failures so a red cell always carries its evidence.
• Triggers: (1) on demand — one command per class or instance; (2) a cron loop on the lab seat (the katrun heartbeat pattern — e.g. the canary set every 30 min against lab1); (3) later, post-checkpoint hook (a commit to the board triggers the board classes).

The choices for you to pick (A/B/C are not mutually exclusive)

Recommendation: A + C now, B later. A gives the dependable grid; C is what makes the "benchmark" idea pay off (your one manual pass keeps guarding the class after you've moved on); B becomes attractive once A is green and boring. On reporting there's no real choice to make — kflow is built for exactly this and katrun has proven the pattern.

8 · Proposed next steps (each small)

1. Bank the existing checks (sand/stone): move this week's five scratchpad harnesses into qa/ on the frontend repo with a one-command runner. Instant regression cover for the physics board.
2. First class file (stone): qa/cases/chipdrop.json per §5 + the pairwise generator (small, constraint-aware). You review the derived ~25 and the 4 picks.
3. kflow wiring (stone): test:start/test:verdict from the runner; confirm cells appear on the lab grid. (Depends on nothing frontend-specific — the checkpoint API already exists.)
4. First benchmark recording (stone, with you): CHIPDROP-01 on your tablet against lab1, kdiag + screenshots captured and blessed.
5. Hydration click-integrity canary (stone, HIGH): the SIGN-LAYOUTJUMP1 bounding-box watch — the one category whose failure class is dangerous rather than cosmetic.
6. Then: grow classes (SIGNING, ROTATION, SETTINGS-PERSISTENCE…) at the rate reality demands — each new incident lands in a category and leaves a canary.

9 · Task cross-references

Maintained under QA-UXGRID1 · first draft 2026-07-16, seat lab1 (KATUI2) · sibling of dimensions.html · plain-first tone per QA-OVERVIEW convention.