Static pages are the easy case for accessibility tools. The interesting failures show up after the page mutates: a toast appears late, a modal steals focus incorrectly, a list recycles DOM nodes, or a screen-reader announcement never fires because the live region changed too early and then got overwritten.

This benchmark plan is for teams that want to compare an accessibility testing tool benchmark dynamic ui workflow on modern SPA behavior, not just on textbook WCAG violations. It separates two things that are often blended together:

  • Automated rule detection, for example missing labels, invalid ARIA, color contrast, and focus-order violations that a scanner can infer from the DOM.
  • Manual verification effort, the extra steps a human still needs to confirm whether the user-facing behavior is actually correct.

That distinction matters because a tool can score well on static rules and still be weak at dynamic interaction coverage. The goal here is not to crown one universal winner. It is to make signal quality visible for the UI patterns that break tools most often.

What this benchmark should answer

A useful benchmark for dynamic accessibility work should answer four questions:

  1. Does the tool catch the seeded defect automatically, or only after a human drives the interaction?
  2. When it flags something, is the finding specific enough to act on?
  3. Does the signal degrade on dynamic UI patterns like delayed announcements, focus traps, and recycled nodes?
  4. How much manual verification is still required before the team can trust the result?

For reference, WCAG is the standard frame of requirements, but the benchmark itself should measure tool behavior against reproducible scenarios, not against vague compliance claims. The relevant baseline is the WCAG standard.

A scanner that can only inspect the final DOM snapshot is not automatically weak, it is just solving a narrower problem than a workflow that validates interaction state over time.

Tools in scope

This plan is designed so it can be applied to tools in the same broad category without pretending they are identical.

The benchmark does not assume that every tool exposes the same execution model. Some are local libraries, some are browser extensions or auditing surfaces, and some sit inside a broader cloud workflow. That matters because the benchmark should record both what the tool finds and what it requires from the tester to surface the issue.

Test harness design

Build a small SPA test app with seeded defects. The app should be boring on purpose. Avoid unrelated complexity so the benchmark measures accessibility signal, not application behavior.

Harness requirements

Use at least these routes or components:

  • toasts, for ARIA live region announcements
  • modal, for focus trap and escape handling
  • feed, for infinite scroll or virtualized content
  • offscreen, for hidden but still focusable nodes
  • baseline, a control page with no seeded defects

Each page should include both a correct and an incorrect variant of the pattern. That lets you test false positives as well as misses.

Suggested seeded defects

Pattern Correct version Seeded defect
ARIA live region Message is placed in a polite or assertive live region after state change Announcement is updated too early, overwritten, or put in a region that is not exposed to assistive tech
Focus trap Focus cycles inside the modal until close Tabbing escapes the modal, or focus lands behind the overlay
Infinite scroll Newly loaded items are announced, keyboard focus remains stable New content appears without programmatic indication, or focus jumps unexpectedly
Virtualized list Offscreen items are removed from the accessibility tree when appropriate Recycled nodes keep stale labels, focusable descendants vanish while focused
Offscreen nodes Hidden content is removed from sequential focus order Invisible controls remain tabbable or discoverable by screen readers

Scoring model

Use separate scores for detection and verification, otherwise the comparison becomes misleading.

1. Detection score

For each seeded defect, record whether the tool:

  • flags the issue automatically,
  • flags it only after interaction,
  • does not flag it,
  • produces a false positive on the correct variant.

A simple rubric works well:

  • 2 points: correctly detects the defect with a specific finding
  • 1 point: partially detects it, or detects it only after manual interaction steps
  • 0 points: misses it
  • -1 point: false positive on the correct variant

2. Verification effort score

Record the number of human steps required to trust the result:

  • open page
  • trigger interaction
  • wait for async update
  • inspect generated findings
  • cross-check with keyboard or screen-reader behavior

The point is not to optimize for the fewest clicks at all costs. The point is to measure how much work remains after automation.

3. Debugging quality score

A useful finding should answer:

  • what failed,
  • where it failed,
  • what element or state caused it,
  • whether the issue is a likely real defect or a possible false alarm.

If a tool only says “accessibility issue found” without enough context, it is low value for dynamic UI work even if it catches the defect.

Benchmark procedure

Keep the procedure fixed across tools.

  1. Load the baseline page and confirm the harness is stable.
  2. Run the tool with default settings, then with the most defensible project settings documented by the vendor.
  3. Trigger each dynamic interaction in the same order.
  4. Capture findings immediately after each state change.
  5. Repeat the run after a fresh page load to reduce carryover effects.
  6. For browser-based or cloud tools, note whether the session state, viewport size, or browser version changes the result.

Example Playwright harness

A tiny interaction script can help standardize the state changes before a scan or manual review.

import { test, expect } from '@playwright/test';
test('exercise dynamic accessibility states', async ({ page }) => {
  await page.goto('http://localhost:3000/baseline');

  await page.goto('http://localhost:3000/toasts');
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByRole('status')).toContainText('Saved');

  await page.goto('http://localhost:3000/modal');
  await page.getByRole('button', { name: 'Open dialog' }).click();
  await expect(page.getByRole('dialog')).toBeVisible();
});

This is not the benchmark itself. It is the reproducible interaction layer that keeps each tool on the same state transition.

Where signal usually degrades

ARIA live regions

Live regions are hard because timing matters. A scanner that snapshots the DOM after the message has already been replaced may miss the original state transition. Another common failure is treating the visible toast as proof that assistive tech received an announcement.

Measure whether the tool can surface:

  • missing live-region attributes,
  • inappropriate use of aria-live,
  • announcements that appear in the wrong order,
  • duplicate or overwritten messages.

For this pattern, the human verification cost is often the real differentiator. If the tool cannot tell you whether the announcement is actually exposed to the accessibility tree, the finding is only partially useful.

Focus traps

A focus trap is not just a modal with a backdrop. The defect is behavioral, focus should remain constrained inside the dialog while it is open, and escape should restore the prior focus target.

A tool that checks only static DOM rules may catch missing dialog semantics, but not the trap itself. To measure this properly, include keyboard traversal and record whether the tool helps you confirm:

  • tab order inside the modal,
  • return focus on close,
  • no hidden focusable siblings outside the dialog.

If the tool cannot observe keyboard flow, the benchmark should mark this as a manual-verification dependency, not as a failure of the benchmark.

Virtualized and infinite-scroll content

Virtualized lists are where a lot of accessibility automation becomes noisy. Offscreen rows may be unmounted, recycled, or kept in the DOM with position changes. That can produce false positives if a tool treats every recycled node as a fresh accessibility object, or misses defects if it inspects only the currently rendered subset.

Check for:

  • stale accessible names on recycled rows,
  • focus loss when items are added or removed,
  • skipped announcements when the feed grows,
  • hidden nodes that remain focusable.

This is a good place to separate “correct after interaction” from “correct after scan.” A tool that reports a warning but cannot localize the row or state change is hard to use on a large list.

A compact decision table

The table below is not a ranking. It is a way to frame likely strengths before you run the benchmark.

Tool type Likely strength Likely weak spot Best use in this benchmark
Rule-focused local scanner Fast static checks, repeatable CI use Late announcements, keyboard flow, UI state transitions Baseline rule detection and regression gating
Browser audit surface Easy one-off inspection in a real browser Limited depth on async or stateful bugs Manual review of a suspected issue
Cloud accessibility workflow Centralized reporting and team visibility Harder to isolate whether a finding is from automation or a human step Multi-browser workflows and governance
Browser extension or guided inspector Good for developer debugging Not ideal for unattended CI Finding context on a single page

How to interpret results

Do not pick the tool with the highest raw detection count unless it also keeps false positives low on the correct variants. Dynamic UI accessibility work has a different cost profile than a static page scan.

A tool is strong for this benchmark if it does three things well:

  • catches the seeded defect without heavy ceremony,
  • provides enough context to debug the defect quickly,
  • keeps signal stable across repeated runs.

A tool is weak if it requires too much manual choreography just to surface the bug, or if its findings are so broad that the team has to re-implement the verification step by hand.

Practical recommendation by team shape

Choose a rule-first tool if

  • your team wants fast regression coverage in CI,
  • most defects are structural or ARIA syntax related,
  • engineers can fix findings directly from a concise report.

Choose a browser-based or cloud workflow if

  • the problem is intermittent state, cross-browser validation, or coordination across testers,
  • you need a shared workflow for QA, accessibility specialists, and developers,
  • the team can tolerate more setup in exchange for better visibility.

Choose a guided inspector if

  • you are debugging a specific modal, feed, or live-region issue,
  • you need high-quality local feedback before you automate anything,
  • you want to confirm keyboard behavior manually.

The correct tool is usually the one that matches your maintenance model, not the one with the longest feature list.

Failure modes to document in the benchmark report

Include these even when the tool “passes” the scenario:

  • browser version drift changes results,
  • viewport size affects whether the virtualized list loads enough content,
  • async rendering finishes after the scanner snapshot,
  • the same defect produces a different message after a refresh,
  • a tool reports a generic issue but not the event or node that caused it.

These are not edge cases to hide. They are the reasons dynamic accessibility work is expensive to automate well.

Not the best fit if

This benchmark plan is less useful if your application is mostly static content with rare modal dialogs. In that case, a simpler compliance-oriented scan may be enough, and the extra effort to build seeded dynamic fixtures will not pay back quickly.

It is also not the right starting point if your team has no reliable way to run browser automation in CI. Without a stable harness, you will end up comparing tool noise instead of tool signal.

Bottom line

If you are evaluating accessibility testing tools for modern SPAs, do not ask only whether they find WCAG issues. Ask whether they can still produce trustworthy signal after the UI changes under them.

This benchmark plan makes that visible by testing the patterns that break shallow audits first: late ARIA live-region updates, modal focus traps, virtualized lists, and offscreen focus leakage. The key output is not a single score. It is a clear separation between:

  • automated detection,
  • manual verification effort,
  • and the failure modes that remain after both.

That separation is what makes a comparison useful to QA engineers, accessibility specialists, frontend teams, and engineering managers who need a workflow they can actually maintain.

FAQ

Why not rely on a single automated accessibility scan?

Because dynamic accessibility bugs are often behavioral, not just structural. A scan can miss timing issues, focus movement, or stale virtualized nodes.

What makes ARIA live regions hard to benchmark?

The announcement can be correct in the app but invisible to a tool if the tool snapshots too late or never observes the event sequence that created the announcement.

Should focus traps be tested manually or automatically?

Both. Automation is useful for detecting missing dialog semantics and repeated regressions, but keyboard traversal is still the fastest way to verify trap behavior.

How do virtualized lists distort results?

They can hide offscreen rows, recycle DOM nodes, and change accessible names as items move. That can cause false positives or missed issues depending on when the tool inspects the DOM.

What is the most important benchmark artifact to keep?

Keep the seeded harness, the exact interaction script, and the scoring sheet. Without those, the comparison is not reproducible.