How to Benchmark Browser Test Stability on Apps With WebAuthn, Passkeys, and Device-Bound Login Steps

Passkeys made login better for users, but they also made Test automation more interesting in the bad old way: the sort of interesting where a flow can fail for several reasons that look identical from the outside. A WebAuthn ceremony can break because the product changed, because the browser cannot satisfy the authenticator request, because the device state is different, because the test runner is in the wrong security context, or because the app is asking for something the browser automation stack simply cannot emulate in a reliable way.

That is exactly why browser test stability for passkeys deserves its own benchmark plan. If you only count pass or fail, you end up blaming the wrong layer. If you measure where instability enters the flow, you can separate product bugs from browser capability gaps and from test harness fragility. That distinction matters for teams shipping login-critical software, because authentication failures are among the least forgiving bugs in the system.

This article is a lab-notebook style plan for benchmarking WebAuthn and passkey flows. It is written for SDETs, QA engineers, frontend teams, and engineering managers who want practical evidence, not a slogan. The goal is not to produce a universal score. The goal is to produce a repeatable way to answer questions like:

• Does browser automation fail at the same step every time, or only under certain device or browser combinations?
• Are our failures tied to resident credentials, user verification, attestation, platform authenticators, or session persistence?
• When a login test flakes, is the app at fault, the browser at fault, or our harness?
• Which flows need real-browser coverage versus protocol-level mocking?

What makes passkey flows unstable in automation

WebAuthn browser testing touches several layers at once. The visible UI is only the surface. Underneath are browser security rules, the authenticator model, device binding, account state, and whatever assumptions the application makes about all of those.

A passkey login flow can be unstable for a few broad reasons:

1. The browser cannot fully simulate the authenticator path

Some browser automation stacks can click buttons and fill forms, but cannot produce a real platform authenticator interaction with the right security properties. This is not a bug in the automation framework, it is a consequence of the protocol and the device trust model. The browser may expose hooks for virtual authenticators, but those hooks are not the same thing as the exact hardware, OS, and user-verification combination that production users have.

2. The app assumes a stable device state

Passkeys are often device-bound in practice, even when the user experience is smooth. If a test account has multiple passkeys, if the device has no resident credential, if the authenticator was previously used in a different profile, or if conditional UI behaves differently between browsers, the flow can branch in ways the test author did not anticipate.

3. Session and origin state leaks into the test

WebAuthn is origin-scoped. Cookies, local storage, profile data, and browser permissions can change the flow. A test that passes in a clean profile may fail in a reused one. That is not random, even though it looks random the first time you see it.

4. The product and the browser both produce similar symptoms

A server-side challenge mismatch, an expired ceremony, a bad RP ID, or a UI state bug can all manifest as “login failed.” Without structured logging, that message tells you almost nothing.

The most useful benchmark is not the one with the highest pass rate, it is the one that tells you which layer broke first.

Define the benchmark question before measuring anything

A common failure mode in stability work is starting with tools instead of questions. For passkeys, that usually leads to a pile of brittle tests and no useful evidence.

Start by stating the benchmark question in plain language. For example:

• How often does our login flow complete across supported browsers under clean and reused profiles?
• Which WebAuthn steps are sensitive to browser version, device type, or test environment?
• How much of our instability is due to authenticator availability versus app logic?
• Can we reliably reproduce the same failure mode in a real browser, or only in a mocked environment?

Then define what you will score.

Suggested stability dimensions

• Completion rate: did the login finish?
• Step consistency: did it fail at the same step, with the same browser and same account state?
• Retry sensitivity: does a rerun succeed without code changes?
• Environment sensitivity: do failures cluster by browser, OS, profile, or CI worker?
• Evidence quality: did the run produce logs, screenshots, network traces, and browser console output useful for triage?
• Harness overhead: how much special setup did the test need to reach a meaningful result?

The last one is often ignored, but it matters. A test that only works after a long chain of manual device preparation is not a stable benchmark, it is a fragile demo.

Scope the flow into measurable checkpoints

A WebAuthn login is not one event. Break it into checkpoints so you can tell where things go wrong.

A useful decomposition looks like this:

1. App reaches the login entry point
2. User chooses passkey or the app auto-prompts conditional mediation
3. Browser requests authentication from a platform or roaming authenticator
4. User verification or approval step completes
5. Assertion is returned to the app
6. Server verifies the assertion and starts a session
7. Redirect or post-login landing state confirms success

Now define the observable markers for each checkpoint. For example:

• A specific button or passkey prompt appears
• Browser permission or authenticator UI opens
• The network request to the WebAuthn verification endpoint is sent
• The app receives a success response
• A session cookie is set
• The dashboard page renders with authenticated content

These markers let you build a failure taxonomy instead of a single red or green label.

Decide what to log, before you run the first test

If you want to separate product bugs from browser capability gaps, your logs need to capture more than screenshots.

At minimum, capture:

• Browser name and version
• OS and device class
• Test account ID, or a stable anonymized account label
• Profile state, fresh or reused
• Whether the run used platform authenticator, virtual authenticator, or hardware security key
• The exact WebAuthn step reached
• Browser console logs
• Network requests around the login attempt
• The RP ID and origin, if your harness can expose them
• Error messages from the app and from the browser, separately
• Whether the session cookie or token was issued after the ceremony

If your framework supports it, also capture a short timeline with timestamps. A passkey flow often fails at the boundary between browser UI and server callback. A timeline makes that boundary visible.

Here is a practical log schema that can be attached to each run, whether you execute tests in Playwright, Selenium, or a codeless platform:

{ “run_id”: “2026-07-29T10:15:22Z-chrome-124-clean-profile”, “browser”: “chrome”, “browser_version”: “124.0.x”, “os”: “macos-14”, “authenticator”: “platform”, “profile”: “clean”, “flow_step”: “awaiting_user_verification”, “result”: “failed”, “error_class”: “browser_capability_gap”, “app_error”: null, “session_issued”: false }

The important thing is not the exact schema, it is consistency. If the fields change between runs, your benchmark is no longer a benchmark.

Separate three different failure classes

Most teams eventually discover that “passkey test failed” is not a single category.

Product bugs

These are defects in the application or backend implementation. Examples include:

• RP ID mismatch
• Incorrect challenge lifecycle
• Bad server-side credential lookup
• Session not persisted after a successful assertion
• Incorrect branching after authentication success

These should be reproducible in a normal browser with a valid authenticator path.

Browser capability gaps

These are gaps between what the app expects and what the browser, automation stack, or device can do in test mode.

Examples include:

• Conditional UI not exposed in that browser version
• Platform authenticator APIs unavailable in the runner
• Restrictions on secure context or same-origin behavior
• Inability to surface the OS-level prompt through automation

These are not product defects, but they are real test coverage limits.

Harness or fixture problems

These are caused by the test setup itself.

Examples include:

• Reused profiles with stale local state
• Wrong test account seed data
• Account already enrolled with different passkey metadata
• Brittle waits for transient UI
• CI worker permissions or device access problems

Harness problems are particularly annoying because they look like intermittent product failures until you inspect the run artifacts closely.

Benchmark matrix, not a single environment

Passkey stability depends heavily on the matrix you choose. Testing only one browser and one account state gives you a false sense of control.

A minimal useful matrix includes:

• Browser family, at least Chromium, Firefox, and WebKit where relevant
• Browser version, especially current stable and one prior version
• Profile state, clean versus reused
• Authenticator type, platform versus roaming versus virtual if supported
• Login method, direct passkey button versus conditional mediation versus account chooser
• Device class, desktop versus mobile if your product supports both

You do not need to test every combination every day. But you should know which combinations are part of your acceptance criteria and which are only exploratory.

If a browser or device combination is not in your benchmark matrix, treat its failures as unknowns, not surprises.

Use control flows to isolate the benchmark signal

A good benchmark has controls. In this context, that means flows that are almost identical to the passkey flow, but omit the WebAuthn step.

Examples:

• Username and password login on the same page
• Magic link or OTP login into the same post-auth session
• A passkey enrollment flow using the same account but different setup
• The same route with authentication mocked at the API layer, only for baseline comparison

Why do this? Because if login fails everywhere, the passkey step may not be the cause. If the non-passkey control passes consistently but the passkey path flakes, you have a much cleaner signal.

A useful benchmark report will compare passkey flow outcomes against at least one control flow on the same browser and device matrix.

Add timing, but use it carefully

Timing can be helpful, but only if you know what it means.

Measure:

• Time from click to prompt
• Time from prompt to assertion completion
• Time from assertion completion to session established
• Time to visual confirmation of logged-in state

Do not treat a slower passkey flow as a failure by itself. A secure authentication step will often be slower than a password form, and that is fine. What you care about is timing drift, timeout sensitivity, and whether the browser waits long enough for the flow to complete.

A common failure mode is a test timeout set for ordinary navigation, not for authenticator interaction. That timeout may be fine for a password form but too short for a user-verification prompt, especially on CI hardware.

Prefer real-browser artifacts over abstract pass/fail labels

When the login flow is complex, a bare assertion is not enough. You want artifacts that help you review the run after the fact.

Useful artifacts include:

• Screenshot at each major checkpoint
• Video capture, if available
• Browser console logs
• Network HAR or request log
• DOM snapshot around the failure
• Test runner stdout with timestamped milestones

This is where real-browser coverage pays off. If the failure is tied to a prompt not appearing, or to the browser suppressing a prompt in a certain mode, the artifact tells you more than a green or red glyph ever will.

A practical Playwright baseline for WebAuthn flows

If your team uses Playwright, keep the test short and instrumented. Do not build the entire benchmark around a giant monolith. The harness should make the checkpoint visible.

import { test, expect } from '@playwright/test';

test('passkey login flow', async ({ page }) => {
  await page.goto('https://app.example.test/login');
  await page.getByRole('button', { name: 'Sign in with passkey' }).click();

await page.waitForLoadState(‘networkidle’); await expect(page.getByText(‘Complete sign in on your device’)).toBeVisible();

// Replace this with your real authenticator setup or browser-supported path. // The benchmark should record where execution stops if the prompt never resolves. });

The exact automation mechanics depend on your browser and authenticator strategy. The important part is that the test records the step boundary clearly. If your framework supports virtual authenticators, use them as one axis in the matrix, not as the whole truth.

How to interpret flaky results

Flakiness is not all the same. For passkeys, classify it by reproducibility.

Stable failure

Fails every time in the same browser and same account state. This is usually the best clue you will get, because it points somewhere specific.

Intermittent failure

Fails sometimes on the same setup. This may mean a race in the app, an environment problem, or a timing mismatch between browser prompt and test timeout.

Environment-local failure

Fails only on some runners, worker pools, or OS versions. This often points to device access, profile handling, or browser permissions.

Cross-browser divergence

Works in one browser and fails in another. This is often a browser capability gap, but it can also be a product assumption that only one browser tolerated.

The benchmark should report these categories explicitly. A raw percentage cannot tell you whether you have a product regression or a test portability issue.

Distinguish protocol coverage from user journey coverage

This is the part that teams sometimes learn the hard way. You do not need the automation stack to control every byte of the WebAuthn protocol to get good test coverage.

There are two different questions:

1. Does our application handle passkey login correctly in real browsers?
2. Can our automation stack fully emulate every authenticator permutation?

Question 1 is a product quality question. Question 2 is a tooling question. Most teams need coverage for the first question far more than they need low-level protocol control for the second.

That is why a practical benchmark should value repeatable, real-browser results and rich artifacts. You want to know whether the user journey works, where it breaks, and what changed since yesterday. You do not necessarily need to impersonate the entire device stack in code.

Where to place assertions

Assertions in a passkey flow should happen at boundaries, not inside the authenticator internals.

Good assertion points include:

• Login entry page is reachable
• Passkey option is visible when expected
• Prompt or handoff step appears
• Authenticated landing page loads
• User identity or account avatar matches the test account
• Session-dependent API returns authorized data

Avoid overly specific assertions on transient UI text that may vary by browser locale, platform, or version. That is a common source of false negatives.

For teams that want more resilient checks around login state, a natural-language assertion model can reduce locator brittleness when the exact wording or element structure shifts. That is one reason platforms that support human-readable validation can be practical in identity-heavy suites, especially when the outcome is “the page is authenticated” rather than “this exact DOM node contains this exact string.”

A simple benchmark report template

At the end of a run, your benchmark report should answer three questions:

1. What worked?
2. What failed?
3. Where did the failure begin?

A concise table is enough.

The actual content will vary, but the structure should not. You want patterns to be visible across runs.

CI strategy, without turning the suite into a science project

Do not run every passkey benchmark on every commit. That is how teams create a lot of expensive noise.

A practical split is:

• Per commit: one or two happy-path authentication checks in a stable browser
• Nightly: the broader browser matrix with artifact capture
• Pre-release: the full matrix, including reused profiles and negative cases
• After auth changes: targeted reruns on the affected browser and account states

If your team already uses continuous integration, treat passkey stability as a scheduled evidence problem, not a punishment for every pull request.

Where Endtest can fit

If your team cares more about real-browser coverage and repeatable artifacts than about scripting every protocol detail, Endtest is one practical platform to evaluate for these flows. Its agentic AI model and editable, platform-native steps are a reasonable fit when the main need is to benchmark authentication journeys across browsers, capture artifacts, and keep the test readable for the rest of the team. For teams comparing tools, also look at how the platform handles AI Assertions when the expected result is a logged-in state rather than a brittle selector match.

That said, the platform choice should follow your benchmark question. If you need very low-level authenticator emulation, you may still want a framework-specific harness. If you need stable, reviewable browser coverage for identity workflows, a maintained platform can reduce the amount of custom code that tends to rot in authentication suites.

Final recommendation

Benchmarking browser test stability for passkeys is mostly an exercise in good observation. The login flow itself is not the hard part. The hard part is naming the layer that failed.

Build the benchmark around checkpoints, not just success or failure. Log browser and device state, not just screenshots. Separate control flows from WebAuthn flows. Classify failures by reproducibility and by layer. Prefer real-browser artifacts, because passkey bugs are often visibility problems before they are logic problems.

If you do that, you will get something more useful than a green dashboard. You will get a map of where your authentication stack is solid, where the browser can help, and where your test harness is still pretending to know more than it does.