AI-Generated Playwright Code Benchmark

AI-generated Playwright code can be a useful starting point, but it is not the same thing as a production-ready test suite. This benchmark-style article evaluates how AI-generated Playwright tests behave across realistic web testing scenarios, where the hard parts are not syntax, but locators, waits, assertions, data setup, authentication, maintainability, and the amount of manual repair needed before tests are safe to run in CI.

Lab note

This is a qualitative benchmark design and scoring article, not a claim that one private run against one model proves universal performance. AI coding tools change quickly, Playwright changes, and application behavior matters. The goal is to give SDETs, developers, QA leaders, and CTOs a practical framework for evaluating AI-generated Playwright code in their own environment.

The target keyword for this study is AI-generated Playwright code benchmark, but the more important question is operational: if an AI assistant writes a Playwright test, how much real testing work remains?

The useful benchmark is not whether AI can generate code. It is whether the generated test verifies the right behavior and can be trusted in CI.

What we benchmarked

The benchmark focuses on AI-generated Playwright code for end-to-end web testing. In a commercial setting, teams usually care less about whether the generated code looks impressive in a demo and more about whether it can survive:

• Real authentication flows
• Dynamic UI states
• Async network behavior
• Role-based permissions
• Data dependencies
• CI execution
• Product changes over time
• Review by engineers who did not generate the test

Playwright itself is a strong automation framework. The official Playwright documentation gives teams a capable foundation for browser automation, including auto-waiting, fixtures, tracing, screenshots, and multi-browser support. The benchmark is not an argument against Playwright. It is an evaluation of what happens when the initial Playwright source code is generated by AI from natural language, tickets, HTML snippets, or partial product knowledge.

The distinction matters. Playwright gives you primitives. AI gives you a draft. A production test suite needs correctness, readability, stability, ownership, and maintenance discipline.

Benchmark questions

We evaluated AI-generated Playwright code against six questions that matter in real engineering teams:

1. Can the generated test compile and run without syntax fixes?
2. Does it test the intended behavior, not just interact with the page?
3. Are locators stable and reviewable?
4. Does the code handle async UI behavior correctly?
5. Can it be maintained by the team six months later?
6. How much manual work is required before it can be trusted in CI?

This produces a more useful Playwright AI benchmark than asking whether a model can write a happy-path login test. Most AI tools can generate something plausible for a login form. The difficult cases are where test automation usually becomes expensive.

Benchmark setup

The benchmark uses realistic web testing scenarios rather than toy pages. Each scenario was expressed as a prompt that a QA engineer or developer might give to an AI coding assistant.

We assessed the generated tests using a five-point scoring model:

We did not assign fabricated numeric results to named AI vendors. Instead, this article defines the benchmark cases, the scoring method, common failure modes, and a representative scorecard pattern that teams can apply to Claude, ChatGPT, Copilot, Cursor, Reflect, or internal AI coding tools.

Scenario 1: Login with role-based redirect

Prompt

text Write a Playwright test for the staging app. Log in as an admin user, verify the dashboard loads, and confirm that the admin navigation includes Users, Billing, and Audit Logs. Then log out and verify the login page is shown.

What good AI-generated Playwright code should do

A strong generated test should:

• Use environment variables for credentials
• Avoid hard-coded passwords
• Use stable locators such as roles, labels, and test IDs
• Assert the post-login URL or a reliable page heading
• Verify role-specific navigation without relying on brittle CSS classes
• Clean up session state between tests

A reasonable Playwright implementation might look like this:

import { test, expect } from '@playwright/test';

test('admin can log in and see admin navigation', async ({ page }) => {
  await page.goto(process.env.APP_URL ?? 'https://staging.example.com/login');
  await page.getByLabel('Email').fill(process.env.ADMIN_EMAIL ?? '');
  await page.getByLabel('Password').fill(process.env.ADMIN_PASSWORD ?? '');
  await page.getByRole('button', { name: 'Log in' }).click();

await expect(page.getByRole(‘heading’, { name: ‘Dashboard’ })).toBeVisible(); await expect(page.getByRole(‘navigation’)).toContainText(‘Users’); await expect(page.getByRole(‘navigation’)).toContainText(‘Billing’); await expect(page.getByRole(‘navigation’)).toContainText(‘Audit Logs’);

await page.getByRole(‘button’, { name: ‘Account’ }).click(); await page.getByRole(‘menuitem’, { name: ‘Log out’ }).click();

await expect(page.getByRole(‘heading’, { name: ‘Log in’ })).toBeVisible(); });

Common AI failures

In this scenario, AI-generated Playwright code tends to be fast and syntactically correct. The weak points are usually assumptions, not TypeScript syntax.

Common problems include:

• Inventing selectors such as #username, .admin-menu, or button.logout
• Assuming the logout control is visible without opening an account menu
• Forgetting to verify role-specific behavior beyond the dashboard page
• Hard-coding credentials into the test body
• Using page.waitForTimeout(2000) instead of state-based waits
• Not isolating storage state between tests

The generated code is often a useful first draft. However, it usually requires a tester or developer who understands the application to correct selectors, choose better assertions, and make the setup safe for CI.

Benchmark score pattern

Login tests are where AI-generated test code looks best. They are also where teams may overestimate the maturity of the approach.

Scenario 2: Checkout flow with dynamic totals

Prompt

text Create a Playwright test for a checkout flow. Add the Basic Plan to the cart, apply coupon QA20, verify the discount, enter payment details using the test card, submit the order, and verify the confirmation page shows the correct total.

Why this is harder

Checkout tests combine UI automation with business rules. A test that clicks through the flow but does not validate the total is not a checkout test. It is a navigation script.

The generated code must understand or infer:

• Product price
• Coupon behavior
• Tax and shipping rules
• Payment iframe handling
• Test card details
• Confirmation state
• Whether the app uses cents, decimals, localization, or currency symbols

AI assistants frequently produce code that is plausible but incomplete. For example, they may verify that the order confirmation page appears but skip the actual total calculation.

Useful assertion pattern

A maintainable Playwright test often separates parsing and business validation from UI steps:

import { test, expect } from '@playwright/test';

function moneyToCents(text: string): number { const normalized = text.replace(/[^0-9.]/g, ‘’); return Math.round(Number(normalized) * 100); }

test('checkout applies QA20 discount', async ({ page }) => {
  await page.goto('/pricing');
  await page.getByRole('button', { name: 'Choose Basic' }).click();
  await page.getByLabel('Coupon code').fill('QA20');
  await page.getByRole('button', { name: 'Apply coupon' }).click();

await expect(page.getByText(‘20% discount applied’)).toBeVisible();

const subtotal = moneyToCents(await page.getByTestId(‘subtotal’).innerText()); const discount = moneyToCents(await page.getByTestId(‘discount’).innerText()); const total = moneyToCents(await page.getByTestId(‘total’).innerText());

expect(total).toBe(subtotal - discount); });

This snippet is intentionally partial. Real payment flows require careful handling of provider-specific iframes, test cards, and compliance boundaries. The point is that the valuable part of the test is not the generated clicking. It is the assertion logic.

If the test does not encode the business oracle, it may pass while the product is wrong.

Common AI failures

For checkout scenarios, AI-generated Playwright code often struggles with:

• Payment fields inside iframes
• Totals that change after tax calculation
• Race conditions after applying a coupon
• Coupons with async validation
• Ambiguous labels such as multiple “Continue” buttons
• Missing negative assertions, such as invalid coupon rejection
• Hard-coded totals that become wrong when pricing changes

AI can draft the flow quickly, but the tester must still encode the business oracle. Without that oracle, the test may pass while the checkout calculation is broken.

Benchmark score pattern

For revenue-critical flows, AI-generated Playwright code should be treated as scaffolding, not as a final artifact.

Scenario 3: Search, filters, and empty states

Prompt

Write Playwright tests for the product search page. Search for "wireless keyboard", filter by In Stock, sort by Price Low to High, verify that results are shown in ascending price order, then search for a nonsense term and verify the empty state.

Why this is a good benchmark case

Search and filtering reveal whether generated code can reason about collections. Many simple tests assert that “some results are visible.” A good test verifies that all visible results satisfy the filter and that sorting is correct.

A useful helper might look like this:

import { Page } from '@playwright/test';

async function visiblePrices(page: Page): Promise<number[]> { const priceTexts = await page.getByTestId(‘product-price’).allInnerTexts(); return priceTexts.map(text => Number(text.replace(/[^0-9.]/g, ‘’))); }

function isAscending(values: number[]): boolean { return values.every((value, index) => index === 0 || values[index - 1] <= value); }

Then the test can assert the actual behavior:

import { test, expect } from '@playwright/test';

test('search results can be filtered and sorted', async ({ page }) => {
  await page.goto('/products');
  await page.getByRole('searchbox', { name: 'Search products' }).fill('wireless keyboard');
  await page.keyboard.press('Enter');

await page.getByRole(‘checkbox’, { name: ‘In Stock’ }).check(); await page.getByLabel(‘Sort by’).selectOption(‘price-asc’);

await expect(page.getByTestId(‘product-card’).first()).toBeVisible();

const prices = await visiblePrices(page); expect(prices.length).toBeGreaterThan(0); expect(isAscending(prices)).toBe(true);

await page.getByRole(‘searchbox’, { name: ‘Search products’ }).fill(‘zzzz-no-results-qa’); await page.keyboard.press(‘Enter’);

await expect(page.getByText(‘No products found’)).toBeVisible(); });

Common AI failures

This scenario exposes shallow assertions. AI-generated code often:

• Checks that the URL contains a query parameter but not that results changed
• Verifies only the first result
• Uses fixed sleeps after search input
• Does not handle debounce or network-driven UI updates
• Assumes all result cards are visible at once, which breaks with pagination or virtualization
• Parses prices incorrectly when currency formatting varies

For teams building an AI Playwright code workflow, search and filters should be part of the acceptance benchmark. They are common enough to matter and complex enough to reveal whether the generated tests validate real behavior.

Scenario 4: Multi-user collaboration

Prompt

text Write a Playwright test where User A creates a shared project and invites User B. User B accepts the invite, adds a comment, and User A sees the comment without refreshing.

Why this scenario matters

Multi-user tests are a strong signal for maturity. They require multiple browser contexts, independent sessions, and often real-time updates via polling, WebSockets, or server-sent events.

A correct Playwright design may use separate contexts:

import { test, expect } from '@playwright/test';

test('invited user can comment on shared project', async ({ browser }) => {
  const userAContext = await browser.newContext({ storageState: 'auth/user-a.json' });
  const userBContext = await browser.newContext({ storageState: 'auth/user-b.json' });
  const userAPage = await userAContext.newPage();
  const userBPage = await userBContext.newPage();

await userAPage.goto(‘/projects/new’); await userAPage.getByLabel(‘Project name’).fill(‘Collaboration QA’); await userAPage.getByRole(‘button’, { name: ‘Create project’ }).click(); await userAPage.getByRole(‘button’, { name: ‘Invite’ }).click(); await userAPage.getByLabel(‘Email’).fill(process.env.USER_B_EMAIL ?? ‘’); await userAPage.getByRole(‘button’, { name: ‘Send invite’ }).click();

await userBPage.goto(‘/invites’); await userBPage.getByRole(‘button’, { name: ‘Accept invite’ }).click(); await userBPage.getByLabel(‘Comment’).fill(‘Looks good from User B’); await userBPage.getByRole(‘button’, { name: ‘Post comment’ }).click();

await expect(userAPage.getByText(‘Looks good from User B’)).toBeVisible();

await userAContext.close(); await userBContext.close(); });

Common AI failures

AI tools often produce single-session code for a multi-user scenario. Typical issues include:

• Logging out and logging back in within one page instead of using independent contexts
• Losing the project URL between users
• Assuming email delivery instead of using an invite inbox, API, or database fixture
• Failing to wait for real-time updates correctly
• Not closing contexts
• Creating data that is not cleaned up

This is one of the most important cases for a Claude Playwright benchmark or any comparison of AI coding assistants. The prompt is easy to understand, but the automation design requires framework knowledge and product-specific choices.

Scenario 5: Visual state, accessibility, and responsive behavior

Prompt

Generate Playwright tests for the account settings page. Verify required field validation, keyboard navigation through the form, mobile layout behavior, and that the save button is disabled until changes are made.

Why this is difficult

This scenario blends functional testing, accessibility expectations, and responsive UI behavior. AI-generated Playwright code can write straightforward form tests, but it often under-specifies accessibility and viewport behavior. For accessibility expectations, teams should align on the relevant parts of WCAG.

Good tests might include:

• Role-based locators
• Keyboard-only navigation
• Focus assertions
• Viewport-specific checks
• Validation messages linked to fields
• Assertions that disabled buttons are actually disabled, not just visually grey

Example:

import { test, expect } from '@playwright/test';

test('save button enables only after a valid change', async ({ page }) => {
  await page.goto('/account/settings');
  const saveButton = page.getByRole('button', { name: 'Save changes' });
  await expect(saveButton).toBeDisabled();

await page.getByLabel(‘Display name’).fill(‘QA Example User’); await expect(saveButton).toBeEnabled();

await page.getByLabel(‘Display name’).fill(‘’); await page.getByLabel(‘Display name’).blur(); await expect(page.getByText(‘Display name is required’)).toBeVisible(); await expect(saveButton).toBeDisabled(); });

For responsive behavior:

import { test, expect } from '@playwright/test';

test('settings navigation collapses on mobile', async ({ page }) => {
  await page.setViewportSize({ width: 390, height: 844 });
  await page.goto('/account/settings');
  await expect(page.getByRole('button', { name: 'Open settings menu' })).toBeVisible();
  await expect(page.getByRole('navigation', { name: 'Settings sections' })).toBeHidden();
});

Common AI failures

This category often exposes weak test intent:

• Confusing visual disabled state with the disabled attribute or ARIA state
• Skipping keyboard interactions
• Using CSS selectors tied to layout classes
• Not setting viewport size before navigation
• Checking only desktop behavior
• Treating accessibility as a separate concern rather than part of the user flow

AI-generated Playwright tests are often serviceable for basic form validation. They are weaker when the scenario requires knowing what “accessible” or “responsive” means in a specific product.

Scorecard: what the benchmark usually reveals

Across these scenarios, a consistent pattern emerges.

The practical conclusion: AI-generated Playwright code is a strong acceleration tool for engineers who already know how to write good Playwright. It is much riskier as a replacement for test design expertise.

How to run your own AI Playwright benchmark

If your team is evaluating AI Playwright code, avoid a single demo prompt. Build a small benchmark pack that reflects your real application.

1. Select five to eight representative workflows

Include at least one from each category:

• Authentication and permissions
• CRUD flow with cleanup
• Search, filters, or sorting
• Payment, billing, or plan changes if relevant
• Multi-user or collaboration behavior
• Responsive or accessibility-sensitive UI
• Negative path validation
• A flaky historical area of your app

2. Use the same prompt for each AI tool

For a fair Claude Playwright benchmark or multi-tool comparison, prompts must be consistent. Do not refine one tool more than another unless prompt iteration is part of the test.

A good prompt template:

text Generate a Playwright test in TypeScript for this scenario. Use @playwright/test. Prefer getByRole, getByLabel, and getByTestId locators. Do not use waitForTimeout. Use environment variables for credentials. Include meaningful assertions for the business behavior. Make the test suitable for CI. Scenario: [paste scenario]

This prompt does not guarantee good output, but it prevents easy failure modes.

3. Track manual fix time

Manual fix time is one of the most honest metrics. For each generated test, measure:

• Time to make it compile
• Time to make it run locally
• Time to make it assert the intended behavior
• Time to make it pass reliably three times in a row
• Time to make it acceptable in code review

Do not count only the time from prompt to first file. That metric rewards code volume, not test quality.

4. Categorize fixes

Create a simple defect taxonomy:

fix_categories:
  syntax_or_imports: "Code does not compile or imports are wrong"
  invented_selectors: "Selectors do not exist in the app"
  weak_assertions: "Test clicks through but does not verify behavior"
  async_race: "Fails because UI or network state is not ready"
  data_dependency: "Requires unavailable or dirty test data"
  auth_state: "Login/session handling is unsafe or incorrect"
  cleanup_missing: "Leaves records that affect later runs"
  maintainability: "Needs refactoring for readability or reuse"

After ten or twenty generated tests, patterns become obvious. You may find that one assistant writes cleaner TypeScript while another chooses better locators. You may also find that prompt engineering helps less than improving your app’s testability with accessible names and stable test IDs.

5. Score CI readiness separately

A generated test can pass locally and still be a bad CI test. CI readiness should consider:

• Deterministic data setup
• No dependence on test order
• No shared mutable user state unless controlled
• Clear artifacts on failure, such as traces and screenshots
• Reasonable execution time
• No fixed sleeps
• Isolation across parallel workers

A flaky AI-generated test is not a productivity gain if it creates reruns, triage noise, and distrust in the suite.

Prompting improvements that actually help

Prompting cannot replace test engineering, but it can reduce predictable failures.

Ask for locator discipline

Bad prompt:

text Write a Playwright test for login.

Better prompt:

text Write a Playwright test for login. Prefer getByRole and getByLabel. Use getByTestId only for elements that do not have stable accessible names. Do not use CSS classes as selectors.

Ask for business assertions

Bad prompt:

text Test checkout.

Better prompt:

text Test checkout and assert that the order total equals subtotal minus discount plus tax. Do not stop at checking that the confirmation page appears.

Ask for no fixed sleeps

Bad generated tests often include:

typescript

await page.waitForTimeout(3000);

A better pattern is:

typescript

await expect(page.getByText('Discount applied')).toBeVisible();

or:

typescript

await page.waitForResponse(response =>
  response.url().includes('/api/coupons') && response.status() === 200
);

Use response waits carefully. UI assertions are usually more user-centered, but network waits can be appropriate when the UI has multiple asynchronous phases.

Ask for cleanup

For CRUD flows, include cleanup expectations:

text Create a project with a unique name using a timestamp. Delete the project at the end of the test. If deletion is not possible through the UI, add a TODO comment for API cleanup rather than ignoring cleanup.

This does not guarantee a perfect cleanup implementation, but it makes the gap visible.

Maintainability is the hidden benchmark

AI-generated code often looks acceptable on day one. The real question is whether the suite is understandable after the product changes.

Maintainable Playwright tests usually have:

• Clear test names that describe behavior
• Minimal helper abstraction, not a framework inside a framework
• Page objects only where they reduce duplication
• Stable locators based on user-visible behavior or intentional test IDs
• Assertions close to the behavior being tested
• Test data that is created and cleaned intentionally
• Comments only where they explain non-obvious product behavior

AI-generated code can violate maintainability in two opposite ways. Sometimes it is too flat, with one long procedural test full of repeated selectors. Other times it over-engineers abstractions, creating page objects, utility classes, and config files before the team has agreed on patterns.

A generated page object like this may be unnecessary:

class LoginPage {
  constructor(private page) {}

async login(email: string, password: string) { await this.page.getByLabel(‘Email’).fill(email); await this.page.getByLabel(‘Password’).fill(password); await this.page.getByRole(‘button’, { name: ‘Log in’ }).click(); } }

It is not wrong. But if only one test uses it, it adds indirection without much value. AI tools are prone to generating patterns that look professional while hiding simple behavior behind premature abstraction.

Where Endtest fits in this benchmark

The benchmark highlights a gap between code generation and reliable test automation. If your team has strong TypeScript skills, owns a Playwright framework, and is willing to review generated code carefully, AI-generated Playwright can speed up authoring. But many QA organizations do not want more fragile code to debug. They want reliable tests that the whole team can inspect, edit, and run.

That is where Endtest can fit. Endtest is an agentic AI test automation platform with low-code/no-code workflows. Its AI Test Creation Agent generates editable, platform-native steps inside Endtest rather than producing Playwright, Selenium, JavaScript, Python, or TypeScript source files. The generated output is not fake Playwright code. It is a set of editable Endtest actions, assertions, and locators that testers can review and adjust in the platform.

This difference matters for commercial teams. With AI-generated Playwright, the output is code, so the team still owns:

• Playwright configuration
• Package updates
• Browser dependencies
• CI integration
• Reporting
• Parallelization strategy
• Flake triage
• Locator maintenance
• Code review standards

Endtest approaches the problem as a managed testing platform. Teams can use the AI Test Creation Agent documentation to understand how AI-assisted creation works in the product. For maintainability, Endtest also provides capabilities such as Self Healing Tests, with additional detail in the Self Healing Tests documentation. For validation beyond basic functional checks, Endtest offers Visual AI and Accessibility Testing, with accessibility details available in the Accessibility Testing documentation.

For teams that are choosing between “AI writes Playwright for us” and “AI helps us create maintainable tests,” the distinction is important. A generated code file may be fast to produce but expensive to own. Editable platform-native steps can be easier for QA engineers, product specialists, and non-developer stakeholders to understand.

The credible tradeoff is this: Playwright gives maximum code-level flexibility. Endtest gives a more accessible and managed workflow. If your product requires deep custom test logic, direct repository integration, and engineers dedicated to framework ownership, Playwright remains attractive. If your bottleneck is maintainable coverage across a team that does not want to debug generated code, an agentic AI test automation platform with low-code/no-code workflows such as Endtest may be the more practical path.

Commercial evaluation checklist

For CTOs and QA leaders, the buying question is not “Can AI write Playwright?” It can. The question is “What operating model do we want after the code is generated?”

Use this checklist when evaluating AI Playwright code or alternatives.

Engineering ownership

• Who reviews the generated tests?
• Who fixes them when Playwright changes?
• Who handles dependency updates?
• Who debugs flaky CI failures?
• Who defines locator standards?

QA ownership

• Can manual testers edit tests safely?
• Can non-developers understand failures?
• Are tests tied to product behavior or implementation details?
• Can the team add coverage without waiting for developers?

Maintenance cost

• How often do locators break?
• How long does triage take?
• Are failures actionable?
• Are generated tests consistent with existing patterns?
• Does the suite become easier or harder to maintain as it grows?

Tooling cost

For Playwright, consider the cost of:

• Test framework development
• CI minutes and parallel execution
• Browser grid or cloud execution
• Reporting and analytics
• Secrets management
• Test data management
• Onboarding and code review

For a managed platform, consider:

• Subscription cost
• Platform fit for your application
• Export or import needs
• Collaboration workflow
• Browser and device coverage
• Integrations with your CI and issue tracker

The cheapest option on paper may not be cheapest after maintenance.

Recommended benchmark rubric

Here is a practical rubric you can copy into a spreadsheet.

Score each from 1 to 5, multiply by weight, and compare tools or workflows. Include manual fix time as a separate column. A tool that scores slightly lower but requires far less maintenance may be the better business choice.

Final findings

The main finding from this AI-generated Playwright code benchmark is that AI is strongest at producing the visible surface of a test: imports, test() blocks, common locators, basic assertions, and straightforward flows. It is weaker at the invisible parts that make test automation trustworthy: data control, business oracles, multi-user state, cleanup, race conditions, and long-term maintainability.

For SDETs and developers, AI-generated Playwright code is worth using as a drafting accelerator. Treat it like a junior contribution that needs review, not like an autonomous QA engineer. Demand meaningful assertions, stable locators, and CI-safe design.

For QA leaders and CTOs, the key decision is whether you want to own a growing body of generated test code. If you already have Playwright expertise and engineering capacity, AI can improve authoring speed. If your goal is broader test creation with less code maintenance, an agentic AI test automation platform with low-code/no-code workflows such as Endtest is worth evaluating because it turns AI-created scenarios into editable platform-native tests rather than fragile source code that still needs debugging.

The best benchmark is the one you run against your own app. Use realistic scenarios, measure manual fixes, score maintainability, and include the cost of ownership. AI can make Playwright faster, but speed is only one part of a reliable testing strategy.