How to Debug Browser Tests That Fail Only After Cache Invalidations or Asset Hash Changes

Browser tests that only fail after a deploy can be some of the hardest failures to triage. The app may look fine in a local run, the test may pass against a preview environment, and then production or staging starts throwing errors only after cache invalidation, asset hash changes, or a CDN refresh. These failures often get labeled as flaky browser tests, but the root cause is usually more specific: the test and the delivery layer have drifted out of sync.

That distinction matters. A real app regression and a delivery-layer issue can produce very similar symptoms, such as missing selectors, blank screens, stale bundles, hydration errors, or timeouts waiting for UI that never appears. If you treat every failure as a product bug, you waste time in the wrong place. If you treat every failure as a test bug, you may miss a broken release.

This guide walks through a practical debugging process for when browser tests fail after cache invalidation or asset hash changes. The goal is to separate application defects from problems caused by CDN cache, service worker behavior, build artifact mismatches, and test assumptions about asset delivery.

Why cache-related failures look like flaky browser tests

Modern frontends rarely ship as a single stable bundle. A release can include hashed JavaScript chunks, CSS files, dynamically loaded assets, preloaded resources, and service worker-controlled caching behavior. Those layers make deployments efficient, but they also introduce a few new failure modes:

• a test lands on an HTML page that references new hashed assets, while the CDN still serves old files
• a service worker returns stale HTML or stale chunks after deployment
• a browser cache keeps old JS, while the app shell expects new module names
• a test assumes a specific network timing that changes when assets are revalidated
• a release process invalidates some caches but not others, creating a mixed-version page

If a browser test fails only after a deployment-related cache change, start by asking whether the page is internally consistent, not just whether the UI looks broken.

That single question often splits the problem into two branches. In one branch, the app code itself is broken. In the other, the browser is receiving a partially updated application, which can break selectors, hydration, routing, or API calls even when the new code is correct.

First, identify the failure pattern

Before changing code or rewriting tests, classify the failure. The most useful categories are based on what changed and when the test fails.

1. Fails only after deploy, passes on the previous build

This often points to one of these:

• asset hash mismatch
• missing or stale CDN objects
• HTML cached separately from static assets
• service worker cache mismatch
• new code expecting new API data or new DOM structure

2. Fails on first load, passes after refresh

This often suggests stale client state, cache race conditions, or a bootstrap issue. The first load might hit one asset version, while the second load picks up the correct one.

3. Fails only in CI, not locally

This could still be cache-related, but now the environment adds variables such as parallel workers, cold browser profiles, ephemeral containers, or different cache-control behavior in test infrastructure.

4. Fails on one browser but not another

That can implicate browser cache persistence, service worker behavior, or differences in how Chrome, Firefox, and WebKit handle resource reuse and preloading.

A useful early rule is to separate the test failure into one of three buckets:

• rendering failure: page never renders correctly
• interaction failure: page renders, but clicks or waits fail
• resource failure: logs show 404, 500, blocked, or wrong-version assets

That lets you decide whether to inspect the DOM, the network, or the build pipeline first.

Reproduce the problem with a cold browser profile

A surprising number of cache-related failures disappear if you run the browser with a truly fresh profile. That does not mean the issue is gone, it means you have isolated the browser cache from the problem.

In Playwright, you can create a clean context every time:

import { test, expect } from '@playwright/test';

test('loads dashboard', async ({ browser }) => {
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto('https://staging.example.com/dashboard');
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
  await context.close();
});

If the failure only appears when you reuse a context, then browser state is involved. That could be localStorage, sessionStorage, service worker state, cookie state, or a cached asset.

For Selenium, use a separate profile or ephemeral user data directory when possible. The broader point is simple: do not debug a cache bug with a browser session that already contains cache history unless you are intentionally reproducing that state.

Inspect the network for mixed-version pages

When asset hash changes are involved, the most common failure is a mixed-version page. The HTML references new chunk names, but the browser receives old chunks or stale manifests from the CDN or browser cache. The result is often a blank app shell, a JS exception, or a test waiting forever for a component that never mounts.

Open DevTools or capture the network log in your test run and look for:

• 404s on hashed JS or CSS files
• requests returning HTML when JS was expected
• old and new asset versions appearing in the same page load
• long cache hit chains from CDN or browser cache
• service worker responses replacing network responses

In Playwright, you can quickly print failed requests:

page.on('requestfailed', request => {
  console.log('FAILED', request.url(), request.failure()?.errorText);
});

page.on(‘response’, response => { if (!response.ok()) { console.log(‘BAD’, response.status(), response.url()); } });

If your browser test fails after cache invalidation, a useful clue is the presence of a 200 response that actually contains the wrong content type or an old asset body. A CDN can return an apparently successful response that still breaks the app because the content is outdated or mismatched.

Check whether your HTML and assets are versioned consistently

Asset hash changes are meant to prevent stale caches, but they only work if the whole delivery chain is consistent. The most common consistency mistakes are:

• HTML cached too aggressively, while assets are content-hashed
• HTML updated immediately, while old assets remain on CDN edges longer than expected
• one manifest updated, another manifest lagging behind
• preload tags pointing to old asset names
• a deploy that deletes old assets too soon

This is where build and caching strategy matter more than the test itself. Browser tests expose the issue because they are usually one of the first systems to load the app end-to-end after deployment.

Ask these questions:

1. Does the entry HTML have a short cache lifetime, or is it being cached like a static asset?
2. Are asset filenames content-hashed, and are those hashes derived from actual content?
3. Are old assets kept available long enough for CDN edge nodes and browser caches to age out?
4. Does the app reference a manifest or runtime chunk that can become stale independently?
5. Are service workers caching the app shell, JS chunks, or API responses in a way that survives release boundaries?

If the answer to any of these is unclear, the test may be surfacing a deployment consistency problem, not a UI problem.

Look for service worker interference

Service workers are a frequent source of browser test failures after deployments. They can serve cached HTML, cached scripts, or stale offline responses even when the server is correct. This is especially common in progressive web apps and apps that aggressively cache static resources.

A service worker issue often presents like this:

• first test run fails after deploy
• hard refresh fixes it
• clearing site data fixes it
• a new browser profile passes
• the server logs show no problem

You can temporarily unregister the service worker in a debugging session:

typescript

await page.goto('https://staging.example.com');
await page.evaluate(async () => {
  const regs = await navigator.serviceWorker.getRegistrations();
  await Promise.all(regs.map(reg => reg.unregister()));
});

If unregistering or disabling the service worker fixes the browser test, the root cause is likely caching behavior in the service worker rather than the app logic itself. At that point, review the service worker update strategy, cache versioning, and whether the app handles new asset manifests safely.

Service workers are powerful, but they make “what version am I actually running?” a real debugging question.

Confirm that the app does not assume assets are instantly available everywhere

Many release pipelines invalidate CDN cache immediately after deploy, but global edge propagation is not instant. A test that runs during that propagation window may see a mismatch between newly served HTML and asset availability at a given edge location.

Common symptoms include:

• intermittent 404s on just-released chunks
• failures only right after deployment, then success minutes later
• a region-specific issue that disappears from one runner location but not another
• asset requests that resolve on refresh but not on initial load

This is where “flake” is a misleading label. The test is deterministic, but the delivery infrastructure is temporarily inconsistent.

You can confirm this by comparing the HTML and chunk references. If HTML points to app.7f3c1.js but the CDN still serves app.7f3c1.js with a 404 or stale content, the browser test is correctly failing because the page cannot boot.

Use a build fingerprint in the test logs

One of the best ways to debug these issues is to make the build version visible in the browser and in test output. If your app can expose a release identifier, build timestamp, commit SHA, or asset manifest version in a response header or DOM marker, the test can record it when failures happen.

Examples of useful fingerprints:

• X-Build-Id response header on HTML
• a version string in a <meta> tag
• a footer or hidden debug element with the deploy SHA
• a manifest file hash in application bootstrap logs

Then your test can emit that data on failure:

typescript

const buildId = await page.locator('meta[name="build-id"]').getAttribute('content');
console.log('build:', buildId);

This helps answer whether the page under test was actually the version you expected. If the browser loaded an older build than the one your CI job deployed, the test might be correct while the environment is stale.

Compare network behavior across fresh and cached runs

A strong debugging pattern is to run the same test in two modes:

• one with a fresh browser profile and disabled cache
• one with a reused profile and normal cache behavior

In Chrome-based tools, you can often disable cache in the context of a debugging session, or use DevTools Protocol to inspect cache behavior. The goal is not to keep cache disabled forever, because that can hide real issues. The goal is to compare behavior.

If the test passes with cache disabled but fails with normal cache, suspect:

• stale assets
• bad cache-control headers
• service worker cache contamination
• browser cache persistence between test steps
• a page that expects immediate consistency from the CDN

If the test fails in both modes, the problem is more likely the app or the environment configuration, not the browser cache itself.

Verify cache-control headers and CDN invalidation rules

Browser tests are downstream consumers of your delivery rules. When a failure appears after a cache invalidation, inspect headers, not just app code.

Useful things to check:

• cache-control on HTML, JS, CSS, and images
• etag and last-modified behavior
• surrogate-control or CDN-specific caching directives
• whether HTML is marked no-cache or short-lived
• whether static assets are immutable and hashed

A typical safe pattern is:

• HTML: short-lived or revalidated on each request
• hashed static assets: long-lived, immutable
• manifests and bootstrap files: carefully versioned and updated atomically

If your app violates that pattern, browser tests often catch the inconsistency first.

Make the failure visible in CI artifacts

When a browser test fails only after a cache or deploy event, the most useful evidence is often the page state at the moment of failure. Capture:

• screenshot
• HTML snapshot
• console logs
• network failures
• trace or video if available

In Playwright, traces are especially helpful because they preserve page actions and network timing. Even without a full trace, simple artifact collection can show whether the app loaded a 404 chunk, a blank page, or a stale bundle.

A minimal GitHub Actions example for preserving failure evidence:

name: browser-tests
on: [push]

jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 - run: npm ci - run: npm test - if: failure() uses: actions/upload-artifact@v4 with: name: test-artifacts path: | playwright-report test-results

Artifacts do not fix the bug, but they shorten the path from symptom to root cause.

Distinguish real regressions from delivery-layer failures

A good triage process should answer one question quickly: did the app break, or did delivery break?

Signs of a real application regression

• UI element exists, but behavior is wrong
• network requests succeed, but state transitions are broken
• assets load cleanly, but component logic fails
• same failure occurs with cache disabled and fresh profiles
• stack traces point to application code

Signs of a delivery-layer issue

• 404s on hashed assets
• blank screen after deploy, then success on refresh
• stale JS or CSS in only some environments
• failures disappear when cache is bypassed
• service worker unregistering fixes the test

This is not always clean-cut. A delivery problem can trigger a real app exception. For example, a missing chunk can prevent bootstrapping, which means your test sees a blank page. The important part is not the symptom, it is where the inconsistency originates.

Strengthen tests so they fail for the right reasons

Once you find the root cause, improve the test so the same class of issue is easier to diagnose next time.

Prefer state-based assertions over fragile timing

If a test waits for a route transition or a rendered element, assert on the actual app state rather than on arbitrary delays. Asset invalidation can change load timing enough to expose timing-based flakiness.

Assert critical boot signals early

If the app needs a JS bundle, a user session, or a manifest to boot, assert those prerequisites explicitly. A clearer early failure is better than a later timeout.

Add a version check in the test setup

If the app exposes a release ID, verify that the test is running against the expected build before continuing.

Avoid selectors that depend on unstable markup

When deployments change components, hashed assets, and hydration timing together, brittle selectors make debugging harder. Prefer accessible roles and stable test IDs where appropriate.

For example, in Playwright:

typescript

await expect(page.getByRole('button', { name: 'Save changes' })).toBeVisible();

This does not solve cache problems, but it reduces noise when you are trying to isolate them.

When the CDN is the real culprit

Sometimes the browser test is the messenger, and the CDN is the problem. If a release invalidates a CDN cache too aggressively or not aggressively enough, the browser may receive inconsistent content depending on which edge node handled the request.

Common release coordination mistakes include:

• deleting old assets before all HTML pages stop referencing them
• invalidating HTML and assets in separate steps with a gap between them
• deploying a new manifest before all assets are uploaded
• using different cache policies across environments

In these cases, browser tests are valuable because they make content integrity issues visible. The fix is usually in deployment orchestration, not in the test.

A practical release rule is to keep old immutable assets available for at least one full deployment cycle, or long enough for any cached HTML to age out. The exact policy depends on your app and CDN, but “remove old files immediately” is often too aggressive for hashed frontends.

A debugging checklist you can reuse

When a browser test fails after cache invalidation or asset hash changes, work through this order:

1. Re-run with a fresh browser profile.
2. Disable cache temporarily and compare behavior.
3. Inspect network errors for 404s, wrong MIME types, or stale assets.
4. Check service worker state and unregister if needed.
5. Verify HTML, manifest, and chunk versions belong to the same build.
6. Compare cache-control headers for HTML and immutable assets.
7. Confirm CDN invalidation finished before the test ran.
8. Capture build ID, screenshots, console logs, and traces.
9. Decide whether the bug is in the app, the test, or the delivery pipeline.

The key takeaway

When browser tests fail after cache invalidation, the most useful mindset is to treat the page as a distributed system artifact, not just a UI screen. The browser is combining HTML, scripts, styles, service worker state, and CDN responses into one runtime. If any part of that chain is out of sync, the failure may look like a flaky browser test even when the underlying issue is deterministic.

If you isolate the browser profile, inspect the network, verify asset versions, and check cache boundaries carefully, you can usually tell whether you are looking at a real app regression or a delivery-layer mismatch. That distinction saves time, keeps release triage focused, and prevents teams from patching over infrastructure problems with brittle test retries.

For background on the broader testing and delivery concepts involved, see software testing, test automation, and continuous integration.