Why Your Playwright Browser Gets Challenged and Your Real Chrome Doesn't

You point Playwright at a page and get a “Just a moment…” screen. It spins. Sometimes it shows a checkbox. Sometimes it reloads into the same challenge again. You give up waiting, open the exact same URL in the Chrome you use every day, on the same laptop and the same Wi-Fi, and the page loads in a second. No checkbox at all.

Same IP. Same machine. Same URL. The only thing that changed is the browser. That is the clue: most of the time the anti-bot system is not reacting to your network, it is reacting to the browser your automation tool launched.

What the Symptom Looks Like

This problem has a recognizable pattern:

• A challenge page that never clears on its own in the automated browser, but clears passively (or never appears) in your normal Chrome.
• A checkbox challenge that, once your script clicks it, turns into another challenge or a hard “access denied” page.
• A scraper that works headed on your laptop and fails headless on a server.
• A setup that worked for months and started failing after a framework upgrade, with no code changes on your side.

If your normal Chrome also gets blocked from the same network, stop reading here: your problem is IP reputation or request rate, not the browser. The rest of this post is about the case where the browser is the difference.

Your Automation Browser Is Not the Chrome You Use

When you run pip install playwright and playwright install, you do not get the Google Chrome that sits on your desktop. You get a browser built for automation:

• Playwright 1.57 and later runs on Chrome for Testing builds rather than open-source Chromium. Headed runs use Chrome for Testing, headless runs use chrome-headless-shell by default.
• Puppeteer v20 and later downloads Chrome for Testing, and since v21.6 it also downloads chrome-headless-shell for headless runs.

Google describes Chrome for Testing as a flavor “created purely for browser automation and testing purposes”. It is pinned to a version and does not auto-update, which is exactly what a test suite needs.

chrome-headless-shell is a different story. It is the old headless implementation, which Chrome’s documentation describes as “a lightweight wrapper around Chromium’s //content module”, and it historically identifies itself with HeadlessChrome in its user agent. It is fast and great for screenshots. It is not the full browser.

Your desktop Chrome is the branded build that a huge share of real users run, kept up to date automatically. To an anti-bot system that scores how normal a visitor looks, the gap between “the browser everyone uses” and “a build made for automation” is a signal worth reading.

Why Anti-Bot Systems Can Tell the Difference

Modern challenges are not puzzles. Cloudflare says Turnstile runs “a series of small non-interactive JavaScript challenges to gather signals about the visitor or browser environment”, including “probing for web APIs” and challenges “for detecting browser-quirks and human behavior”. Other vendors work along the same lines. A few places where automation builds stand out:

Automation flags

Per MDN, Chrome sets navigator.webdriver to true when it is started with --enable-automation, --headless, or --remote-debugging-port=0. Automation frameworks launch Chrome with a set of default switches, and --enable-automation is one of the classic ones. It is also what triggers the “Chrome is being controlled by automated test software” bar.

Build and feature differences

Playwright’s own docs note that “Chromium does not have all the codecs that Google Chrome or Microsoft Edge are bundling”. A headless shell is missing larger parts of the real browser. Any check that asks “what can this browser actually do?” and compares it with “what does it claim to be?” can find gaps like these.

Version drift

A pinned build is the same version every day, while the real Chrome population moves forward every few weeks. Months later, your scraper is reporting a browser version that fewer and fewer real visitors still run.

The control channel

Frameworks talk to Chrome over the DevTools Protocol, and some of the commands they use by default have side effects a page script can notice. Tools like Patchright exist specifically to avoid the best-known of these, such as the Runtime.enable leak.

The environment around the browser

A fresh profile with no history, a fixed default viewport, server fonts, and software rendering inside a container without a GPU all add up. None of these is damning alone. Together they paint a picture that does not look like a person on a laptop.

Why clicking the checkbox does not help

By the time a checkbox appears, the system has already decided your environment is uncertain. A scripted click is just one more data point, measured inside the same suspicious environment. That is why clicking often escalates to a harder challenge or a block instead of a pass. If the challenge does not clear passively, fix the environment first.

How to Diagnose It Locally

Do not guess. Run the same probe in each browser setup and compare. This script launches the bundled build and your installed Chrome, headed and headless, and prints what a page script can see:

from playwright.sync_api import sync_playwright

PROBE = """() => {
  const gl = document.createElement('canvas').getContext('webgl');
  const dbg = gl && gl.getExtension('WEBGL_debug_renderer_info');
  return {
    userAgent: navigator.userAgent,
    webdriver: navigator.webdriver,
    brands: navigator.userAgentData
      ? navigator.userAgentData.brands.map(b => `${b.brand} ${b.version}`)
      : null,
    h264: window.MediaSource
      ? MediaSource.isTypeSupported('video/mp4; codecs="avc1.42E01E"')
      : null,
    webglRenderer: dbg ? gl.getParameter(dbg.UNMASKED_RENDERER_WEBGL) : null,
    viewport: `${innerWidth}x${innerHeight}`,
  };
}"""

def probe(channel, headless):
    with sync_playwright() as p:
        browser = p.chromium.launch(channel=channel, headless=headless)
        page = browser.new_page()
        page.goto("https://example.com")
        result = page.evaluate(PROBE)
        browser.close()
        return result

for channel in (None, "chrome"):
    for headless in (True, False):
        label = f"{channel or 'bundled'} / {'headless' if headless else 'headed'}"
        print(label)
        for key, value in probe(channel, headless).items():
            print(f"  {key}: {value}")

Every line that differs between the bundled build and your real Chrome is something a site can read too. Then run the real test: load the protected page with each of the four setups, from the same network. Whichever combination passes tells you which layer to fix. If none of them pass, the browser is not your bottleneck.

How to Fix It

1. Use the branded Chrome

Both frameworks can drive the Chrome you already have:

browser = p.chromium.launch(channel="chrome", headless=False)

const browser = await puppeteer.launch({ channel: 'chrome', headless: false });

On a machine without Chrome, playwright install chrome installs Google Chrome at the system’s default location (it replaces an existing installation, so do this on servers, not your laptop). As a bonus, branded Chrome auto-updates on desktops, so you stop drifting behind the real population.

2. Run headed, or at least the real headless mode

Headed is the most authentic option. On a Linux server, run it inside a virtual display with xvfb-run python scraper.py. If you must run headless, use channel="chrome" with headless=True: since Playwright 1.49 the Chrome channel uses Chrome’s new headless mode, which the docs call “the real Chrome browser”, rather than the headless shell.

3. Drop the loudest automation flags

Remove --enable-automation and turn off the AutomationControlled blink feature. This takes away the cheapest checks. It does not make you invisible.

4. Keep a persistent profile

A brand-new, empty profile on every run looks like a first-time visitor every time. A persistent profile keeps cookies, cache, and site storage between runs, the same way your own browser does. Pair it with the cookie-reuse ideas from our post on session persistence.

5. Do not fake what you cannot back up

Setting a custom user agent on top of a different browser build creates a mismatch between what the browser says and what it can do. Let the real Chrome report itself.

6. Let the challenge clear on its own

Wait for the challenge to go away instead of clicking it. Putting it together:

from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeout

URL = "https://example.com/protected-page"

with sync_playwright() as p:
    context = p.chromium.launch_persistent_context(
        user_data_dir="./chrome-profile",
        channel="chrome",
        headless=False,
        no_viewport=True,
        ignore_default_args=["--enable-automation"],
        args=["--disable-blink-features=AutomationControlled"],
    )
    page = context.pages[0] if context.pages else context.new_page()
    page.goto(URL, wait_until="domcontentloaded")

    try:
        # Adapt this check to the challenge you actually see.
        page.wait_for_function(
            "() => !document.title.toLowerCase().includes('just a moment')",
            timeout=30_000,
        )
        html = page.content()
    except PlaywrightTimeout:
        html = None
        print("Challenge did not clear: check the environment, IP and request rate")

    context.close()

7. Consider a patched driver

If you want these fixes packaged, Patchright is a drop-in replacement for Playwright that patches known leaks such as Runtime.enable and the automation flags. Its recommended setup is the same as above: a persistent context, channel="chrome", headed, no fixed viewport, and no custom user agent. nodriver, the successor to undetected-chromedriver, takes a similar approach by driving your installed Chrome without WebDriver.

The Browser Is Only One Layer

Fixing the browser removes one reason to challenge you. It does not remove the others. A perfect real Chrome that sends thousands of requests an hour from one IP will still burn that IP’s reputation, and a raw HTTP client still has the TLS fingerprint problem no matter which browser you used to get the cookies. For the full checklist on one of the most common protections, see how to scrape a Cloudflare-protected site without a 403. And always check content, not status codes: a challenge page can arrive as a 200 response.

Respect the site’s terms and keep your request rate reasonable. Being polite is also the cheapest way to stay unblocked.

FAQ

Is Chrome for Testing the same as Google Chrome?

It is built from the same release process and is “as close to regular Chrome as possible”, in Google’s words, but it is a separate flavor meant only for automation and testing. It does not auto-update, and it is not the build that ordinary visitors run.

Does setting navigator.webdriver to false make Playwright undetectable?

No. It removes one cheap check. Build differences, the DevTools control channel, the environment, your IP, and your behavior are all still visible. Treat it as hygiene, not a solution.

Should my scraper click the Turnstile checkbox automatically?

Usually not as a first move. The checkbox appears because the environment already looks uncertain, and a scripted click inside that same environment rarely changes the verdict. Fix the browser setup until the challenge clears passively.

Can I run branded Chrome in Docker?

Yes. Install Google Chrome in the image (playwright install chrome can do it for you) and run it headed under Xvfb, or headless with channel="chrome" to get the new headless mode instead of the headless shell.

Skip the Browser Maintenance

Keeping browser builds current, profiles warm, and flags clean is real ongoing work, and it breaks every time a framework or an anti-bot system updates. If you would rather spend that time on the data, ScrapeUnblocker renders the page in a real browser for you and returns the HTML from a single API request. The documentation shows how to make your first call in a few minutes.