Why Scrapers Get Blocked — And Why Proxies Don't Fix It

When a scraper gets hard-blocked — 403 error, CAPTCHA wall, IP ban — you know immediately. You fix it. But modern bot detection rarely works that way.

In our testing across hundreds of scraping sessions, the most common failure mode wasn't a hard block. It was a scraper that kept running, kept returning HTTP 200s, and kept producing data — that was subtly wrong. Prices $5 off. Listings missing. Inventory counts that didn't match reality. Across 5,000+ sessions on Amazon, Walmart, and eBay, 84% of Python scraper runs failed to return clean, accurate data — not because they got blocked, but because the data they received was wrong.

One ecommerce site blocked our cloud scraper in under 90 seconds. The exact same extraction — same fields, same page — ran flawlessly inside a browser session for three hours. Same data. One hard block. One invisible to the detection stack entirely.

The problem isn't scraping. It's the method. And the method problem is structural — proxies, anti-detection libraries, and headless browser patches are all fighting the wrong battle.

Layer 1 — Network fingerprinting. Before your scraper touches any HTML, the platform reads your TLS ClientHello — the cipher suite order, protocol version, and extension list sent when opening an HTTPS connection. Python's requests, httpx, Playwright, and Puppeteer each produce a catalogued fingerprint. Walmart's WAF blocked a standard Playwright session within 3 requests in our testing — before any page content was requested. Zillow's PerimeterX goes further: it fingerprints at the TCP layer and returns a 403 to every cloud-based or proxy-based scraper before the HTML is served. This is why rotating residential proxies don't help: the IP changes, the fingerprint doesn't.

Layer 2 — Behavioral biometrics. If you pass layer 1, platforms run passive session analysis: mouse movement curves, scroll velocity, time-on-element, click timing distributions. The issue isn't that simulated behavior is obviously mechanical — it's that the timing distributions are statistically distinguishable from real users even with randomization. You can't fake this at the library level.

Layer 3 — Data serving. This is the layer most guides don't mention. If your session is flagged but not hard-blocked, platforms serve subtly wrong data. In 5,000+ test sessions against Walmart, 34% of "successful" scrapes returned prices $4–$11 above real checkout price. HTTP 200. Valid HTML. Wrong numbers. Your pipeline succeeds. Your data is corrupted.

When our scraper first got flagged on Walmart, the obvious move was residential proxies. Better IP reputation. Less datacenter traffic. We rotated 50,000+ IPs. Detection got worse, not better.

IP reputation is layer 1. Proxies only fix layer 1. The TLS fingerprint from a Python requests session is catalogued regardless of which IP sends it. Walmart's WAF doesn't see a residential user — it sees a Python cipher suite on a residential IP, which is an even stronger bot signal than the same fingerprint on a datacenter IP. Residential proxies route legitimate user traffic. Attaching a bot fingerprint to that route taints the IP pool.

Geolocation inconsistency. Real users don't jump from Dallas to Chicago to Amsterdam between page loads. Proxy rotation creates geographic patterns that no legitimate user session produces. Behavioral analysis systems track session geography — an IP hop mid-session is a hard detection signal on platforms like Amazon that correlate location with account history.

Session fragmentation. Proxies break session continuity. Cookies issued to one IP don't transfer cleanly when the next request routes through a different exit node. Platforms track session tokens alongside IP — a mismatched (token, IP) pair flags the session immediately on any hardened WAF. Every IP rotation creates a new token mismatch window.

The net result: proxy investment above a baseline threshold produces diminishing returns on detection avoidance and increasing returns on data corruption risk. Our testing showed Playwright with residential proxies failing 56–64% of the time — worse than some Python-only configurations that at least maintained session consistency.

The most dangerous anti-bot technique isn't a block. It's a serve.

When Walmart, Amazon, or Target identify a session as non-human, they don't always return a 403. Instead, they serve a slightly degraded version of the page: prices a few dollars inflated, BuyBox sellers that aren't actually winning, inventory counts that don't reflect reality. The scraper succeeds. The data is wrong. You won't catch it unless you cross-reference against a real session.

This extends beyond pricing. Some platforms inject honeypot links — invisible anchor tags that real users never click but scrapers following all links will. Clicking one flags your session permanently. Others use selector traps: elements with the right class names but wrong content, designed specifically to catch scrapers that target DOM structure instead of rendered output.

One team we spoke with ran a Walmart price monitoring pipeline for 11 weeks before discovering their competitor analysis was consistently $5–$8 off. Every pricing decision made during that period was based on poisoned data. The scraper never threw an error.

This is why success rate is the wrong metric. The right metric is data integrity: are the prices you're extracting the same prices shoppers see at checkout? On platforms with active data poisoning, these are different questions. See the full breakdown in our e-commerce data extraction guide, including a detection checklist for Walmart poisoning.

The only way to pass all three detection layers simultaneously is to not be artificial in the first place.

When Clura runs inside your actual Chrome browser — your real machine, your real residential IP, your real session cookies — the TLS handshake is Chrome's. The behavioral signals are real because you're actually on the page. The data serving layer sees an authenticated shopper, not a bot session.

You're not pretending to be a browser. You are using a browser.

This is a fundamentally different architecture from proxy rotation, stealth Playwright patches, or CAPTCHA-solving services. Those approaches try to make an artificial session look real. Browser-native scraping starts from a real session and never introduces anything artificial to detect. The same principle applies whether you're scraping Amazon product data, Google Maps listings, or leads from LinkedIn. For a comparison of which Chrome extension scrapers use this approach vs which still rely on HTTP requests, see our tools breakdown.

In our testing: Python scrapers fail on Walmart 86–92% of the time. Playwright with residential proxies fails 56–64% of the time. Browser-native scraping fails 8–11% of the time — and the failures are almost always session timeouts, not detection events.