Scrapy vs Playwright vs the hidden API: how to choose

“Should I use Scrapy or Playwright?” is the question I get most often, and the honest answer is usually “neither, first.” Before choosing a tool, check where the data actually comes from. Often the page is only a shell, and the data arrives from an API you can call directly.

This is the order I work through for every new site.

Step 1: Open the network tab before writing any code

Load the page with DevTools open, filter by Fetch/XHR, and reload. Look for responses that contain the data you want, as JSON. Common places:

• A REST endpoint like /api/products?page=2
• A GraphQL endpoint (/graphql) with a query in the request body
• Data embedded in the HTML as JSON, in a <script id="__NEXT_DATA__"> tag or window.__INITIAL_STATE__

If you find it, you usually don’t need to parse HTML at all:

import httpx

r = httpx.get(
    "https://example.com/api/products",
    params={"page": 2, "per_page": 100},
    headers={"accept": "application/json"},
)
items = r.json()["items"]

Why the API wins: structured data, fewer requests (often 100 items per call instead of 20 per page), and no selectors that break when the design changes. Frontends get redesigned far more often than their APIs.

Watch for: auth tokens, signed parameters and rate limits. If a token comes from an earlier response, trace where it’s set and fetch it the same way the browser does.

Step 2: Is the HTML server-rendered? Use Scrapy

Right-click → View page source (not Inspect). If the data is in the raw HTML, the site is server-rendered, and Scrapy is the right tool:

• Fast and asynchronous, with hundreds of concurrent requests on a small server
• Built-in retries, throttling, caching and export pipelines
• Very cheap to run compared with a browser

import scrapy

class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/catalog"]

    def parse(self, response):
        for card in response.css("article.product"):
            yield {
                "url": response.urljoin(card.css("a::attr(href)").get()),
                "title": card.css("h2::text").get(default="").strip(),
                "price": card.css(".price::text").get(),
            }
        yield from response.follow_all(css="a.next", callback=self.parse)

For a full walkthrough, see Scrapy part 1.

Step 3: Rendered by JavaScript and no usable API? Use a browser

Use Playwright (or a stealth-focused browser) when:

• The data only appears after JavaScript runs, and you can’t find or reproduce the API call
• The site’s anti-bot challenge requires a real browser to pass
• You need to interact: click through filters, scroll infinite lists, submit forms

The cost: a browser page uses far more CPU, memory and bandwidth than an HTTP request. It’s easily an order of magnitude more expensive per page, and slower. On a big crawl that’s the difference between one small server and a fleet.

Step 4: Often the best answer is both

Most production scrapers I build are hybrids:

• Browser for the session, HTTP for the data. Use Playwright to pass the challenge and log in, then hand the cookies to a fast HTTP client for thousands of API calls.
• scrapy-playwright. Keep Scrapy’s scheduling and pipelines, and render only the pages that need it with meta={"playwright": True}.
• Browser for discovery, Scrapy for detail pages. The listing is JavaScript-heavy, but the detail pages are server-rendered.

The decision table

FAQ

Is using a site’s internal API allowed? It’s the same data the site sends to every visitor’s browser, but the rules are the site’s terms and your local law. Stick to public data, respect rate limits, and avoid personal data.

Isn’t Playwright easier for beginners? It feels easier because it “just shows the page”. But it’s slower, costs more to run, and breaks just as easily on layout changes. Spend 10 minutes in the network tab first.

Can Scrapy handle JavaScript? Not by itself. Add scrapy-playwright for the pages that need it, or better, call the API that the JavaScript uses.

What about login-protected data? Log in with a browser or an HTTP session, keep the cookies, and reuse them. Only scrape behind a login when you have the right to access that data.