What web scraping really costs per 1,000 pages

Clients usually ask what a scraper costs to build. The more useful question is what it costs to run, every month, for as long as they need the data. A cheap build on the wrong approach can cost more in proxies within a few months than the build itself.

This is the model I use to estimate running cost before writing any code.

The four costs

1. Proxies. Usually the largest variable cost on protected sites. Residential and mobile proxies are mostly billed per GB of traffic.
2. Compute. Servers or containers to run the crawlers. Cheap for HTTP scraping, much more for browsers.
3. Storage and delivery. Database, object storage, and whatever sends the data to the client. Usually small.
4. Maintenance. Engineering time when the site changes or anti-bot rules tighten. Often the largest cost over a year, and the one most estimates leave out.

Proxies: page weight × volume × price per GB

Per-GB billing means how much data each page transfers matters as much as how many pages you fetch:

These are rough ranges. Measure your own target in the DevTools network tab: the “transferred” total at the bottom is what your proxy bills.

The formula:

monthly proxy cost = pages per month × MB per page ÷ 1,024 × price per GB

Two levers cut this dramatically:

• Use the API instead of the page when one exists. See Scrapy vs Playwright vs the hidden API.
• Block what you don’t need in browsers: images, fonts, media and third-party scripts. Page weight can often drop by more than half.

Also route unprotected requests (sitemaps, static assets, lightly protected pages) through cheap datacenter proxies or no proxy at all. Save the residential traffic for pages that need it.

Compute: HTTP is cheap, browsers are not

A single small server can run Scrapy at hundreds of concurrent requests. A headless browser needs real CPU and memory per open page, so the same server handles only a handful of browser pages at once.

In practice, browser-based scraping often needs 5–20× the compute of an HTTP approach for the same page count, and it’s slower. For a small daily crawl that’s still cheap. For millions of pages a month, it decides the architecture.

Maintenance: budget it, or it budgets you

Sites change layouts, add fields, move to a new frontend or tighten their anti-bot settings. Each change means a fix. How often depends on the site: some stay stable for a year, some need attention monthly.

This is why I price ongoing work as a monitored data feed rather than hoping nothing breaks. The monitoring catches the break (zero rows, a rising block rate), and the fix is already paid for.

A worked example

Illustrative numbers only. Plug in your own page weights and current proxy prices.

A client wants 50 retailers’ product pages checked daily, about 20,000 pages a day, or 600,000 pages a month.

Same data, but running costs roughly 20× apart. The difference comes almost entirely from the approach, decided before any code is written. That’s why a paid phase-1 on the real targets matters: it measures the true cost per record before the client commits.

Cost per record, not per page

Clients don’t care about pages. They care about records: products, listings, filings. Divide total monthly cost by delivered, deduplicated, valid records:

cost per record = (proxies + compute + storage + maintenance) ÷ valid records delivered

That number makes it easy to compare approaches and vendors, and to decide whether the data is worth buying at all.

FAQ

Why not just use a scraping API service? Services that handle proxies and unblocking are great for starting quickly and for low volumes. At high volume, a custom setup with smart routing is often much cheaper. Compare them on cost per record, not on price per request.

Datacenter or residential proxies? Start with datacenter proxies and move a source to residential only when its block rate proves it’s needed. Many sites don’t need residential at all.

How do I estimate maintenance? Look at how often the site has been redesigned, how aggressive its anti-bot protection is, and how many sources you have. Then budget a monthly allowance of hours instead of guessing per incident.

What’s the cheapest way to cut costs on an existing scraper? Measure transfer per page first. Blocking images, fonts and third-party scripts, or switching to the site’s API, usually saves more than negotiating proxy prices.