Surendra Tamang

From a scraper to a monitored data pipeline

Sat, 26 Sep 2026 00:00:00 GMT

Most scraping projects end the same way. The script works on delivery day, the client is happy, and three weeks later someone notices the dashboard hasn't changed since the 4th. Nobody got an error. The scraper ran, found nothing, and exited cleanly.

The scraper wasn't the problem. Everything around it was missing. This post covers the parts I add to every scraper before I call it done: scheduling, alerting, deduplication, change history, and a clean handoff.

Why scrapers die in week two

Scrapers rarely crash loudly. They break in a few quiet ways:

Layout changes. A class name changes, the selector matches nothing, and the run returns zero rows with exit code 0.
Soft blocks. Cloudflare or DataDome serves a challenge page with HTTP 200. Your parser reads the challenge page, finds no products, and moves on.
Burned proxies. A proxy pool that worked in testing gets flagged. Success rate drops from 98% to 40%, and the dataset quietly shrinks.
Duplicates. The same listing gets scraped from page 3 today and page 4 tomorrow. Counts go up, but the data doesn't.

None of these throw an exception. So "it didn't error" tells you nothing about whether the data is right.

Scheduling: cron is fine until it isn't

For one scraper on one server, cron is fine. Don't reach for Airflow on day one.

# every day at 02:15, log output, never overlap runs
15 2 * * * flock -n /tmp/prices.lock /app/run.sh >> /var/log/prices.log 2>&1

flock -n matters more than it looks. Without it, a slow run overlaps the next one and you get two scrapers fighting over the same proxy pool.

Cron stops being enough when you need:

Retries with backoff for one failed source, without rerunning all of them
Backfills, such as "rerun last Tuesday for source B"
Dependencies, such as "load to the warehouse only after all 12 sources finish"
A place to see history of what ran, how long it took and what failed

That's the point to move to an orchestrator like Airflow. Moving is easy if each source is already its own command with a date argument, like run.sh --source b --date 2026-09-30. Design for that from day one, even on cron.

Alert on zero rows, not on exceptions

Exceptions are the failures you already handle. The dangerous ones are silent, so check the output, not the exit code.

After every run I record a few numbers per source and compare them with recent history:

def check_run(source: str, rows: int, recent: list[int]) -> list[str]:
    problems = []
    if rows == 0:
        problems.append(f"{source}: zero rows")
    elif recent and rows < 0.5 * (sum(recent) / len(recent)):
        problems.append(f"{source}: {rows} rows, under half the recent average")
    return problems

Add three more checks and you catch most real failures:

Freshness: the newest record is older than expected.
Nulls: a required field (price, title, URL) is suddenly empty on many rows. That usually means a selector broke.
Block rate: the share of responses that were challenge pages or 403s. Rising block rate is the early warning before rows drop.

Send problems to a place people actually read, like Slack or email. Send nothing when everything is fine. An alert channel that posts "all good" every day gets muted within a week.

Deduplication and change history in PostgreSQL

Scraped data needs two things most scripts don't give you: one row per real-world thing, and a record of how it changed.

Pick a natural key per source. That's usually the site's own ID or a normalised URL, never the scrape timestamp. Then upsert:

create table listings (
    source      text not null,
    external_id text not null,
    title       text,
    price       numeric,
    first_seen  timestamptz not null default now(),
    last_seen   timestamptz not null default now(),
    primary key (source, external_id)
);

create table listing_changes (
    source      text not null,
    external_id text not null,
    field       text not null,
    old_value   text,
    new_value   text,
    changed_at  timestamptz not null default now()
);

insert into listings (source, external_id, title, price)
values ($1, $2, $3, $4)
on conflict (source, external_id) do update
set title = excluded.title,
    price = excluded.price,
    last_seen = now();

Before the upsert, compare the incoming row with the stored one and write any differences to listing_changes. Now "what was the price on the 12th?" and "which listings disappeared this week?" are simple queries. You don't have to rescrape.

Two things this gives you for free:

Reruns are safe. Running the same day twice updates rows instead of doubling them, so a retry can never corrupt the dataset.
Disappearances show up. A listing whose last_seen stops moving was removed or sold. For price and inventory monitoring, that's often the most valuable signal in the data.

Handoff: the client owns the code and the accounts

A pipeline that only I can run isn't a deliverable. At handoff the client gets:

The code in their repository, with a README that says how to run one source for one date
Proxy, server and alerting accounts in their name, or clear instructions to move them
A short video walkthrough of where the data lands, what the alerts mean, and what to do when one fires
A list of known risks, such as which sources are fragile and what changes most often

Then they choose. They run it themselves, or I keep it running as a monitored data feed, and a site change becomes my problem instead of theirs. Both are fine. What isn't fine is a scraper nobody watches.

FAQ

Do I need Airflow for a single scraper? No. Cron with flock and output checks is enough for a few sources. Move to an orchestrator when you need retries per source, backfills or dependencies.

What should I alert on first? Zero rows and sudden drops in row count. They catch layout changes, soft blocks and proxy problems before anyone downstream notices.

How do I stop duplicate records? Use a natural key from the source (site ID or normalised URL) as the primary key, and upsert instead of insert. Reruns then update rows instead of adding new ones.

How often do scrapers need fixing? It depends on the site. Heavily protected or frequently redesigned sites need attention more often than stable ones. That's why monitoring matters more than the first version of the code.

LLM extraction that never invents a value

Wed, 17 Jun 2026 00:00:00 GMT

LLMs are very good at reading messy documents: court filings, invoices, product pages with no consistent layout. They're also very good at producing an answer that looks right when the document doesn't contain one. For data extraction, that second habit is the real risk.

This is the setup I use so that a wrong value is caught, not shipped.

The failure mode: confident, wrong, unnoticed

Traditional scrapers fail loudly. A selector matches nothing, and you get an empty field. LLMs fail quietly:

A missing date gets filled with the document's filing date instead of the hearing date.
A price with no currency gets USD because that's the most common currency.
Two similar parties get merged, or a name gets a middle initial it never had.

Every output is valid JSON and every value is plausible. Without measurement, you won't know your accuracy is 85% until a client finds the other 15%.

Schema-first: tool use and JSON schema

Never ask for "JSON with these fields" in plain prompt text. Define the schema and make the model fill it through tool use / structured outputs, which both Anthropic and OpenAI support. Make every uncertain field nullable and tell the model that null is the correct answer when the value isn't in the text.

import os

import anthropic

client = anthropic.Anthropic()

CASE_TOOL = {
    "name": "record_case",
    "description": "Record fields found in the document. Use null when a value is not stated.",
    "input_schema": {
        "type": "object",
        "properties": {
            "case_number": {"type": ["string", "null"]},
            "filing_date": {"type": ["string", "null"], "description": "YYYY-MM-DD"},
            "case_type": {"type": ["string", "null"], "enum": ["civil", "criminal", "family", "traffic", None]},
            "amount_claimed": {"type": ["number", "null"]},
        },
        "required": ["case_number", "filing_date", "case_type", "amount_claimed"],
    },
}

def extract(text: str) -> dict:
    msg = client.messages.create(
        model=os.environ["EXTRACTION_MODEL"],
        max_tokens=1024,
        tools=[CASE_TOOL],
        tool_choice={"type": "tool", "name": "record_case"},
        messages=[{"role": "user", "content": f"Extract the case fields.\n\n<document>\n{text}\n</document>"}],
    )
    return next(b.input for b in msg.content if b.type == "tool_use")

Enums stop invented categories. Nullable fields give the model a legitimate way to say "not found".

Validation rules the model cannot bypass

The schema controls shape. Validation controls truth. After extraction, run checks in plain code:

import re
from datetime import date

def validate(fields: dict, text: str) -> list[str]:
    errors = []
    cn = fields.get("case_number")
    if cn and cn not in text:
        errors.append("case_number not found verbatim in document")
    fd = fields.get("filing_date")
    if fd:
        try:
            if date.fromisoformat(fd) > date.today():
                errors.append("filing_date in the future")
        except ValueError:
            errors.append("filing_date not a valid date")
    amt = fields.get("amount_claimed")
    if amt is not None and not re.search(rf"{int(amt):,}|{int(amt)}", text):
        errors.append("amount_claimed not present in document")
    return errors

The most useful single rule is grounding: identifiers, names and amounts must appear in the source text. It catches most invented values with a few lines of code. Add range checks, date sanity checks and cross-field rules (a closing date can't come before the filing date) as you find real errors.

Building a golden set and measuring accuracy

You can't improve what you don't measure. Build a golden set:

Pick 50–200 real documents that cover the variety: clean ones, scanned ones, odd layouts, documents where fields are genuinely missing.
Label the correct values by hand, including null where a field isn't present.
Run the pipeline and compare per field.

def score(golden: list[dict], predicted: list[dict], field: str) -> dict:
    pairs = [(g[field], p[field]) for g, p in zip(golden, predicted)]
    correct = sum(g == p for g, p in pairs)
    invented = sum(g is None and p is not None for g, p in pairs)
    return {"accuracy": correct / len(pairs), "invented": invented}

Track invented values separately from misses. A missed value is an empty cell someone can fill. An invented value is wrong data someone will trust.

Rerun the golden set every time you change the prompt, the model or the schema. Prompts that "look better" regularly make one field worse.

Fail closed: when to route to a human

Decide up front what happens when a record fails validation:

Retry once with the validation errors included in the prompt. This often fixes format errors.
Route to a review queue if it still fails. A person checks a few records a day instead of the whole dataset.
Never write failed records into the main table as if they were good.

On a well-tuned pipeline the review queue is a small share of volume, and it's also your best source of new golden-set examples.

Keeping cost under control

Send only what's needed. Strip boilerplate, navigation and repeated headers before extraction.
Cache by document hash. Unchanged documents never get extracted twice.
Start with a smaller model and escalate to a larger one only for records that fail validation.
Measure cost per valid record, not per API call. It's the same principle as scraping cost per 1,000 pages.

FAQ

Claude or OpenAI? Both support structured outputs well. Pick per project by running the same golden set through each and comparing accuracy, invented values and cost.

Do I still need OCR for scanned PDFs? Often, yes. Modern models can read page images directly, but a text layer from OCR is cheaper and easier to ground-check. Test both on your golden set.

How big should the golden set be? Start with 50 documents to catch obvious problems. Grow it to a few hundred, weighted toward the document types that fail most.

Can I trust the model's own confidence score? Not by itself. Use validation rules and golden-set accuracy instead. They measure what actually happened.

What web scraping really costs per 1,000 pages

Wed, 11 Feb 2026 00:00:00 GMT

Clients usually ask what a scraper costs to build. The more useful question is what it costs to run, every month, for as long as they need the data. A cheap build on the wrong approach can cost more in proxies within a few months than the build itself.

This is the model I use to estimate running cost before writing any code.

The four costs

Proxies. Usually the largest variable cost on protected sites. Residential and mobile proxies are mostly billed per GB of traffic.
Compute. Servers or containers to run the crawlers. Cheap for HTTP scraping, much more for browsers.
Storage and delivery. Database, object storage, and whatever sends the data to the client. Usually small.
Maintenance. Engineering time when the site changes or anti-bot rules tighten. Often the largest cost over a year, and the one most estimates leave out.

Proxies: page weight × volume × price per GB

Per-GB billing means how much data each page transfers matters as much as how many pages you fetch:

Approach	Typical transfer per page
JSON API call	a few KB to ~100 KB
Server-rendered HTML (Scrapy)	~50–500 KB
Full browser page load	~1–5 MB, because of scripts, images, fonts and trackers

These are rough ranges. Measure your own target in the DevTools network tab: the "transferred" total at the bottom is what your proxy bills.

The formula:

monthly proxy cost = pages per month × MB per page ÷ 1,024 × price per GB

Two levers cut this dramatically:

Use the API instead of the page when one exists. See Scrapy vs Playwright vs the hidden API.
Block what you don't need in browsers: images, fonts, media and third-party scripts. Page weight can often drop by more than half.

Also route unprotected requests (sitemaps, static assets, lightly protected pages) through cheap datacenter proxies or no proxy at all. Save the residential traffic for pages that need it.

Compute: HTTP is cheap, browsers are not

A single small server can run Scrapy at hundreds of concurrent requests. A headless browser needs real CPU and memory per open page, so the same server handles only a handful of browser pages at once.

In practice, browser-based scraping often needs 5–20× the compute of an HTTP approach for the same page count, and it's slower. For a small daily crawl that's still cheap. For millions of pages a month, it decides the architecture.

Maintenance: budget it, or it budgets you

Sites change layouts, add fields, move to a new frontend or tighten their anti-bot settings. Each change means a fix. How often depends on the site: some stay stable for a year, some need attention monthly.

This is why I price ongoing work as a monitored data feed rather than hoping nothing breaks. The monitoring catches the break (zero rows, a rising block rate), and the fix is already paid for.

A worked example

Illustrative numbers only. Plug in your own page weights and current proxy prices.

A client wants 50 retailers' product pages checked daily, about 20,000 pages a day, or 600,000 pages a month.

	Browser, residential proxies	Hidden API + HTTP, residential only where needed
Transfer per page	2 MB	0.1 MB
Monthly transfer	~1,170 GB	~59 GB
Proxy cost at an assumed $4/GB	~$4,700	~$235
Compute	several browser servers	one small server

Same data, but running costs roughly 20× apart. The difference comes almost entirely from the approach, decided before any code is written. That's why a paid phase-1 on the real targets matters: it measures the true cost per record before the client commits.

Cost per record, not per page

Clients don't care about pages. They care about records: products, listings, filings. Divide total monthly cost by delivered, deduplicated, valid records:

cost per record = (proxies + compute + storage + maintenance) ÷ valid records delivered

That number makes it easy to compare approaches and vendors, and to decide whether the data is worth buying at all.

FAQ

Why not just use a scraping API service? Services that handle proxies and unblocking are great for starting quickly and for low volumes. At high volume, a custom setup with smart routing is often much cheaper. Compare them on cost per record, not on price per request.

Datacenter or residential proxies? Start with datacenter proxies and move a source to residential only when its block rate proves it's needed. Many sites don't need residential at all.

How do I estimate maintenance? Look at how often the site has been redesigned, how aggressive its anti-bot protection is, and how many sources you have. Then budget a monthly allowance of hours instead of guessing per incident.

What's the cheapest way to cut costs on an existing scraper? Measure transfer per page first. Blocking images, fonts and third-party scripts, or switching to the site's API, usually saves more than negotiating proxy prices.

Scheduling Scrapy with Airflow: retries, backfills, alerts

Wed, 08 Oct 2025 00:00:00 GMT

Cron is fine for one spider. Once you have ten sources feeding the same database, the gaps show: one failed source means rerunning everything, nobody can see what ran last night, and backfilling "last Tuesday for source B" means editing a crontab by hand.

This is how I move Scrapy projects to Apache Airflow without rewriting the spiders.

Make every spider rerunnable first

Airflow can only retry and backfill safely if a run for a given date is idempotent: running it twice gives the same result as running it once. Two things make that true:

The spider takes the logical date as an argument, instead of reading "today" from the clock:

class PricesSpider(scrapy.Spider):
    name = "prices"

    def __init__(self, run_date=None, **kwargs):
        super().__init__(**kwargs)
        self.run_date = run_date  # "2025-10-07", set by Airflow

The pipeline upserts on a natural key (the site's own ID) instead of inserting. Reruns then update rows instead of duplicating them. The schema for this is in From a scraper to a monitored data pipeline.

If your spiders already work like that, the Airflow part is small.

One DAG, one task per source

This example uses Airflow 3's airflow.sdk imports; on Airflow 2.x the same ideas apply with airflow.decorators and airflow.operators.bash.

from datetime import datetime, timedelta

from airflow.providers.standard.operators.bash import BashOperator
from airflow.sdk import dag, task

SOURCES = ["retailer_a", "retailer_b", "retailer_c"]

@dag(
    schedule="15 2 * * *",
    start_date=datetime(2025, 10, 1),
    catchup=False,
    default_args={
        "retries": 3,
        "retry_delay": timedelta(minutes=10),
        "retry_exponential_backoff": True,
    },
)
def prices():
    for source in SOURCES:
        crawl = BashOperator(
            task_id=f"crawl_{source}",
            bash_command=f"cd /opt/scrapers && scrapy crawl {source} -a run_date={{{{ ds }}}}",
            pool="residential_proxies",
        )

        @task(task_id=f"check_{source}")
        def check(source=source, ds=None):
            rows = count_rows(source, ds)  # your query against the target table
            if rows == 0:
                raise ValueError(f"{source}: zero rows for {ds}")

        crawl >> check()

prices()

What each part gives you:

One task per source. Retailer B failing doesn't block A and C, and a retry reruns only B.
{{ ds }}: Airflow passes the run's logical date, so a rerun of Tuesday scrapes as Tuesday.
Retries with exponential backoff: temporary blocks and proxy hiccups often clear on their own after a few minutes.
A check task after each crawl: a spider that exits cleanly with zero rows now fails loudly, which is the most common silent failure in scraping.

Pools: limit concurrency by proxy, not by server

Scrapy already controls concurrency inside one spider. The problem is five spiders starting at 02:15 and all hitting the same residential proxy pool at once.

An Airflow pool caps how many tasks can use a shared resource at the same time:

airflow pools set residential_proxies 2 "Paid residential proxy pool"

Now at most two proxy-heavy crawls run together, and the rest queue. Sources on cheap datacenter IPs can use a different pool with more slots.

Backfills without editing anything

A site was down for three days, or you fixed a parser bug and want to rebuild last week. With idempotent spiders, that's one command:

# Airflow 3
airflow backfill create --dag-id prices --from-date 2025-09-29 --to-date 2025-10-05

# Airflow 2.x
airflow dags backfill prices -s 2025-09-29 -e 2025-10-05

Each day runs with its own ds, and the upserts make the reruns safe.

Alerts that fire on bad data

Airflow can call a function whenever a task fails. Send that to Slack or email:

def notify(context):
    ti = context["task_instance"]
    post_to_slack(f"{ti.dag_id}.{ti.task_id} failed for {context['ds']}: {ti.log_url}")

# add to default_args: "on_failure_callback": notify

Because the check tasks raise on zero or low row counts, this one callback covers both crashes and silent failures. It stays quiet when everything is fine, so people keep reading it.

When Airflow is overkill

Stay on cron if you have one or two spiders, no downstream dependencies, and someone checks the output anyway. Airflow is a service you have to run and upgrade. Move when retries per source, backfills, dependencies or run history start costing you real time.

FAQ

Should the spider run inside the Airflow worker? For small setups, yes (a BashOperator is fine). For heavier crawls, run each spider in its own container with KubernetesPodOperator or DockerOperator, so a memory-hungry browser crawl can't take down the scheduler.

What about Scrapyd or Zyte's Scrapy Cloud? They're good for running spiders. Airflow adds the orchestration around them: dependencies, checks, backfills and one place to see everything. You can trigger Scrapyd jobs from Airflow too.

How do I load into a warehouse after all sources finish? Add a final task downstream of every check task. It only runs when all sources pass, so your warehouse never gets a half-loaded day.

Airflow, Prefect or Dagster? All three work. Airflow has the largest ecosystem and is the one most job postings ask for, which matters if clients will maintain it after you.

Scrapy vs Playwright vs the hidden API: how to choose

Wed, 18 Jun 2025 00:00:00 GMT

"Should I use Scrapy or Playwright?" is the question I get most often, and the honest answer is usually "neither, first." Before choosing a tool, check where the data actually comes from. Often the page is only a shell, and the data arrives from an API you can call directly.

This is the order I work through for every new site.

Step 1: Open the network tab before writing any code

Load the page with DevTools open, filter by Fetch/XHR, and reload. Look for responses that contain the data you want, as JSON. Common places:

A REST endpoint like /api/products?page=2
A GraphQL endpoint (/graphql) with a query in the request body
Data embedded in the HTML as JSON, in a <script id="__NEXT_DATA__"> tag or window.__INITIAL_STATE__

If you find it, you usually don't need to parse HTML at all:

import httpx

r = httpx.get(
    "https://example.com/api/products",
    params={"page": 2, "per_page": 100},
    headers={"accept": "application/json"},
)
items = r.json()["items"]

Why the API wins: structured data, fewer requests (often 100 items per call instead of 20 per page), and no selectors that break when the design changes. Frontends get redesigned far more often than their APIs.

Watch for: auth tokens, signed parameters and rate limits. If a token comes from an earlier response, trace where it's set and fetch it the same way the browser does.

Step 2: Is the HTML server-rendered? Use Scrapy

Right-click → View page source (not Inspect). If the data is in the raw HTML, the site is server-rendered, and Scrapy is the right tool:

Fast and asynchronous, with hundreds of concurrent requests on a small server
Built-in retries, throttling, caching and export pipelines
Very cheap to run compared with a browser

import scrapy

class ProductsSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com/catalog"]

    def parse(self, response):
        for card in response.css("article.product"):
            yield {
                "url": response.urljoin(card.css("a::attr(href)").get()),
                "title": card.css("h2::text").get(default="").strip(),
                "price": card.css(".price::text").get(),
            }
        yield from response.follow_all(css="a.next", callback=self.parse)

For a full walkthrough, see Scrapy part 1.

Step 3: Rendered by JavaScript and no usable API? Use a browser

Use Playwright (or a stealth-focused browser) when:

The data only appears after JavaScript runs, and you can't find or reproduce the API call
The site's anti-bot challenge requires a real browser to pass
You need to interact: click through filters, scroll infinite lists, submit forms

The cost: a browser page uses far more CPU, memory and bandwidth than an HTTP request. It's easily an order of magnitude more expensive per page, and slower. On a big crawl that's the difference between one small server and a fleet.

Step 4: Often the best answer is both

Most production scrapers I build are hybrids:

Browser for the session, HTTP for the data. Use Playwright to pass the challenge and log in, then hand the cookies to a fast HTTP client for thousands of API calls.
scrapy-playwright. Keep Scrapy's scheduling and pipelines, and render only the pages that need it with meta={"playwright": True}.
Browser for discovery, Scrapy for detail pages. The listing is JavaScript-heavy, but the detail pages are server-rendered.

The decision table

Situation	Use
JSON API or embedded JSON exists	HTTP client on the API
Data is in the page source	Scrapy
JS-rendered, API reproducible	HTTP client on the API
JS-rendered, API unusable	Playwright / browser
Anti-bot challenge needs a browser	Browser for session, HTTP for data
Millions of pages	Scrapy or HTTP client. Browsers only where unavoidable.

FAQ

Is using a site's internal API allowed? It's the same data the site sends to every visitor's browser, but the rules are the site's terms and your local law. Stick to public data, respect rate limits, and avoid personal data.

Isn't Playwright easier for beginners? It feels easier because it "just shows the page". But it's slower, costs more to run, and breaks just as easily on layout changes. Spend 10 minutes in the network tab first.

Can Scrapy handle JavaScript? Not by itself. Add scrapy-playwright for the pages that need it, or better, call the API that the JavaScript uses.

What about login-protected data? Log in with a browser or an HTTP session, keep the cookies, and reuse them. Only scrape behind a login when you have the right to access that data.

How antibot systems fingerprint your scraper

Wed, 12 Mar 2025 00:00:00 GMT

When a scraper gets blocked, most people change the User-Agent and add a proxy. Sometimes that works for a day. Then the blocks come back, because the User-Agent was never the main thing being checked.

Cloudflare, DataDome, Akamai, PerimeterX (HUMAN) and Kasada all score many signals together. This post walks through them in the order a request actually meets them, and ends with what holds up in production.

What a protection vendor sees before your first request finishes

Before your HTTP request is even read, the server already knows several things:

Your IP and its network. Is it a datacenter range (AWS, Hetzner, OVH), a residential ISP or a mobile carrier? Has this IP been seen on other protected sites recently?
Your TLS handshake. Which ciphers, extensions and curves your client offered, in what order.
Your HTTP/2 connection. The SETTINGS values, window sizes and pseudo-header order your client sends.

Only then come headers, cookies, JavaScript challenges and behaviour. Each layer adds to a risk score. Crossing a threshold gets you a challenge page, a block or, worse, quietly fake data.

TLS fingerprinting: you're identified at the handshake

Every TLS client says hello differently. Chrome, Firefox, Python requests, Go's net/http and Node all send a distinct ClientHello. Hashing its fields gives a fingerprint. JA3 was the original; JA4 is the newer, more robust version.

This is why this fails:

import requests
requests.get(url, headers={"User-Agent": "Mozilla/5.0 ... Chrome/124.0 ..."})

The header claims Chrome, but the TLS handshake says Python urllib3. That mismatch is one of the cheapest signals a vendor can check, and one of the most reliable.

The fix is a client that reproduces a real browser handshake, like curl_cffi:

from curl_cffi import requests

r = requests.get(url, impersonate="chrome")

Now the TLS and HTTP/2 layers look like Chrome. Headers alone never could.

HTTP/2 frames and header order

HTTP/2 adds another fingerprint. Browsers send specific SETTINGS values (header table size, initial window size, max concurrent streams), a WINDOW_UPDATE of a specific size, and pseudo-headers (:method, :authority, :scheme, :path) in a fixed order. Each browser family does this slightly differently, and HTTP libraries differ from all of them.

Regular header order matters too. Chrome sends sec-ch-ua, sec-ch-ua-mobile and sec-ch-ua-platform near the top, then user-agent, accept and the sec-fetch-* headers in a consistent order. A request with the right headers in the wrong order, or with sec-ch-ua claiming Chrome 124 while the User-Agent says 120, stands out.

The rule: every layer must tell the same story. IP location, TLS, HTTP/2, headers, and later the JavaScript environment all need to agree on who you are.

Browser-side signals: canvas, fonts, and API probing

If the site serves a JavaScript challenge, it runs code in your browser and reports back. Typical probes:

Automation flags: navigator.webdriver, and traces left by the Chrome DevTools Protocol that Playwright and Puppeteer use.
Environment consistency: does navigator.platform match the User-Agent? Do the screen size, timezone and language match the IP's country?
Rendering: canvas and WebGL output, the GPU renderer string, and installed fonts. Headless browsers on Linux servers produce recognisable results.
API behaviour: how native functions stringify, whether properties were patched, and timing quirks that differ in headless mode.
Behaviour: mouse movement, scrolling, and time between page load and interaction.

Challenge scripts are obfuscated and change often, so the exact checks shift. The categories stay the same.

What actually works in production

No single trick. It's a set of habits:

Use the lightest client that passes. Try a TLS-impersonating HTTP client first. It's fast and cheap. Go to a real browser only for sites whose JavaScript challenge requires one.
When you need a browser, use one built for it. Patched or stealth-focused browsers (for example Camoufox, or Chromium driven without the usual automation leaks) pass far more often than stock headless Chrome.
Match the proxy to the target. Datacenter IPs are fine for unprotected sites. Heavily protected ones need residential or mobile IPs, geolocated to match your browser's timezone and language.
Keep sessions sticky. Reuse one IP, one fingerprint and one cookie jar per session. Rotating the IP every request while keeping the same cookies is a red flag, not camouflage.
Slow down. Most blocks I debug come from request rate, not fingerprints. Realistic concurrency per IP beats clever evasion.
Measure block rate. Log challenge pages and 403s per source. A rising block rate warns you before your data drops. That's the core of monitoring a scraper properly.

And stay on the right side of the line. Collect publicly available data, respect rate limits, avoid personal data you don't need, and read the site's terms before you start.

FAQ

Why do I get blocked even with residential proxies? Because the IP is only one signal. If your TLS fingerprint says Python while your User-Agent says Chrome, a clean residential IP won't save you.

Is headless Chrome enough? For lightly protected sites, often yes. For DataDome, Kasada or strict Cloudflare settings, stock headless Chrome is easy to detect. Use a stealth-focused browser, or find the site's underlying API.

What's JA3 and JA4? Ways to hash a TLS ClientHello into a fingerprint. JA4 is the newer format, and it's harder to fool by shuffling extension order.

Can I bypass these systems permanently? No. Vendors update their checks continuously. What lasts is a setup that detects blocks early and adapts: monitoring, a fallback client, and time budgeted for maintenance.

Scrapy, part 5: deploying to production

Wed, 25 Dec 2024 00:00:00 GMT

Scrapy Part 5: Production Deployment

Final part of the series. The scraper works; now it has to run unattended. This part covers packaging it with a multi-stage Docker build, running it on Kubernetes with autoscaling, wiring up a CI/CD pipeline with automated tests, monitoring it with Prometheus and Grafana, and hardening it for production.

Docker Containerization

Multi-Stage Dockerfile

# webscraper/Dockerfile
# Multi-stage build for optimized production image

# Build stage
FROM python:3.11-slim as builder

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    libxml2-dev \
    libxslt-dev \
    libffi-dev \
    libssl-dev \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Create and activate virtual environment
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install Python dependencies
COPY requirements.txt .
RUN pip install --upgrade pip && \
    pip install -r requirements.txt

# Production stage
FROM python:3.11-slim as production

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PATH="/opt/venv/bin:$PATH"

# Create non-root user
RUN groupadd -r scrapy && useradd -r -g scrapy scrapy

# Install runtime dependencies
RUN apt-get update && apt-get install -y \
    curl \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Copy virtual environment from builder stage
COPY --from=builder /opt/venv /opt/venv

# Create application directory
WORKDIR /app

# Copy application code
COPY --chown=scrapy:scrapy . .

# Create necessary directories
RUN mkdir -p /app/logs /app/data /app/exports && \
    chown -R scrapy:scrapy /app

# Switch to non-root user
USER scrapy

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:6023/ || exit 1

# Default command
CMD ["scrapy", "list"]

# Development stage
FROM production as development

USER root

# Install development dependencies
RUN pip install pytest pytest-cov black flake8 mypy

# Install debugging tools
RUN apt-get update && apt-get install -y \
    vim \
    htop \
    net-tools \
    && rm -rf /var/lib/apt/lists/*

USER scrapy

# Override default command for development
CMD ["tail", "-f", "/dev/null"]

Docker Compose for Development

# docker-compose.yml
version: '3.8'

services:
  # Main scraper service
  scraper:
    build:
      context: .
      target: development
      dockerfile: Dockerfile
    volumes:
      - .:/app
      - ./data:/app/data
      - ./logs:/app/logs
    environment:
      - SCRAPY_SETTINGS_MODULE=webscraper.settings.development
      - REDIS_URL=redis://redis:6379/0
      - MONGO_URI=mongodb://mongo:27017/scrapy_dev
      - POSTGRES_HOST=postgres
      - POSTGRES_DB=scrapy_dev
      - POSTGRES_USER=scrapy
      - POSTGRES_PASSWORD=scrapy_password
    depends_on:
      - redis
      - mongo
      - postgres
    networks:
      - scrapy-network

  # Redis for distributed scraping
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes
    networks:
      - scrapy-network

  # MongoDB for document storage
  mongo:
    image: mongo:6
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db
    environment:
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin_password
    networks:
      - scrapy-network

  # PostgreSQL for structured data
  postgres:
    image: postgres:15
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=scrapy_dev
      - POSTGRES_USER=scrapy
      - POSTGRES_PASSWORD=scrapy_password
    networks:
      - scrapy-network

  # Elasticsearch for search and analytics
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.0
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    networks:
      - scrapy-network

  # Kibana for data visualization
  kibana:
    image: docker.elastic.co/kibana/kibana:8.8.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - scrapy-network

  # Prometheus for monitoring
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
    networks:
      - scrapy-network

  # Grafana for dashboards
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
      - ./monitoring/grafana:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus
    networks:
      - scrapy-network

  # Splash for JavaScript rendering
  splash:
    image: scrapinghub/splash:latest
    ports:
      - "8050:8050"
    command: --max-timeout=3600 --slots=5
    networks:
      - scrapy-network

volumes:
  redis-data:
  mongo-data:
  postgres-data:
  es-data:
  prometheus-data:
  grafana-data:

networks:
  scrapy-network:
    driver: bridge

Production Docker Compose

# docker-compose.prod.yml
version: '3.8'

services:
  # Load balancer
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/ssl:/etc/nginx/ssl
    depends_on:
      - scraper-coordinator
    networks:
      - scrapy-network

  # Coordinator service
  scraper-coordinator:
    build:
      context: .
      target: production
    command: ["python", "scripts/coordinator.py"]
    environment:
      - SCRAPY_SETTINGS_MODULE=webscraper.settings.production
      - REDIS_URL=redis://redis-cluster:6379/0
      - SENTRY_DSN=${SENTRY_DSN}
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
    depends_on:
      - redis-cluster
    networks:
      - scrapy-network

  # Worker services
  scraper-worker:
    build:
      context: .
      target: production
    command: ["python", "scripts/worker.py"]
    environment:
      - SCRAPY_SETTINGS_MODULE=webscraper.settings.production
      - REDIS_URL=redis://redis-cluster:6379/0
      - WORKER_ID=${HOSTNAME}
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M
    depends_on:
      - redis-cluster
      - scraper-coordinator
    networks:
      - scrapy-network

  # Redis cluster
  redis-cluster:
    image: redis:7-alpine
    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
    volumes:
      - redis-prod-data:/data
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    networks:
      - scrapy-network

  # Monitoring stack
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./monitoring/prometheus.prod.yml:/etc/prometheus/prometheus.yml
      - prometheus-prod-data:/prometheus
    deploy:
      resources:
        limits:
          cpus: '0.3'
          memory: 256M
    networks:
      - scrapy-network

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana-prod-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    deploy:
      resources:
        limits:
          cpus: '0.3'
          memory: 256M
    networks:
      - scrapy-network

volumes:
  redis-prod-data:
  prometheus-prod-data:
  grafana-prod-data:

networks:
  scrapy-network:
    driver: overlay
    attachable: true

Kubernetes Deployment

Kubernetes Manifests

# k8s/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: scrapy-production
  labels:
    name: scrapy-production

---
# k8s/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: scrapy-config
  namespace: scrapy-production
data:
  SCRAPY_SETTINGS_MODULE: "webscraper.settings.production"
  REDIS_URL: "redis://redis-service:6379/0"
  LOG_LEVEL: "INFO"
  CONCURRENT_REQUESTS: "16"
  DOWNLOAD_DELAY: "1"

---
# k8s/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: scrapy-secrets
  namespace: scrapy-production
type: Opaque
data:
  # Base64 encoded values
  POSTGRES_PASSWORD: c2NyYXB5X3Bhc3N3b3Jk  # scrapy_password
  MONGO_PASSWORD: bW9uZ29fcGFzc3dvcmQ=      # mongo_password
  SENTRY_DSN: aHR0cHM6Ly9zZW50cnkuaW8=    # https://sentry.io

---
# k8s/redis-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: scrapy-production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        ports:
        - containerPort: 6379
        command: ["redis-server"]
        args: ["--appendonly", "yes", "--maxmemory", "256mb"]
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        volumeMounts:
        - name: redis-storage
          mountPath: /data
      volumes:
      - name: redis-storage
        persistentVolumeClaim:
          claimName: redis-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  namespace: scrapy-production
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379
  type: ClusterIP

---
# k8s/coordinator-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scraper-coordinator
  namespace: scrapy-production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: scraper-coordinator
  template:
    metadata:
      labels:
        app: scraper-coordinator
    spec:
      containers:
      - name: coordinator
        image: your-registry/webscraper:latest
        command: ["python", "scripts/coordinator.py"]
        envFrom:
        - configMapRef:
            name: scrapy-config
        - secretRef:
            name: scrapy-secrets
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

---
# k8s/worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scraper-worker
  namespace: scrapy-production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: scraper-worker
  template:
    metadata:
      labels:
        app: scraper-worker
    spec:
      containers:
      - name: worker
        image: your-registry/webscraper:latest
        command: ["python", "scripts/worker.py"]
        envFrom:
        - configMapRef:
            name: scrapy-config
        - secretRef:
            name: scrapy-secrets
        env:
        - name: WORKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
          requests:
            memory: "512Mi"
            cpu: "300m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        ports:
        - containerPort: 8081
        livenessProbe:
          httpGet:
            path: /health
            port: 8081
          initialDelaySeconds: 60
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 8081
          initialDelaySeconds: 10
          periodSeconds: 10

---
# k8s/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: scraper-worker-hpa
  namespace: scrapy-production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: scraper-worker
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60

---
# k8s/cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: scraper-job
  namespace: scrapy-production
spec:
  schedule: "0 */6 * * *"  # Every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: scraper
            image: your-registry/webscraper:latest
            command: ["scrapy", "crawl", "ecommerce"]
            envFrom:
            - configMapRef:
                name: scrapy-config
            - secretRef:
                name: scrapy-secrets
            resources:
              requests:
                memory: "1Gi"
                cpu: "500m"
              limits:
                memory: "2Gi"
                cpu: "1000m"
          restartPolicy: OnFailure
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

---
# k8s/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc
  namespace: scrapy-production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

CI/CD Pipeline

GitHub Actions Workflow

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # Test and quality checks
  test:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Cache dependencies
      uses: actions/cache@v3
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install -r requirements-dev.txt

    - name: Code formatting check
      run: |
        black --check --diff .

    - name: Linting
      run: |
        flake8 webscraper tests

    - name: Type checking
      run: |
        mypy webscraper

    - name: Security scan
      run: |
        bandit -r webscraper

    - name: Run tests
      run: |
        pytest tests/ --cov=webscraper --cov-report=xml --cov-report=html
      env:
        REDIS_URL: redis://localhost:6379/0

    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        file: ./coverage.xml

  # Build and push Docker image
  build:
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
    - uses: actions/checkout@v4

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Login to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=ref,event=branch
          type=ref,event=pr
          type=sha,prefix={{branch}}-
          type=raw,value=latest,enable={{is_default_branch}}

    - name: Build and push
      uses: docker/build-push-action@v5
      with:
        context: .
        target: production
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max

  # Deploy to staging
  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/develop'
    environment: staging

    steps:
    - uses: actions/checkout@v4

    - name: Configure kubectl
      uses: azure/k8s-set-context@v3
      with:
        method: kubeconfig
        kubeconfig: ${{ secrets.KUBE_CONFIG_STAGING }}

    - name: Deploy to staging
      run: |
        # Update image in k8s manifests
        sed -i "s|your-registry/webscraper:latest|${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:develop|g" k8s/*.yaml
        
        # Apply manifests
        kubectl apply -f k8s/ -n scrapy-staging
        
        # Wait for rollout
        kubectl rollout status deployment/scraper-coordinator -n scrapy-staging
        kubectl rollout status deployment/scraper-worker -n scrapy-staging

    - name: Run smoke tests
      run: |
        # Wait for services to be ready
        kubectl wait --for=condition=ready pod -l app=scraper-coordinator -n scrapy-staging --timeout=300s
        
        # Run basic smoke tests
        python scripts/smoke_tests.py --environment=staging

  # Deploy to production
  deploy-production:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    environment: production

    steps:
    - uses: actions/checkout@v4

    - name: Configure kubectl
      uses: azure/k8s-set-context@v3
      with:
        method: kubeconfig
        kubeconfig: ${{ secrets.KUBE_CONFIG_PRODUCTION }}

    - name: Deploy to production
      run: |
        # Update image in k8s manifests
        sed -i "s|your-registry/webscraper:latest|${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest|g" k8s/*.yaml
        
        # Apply manifests with rolling update
        kubectl apply -f k8s/ -n scrapy-production
        
        # Wait for rollout
        kubectl rollout status deployment/scraper-coordinator -n scrapy-production --timeout=600s
        kubectl rollout status deployment/scraper-worker -n scrapy-production --timeout=600s

    - name: Verify deployment
      run: |
        # Check pod health
        kubectl get pods -n scrapy-production
        
        # Run production health checks
        python scripts/health_check.py --environment=production

    - name: Notify deployment
      uses: 8398a7/action-slack@v3
      with:
        status: ${{ job.status }}
        channel: '#deployments'
        webhook_url: ${{ secrets.SLACK_WEBHOOK }}
      if: always()

Testing Framework

# tests/test_spiders.py
import pytest
import responses
from scrapy.http import HtmlResponse, Request
from webscraper.spiders.ecommerce_spider import EcommerceSpider
from webscraper.items import ProductItem

class TestEcommerceSpider:
    
    @pytest.fixture
    def spider(self):
        return EcommerceSpider()
    
    @pytest.fixture
    def sample_product_html(self):
        return """
        <html>
        <body>
            <h1 class="product-title">Test Product</h1>
            <span class="price-current">$99.99</span>
            <div class="product-description">
                <p>This is a test product description</p>
            </div>
            <span class="brand-name">TestBrand</span>
            <span class="rating-value">4.5</span>
            <div class="stock-status">In Stock</div>
        </body>
        </html>
        """
    
    def create_response(self, html, url="http://test.com"):
        request = Request(url=url)
        return HtmlResponse(url=url, request=request, body=html.encode('utf-8'))
    
    def test_parse_product_basic(self, spider, sample_product_html):
        """Test basic product parsing"""
        response = self.create_response(sample_product_html)
        
        items = list(spider.parse_product(response))
        
        assert len(items) == 1
        item = items[0]
        
        assert item['name'] == 'Test Product'
        assert item['price'] == 99.99
        assert item['brand'] == 'TestBrand'
        assert item['rating'] == 4.5
    
    def test_parse_product_missing_fields(self, spider):
        """Test handling of missing fields"""
        html = "<html><body><h1>Product</h1></body></html>"
        response = self.create_response(html)
        
        items = list(spider.parse_product(response))
        
        assert len(items) == 1
        item = items[0]
        assert item['name'] == 'Product'
        assert 'price' not in item or item['price'] is None
    
    @responses.activate
    def test_api_integration(self, spider):
        """Test API integration"""
        # Mock API response
        responses.add(
            responses.GET,
            'http://api.test.com/products',
            json={'products': [{'id': 1, 'name': 'API Product'}]},
            status=200
        )
        
        # Test API call logic
        import requests
        response = requests.get('http://api.test.com/products')
        assert response.status_code == 200
        assert response.json()['products'][0]['name'] == 'API Product'

# tests/test_pipelines.py
import pytest
from itemadapter import ItemAdapter
from webscraper.pipelines import AdvancedValidationPipeline, ValidationError
from webscraper.items import ProductItem

class TestValidationPipeline:
    
    @pytest.fixture
    def pipeline(self):
        return AdvancedValidationPipeline()
    
    @pytest.fixture
    def valid_item(self):
        return ProductItem({
            'name': 'Test Product',
            'url': 'https://test.com/product',
            'price': 99.99,
            'rating': 4.5,
            'in_stock': True
        })
    
    def test_valid_item_passes(self, pipeline, valid_item):
        """Test that valid item passes validation"""
        result = pipeline.process_item(valid_item, None)
        assert result is not None
        assert result['validation_passed'] is True
    
    def test_missing_required_field_fails(self, pipeline):
        """Test that missing required fields cause validation failure"""
        item = ProductItem({'price': 99.99})
        
        with pytest.raises(ValidationError):
            pipeline.process_item(item, None)
    
    def test_invalid_price_type_fails(self, pipeline):
        """Test that invalid price type causes validation failure"""
        item = ProductItem({
            'name': 'Test Product',
            'url': 'https://test.com/product',
            'price': 'invalid'
        })
        
        with pytest.raises(ValidationError):
            pipeline.process_item(item, None)
    
    def test_price_cleaning(self, pipeline):
        """Test price cleaning functionality"""
        price_tests = [
            ('$99.99', 99.99),
            ('€1,234.56', 1234.56),
            ('1.234,56', 1234.56),
            ('FREE', None),
            ('', None)
        ]
        
        for input_price, expected in price_tests:
            cleaned = pipeline._clean_price(input_price)
            assert cleaned == expected

# scripts/smoke_tests.py
import requests
import time
import sys
import argparse
from typing import Dict, List

class SmokeTests:
    """Basic smoke tests for deployed environment"""
    
    def __init__(self, environment: str):
        self.environment = environment
        self.base_urls = {
            'staging': 'https://staging-scraper.yourcompany.com',
            'production': 'https://scraper.yourcompany.com'
        }
        self.base_url = self.base_urls[environment]
    
    def test_health_endpoints(self) -> bool:
        """Test health endpoints"""
        endpoints = ['/health', '/ready', '/metrics']
        
        for endpoint in endpoints:
            try:
                response = requests.get(f"{self.base_url}{endpoint}", timeout=10)
                if response.status_code != 200:
                    print(f"❌ Health check failed for {endpoint}: {response.status_code}")
                    return False
                print(f"✅ Health check passed for {endpoint}")
            except Exception as e:
                print(f"❌ Health check failed for {endpoint}: {e}")
                return False
        
        return True
    
    def test_basic_functionality(self) -> bool:
        """Test basic scraping functionality"""
        try:
            # Trigger a test spider run
            response = requests.post(
                f"{self.base_url}/api/spiders/test/start",
                json={'test_mode': True},
                timeout=30
            )
            
            if response.status_code != 200:
                print(f"❌ Failed to start test spider: {response.status_code}")
                return False
            
            job_id = response.json().get('job_id')
            
            # Check job status
            for _ in range(10):  # Wait up to 50 seconds
                status_response = requests.get(
                    f"{self.base_url}/api/jobs/{job_id}/status",
                    timeout=10
                )
                
                if status_response.status_code == 200:
                    status = status_response.json().get('status')
                    if status == 'completed':
                        print("✅ Basic functionality test passed")
                        return True
                    elif status == 'failed':
                        print("❌ Basic functionality test failed")
                        return False
                
                time.sleep(5)
            
            print("❌ Basic functionality test timed out")
            return False
            
        except Exception as e:
            print(f"❌ Basic functionality test failed: {e}")
            return False
    
    def run_all_tests(self) -> bool:
        """Run all smoke tests"""
        print(f"🚀 Running smoke tests for {self.environment} environment")
        
        tests = [
            self.test_health_endpoints,
            self.test_basic_functionality
        ]
        
        results = []
        for test in tests:
            results.append(test())
        
        success = all(results)
        
        if success:
            print("🎉 All smoke tests passed!")
        else:
            print("💥 Some smoke tests failed!")
        
        return success

def main():
    parser = argparse.ArgumentParser(description='Run smoke tests')
    parser.add_argument('--environment', required=True, 
                       choices=['staging', 'production'],
                       help='Target environment')
    args = parser.parse_args()
    
    tests = SmokeTests(args.environment)
    success = tests.run_all_tests()
    
    sys.exit(0 if success else 1)

if __name__ == "__main__":
    main()

Production Monitoring and Observability

Prometheus Metrics

# webscraper/monitoring/metrics.py
from prometheus_client import Counter, Histogram, Gauge, CollectorRegistry, push_to_gateway
import time
from functools import wraps
from typing import Callable
import logging

class ScrapingMetrics:
    """Prometheus metrics for scraping operations"""
    
    def __init__(self, pushgateway_url: str = None):
        self.registry = CollectorRegistry()
        self.pushgateway_url = pushgateway_url
        
        # Counters
        self.requests_total = Counter(
            'scrapy_requests_total',
            'Total number of requests made',
            ['spider', 'status'],
            registry=self.registry
        )
        
        self.items_scraped_total = Counter(
            'scrapy_items_scraped_total',
            'Total number of items scraped',
            ['spider', 'item_type'],
            registry=self.registry
        )
        
        self.errors_total = Counter(
            'scrapy_errors_total',
            'Total number of errors',
            ['spider', 'error_type'],
            registry=self.registry
        )
        
        # Histograms
        self.response_time = Histogram(
            'scrapy_response_time_seconds',
            'Response time for requests',
            ['spider', 'domain'],
            registry=self.registry
        )
        
        self.item_processing_time = Histogram(
            'scrapy_item_processing_time_seconds',
            'Time to process items',
            ['spider', 'pipeline'],
            registry=self.registry
        )
        
        # Gauges
        self.active_requests = Gauge(
            'scrapy_active_requests',
            'Number of active requests',
            ['spider'],
            registry=self.registry
        )
        
        self.queue_size = Gauge(
            'scrapy_queue_size',
            'Size of request queue',
            ['spider'],
            registry=self.registry
        )
        
        self.memory_usage = Gauge(
            'scrapy_memory_usage_bytes',
            'Memory usage in bytes',
            ['spider'],
            registry=self.registry
        )
    
    def record_request(self, spider: str, status: str, response_time: float = None, domain: str = None):
        """Record request metrics"""
        self.requests_total.labels(spider=spider, status=status).inc()
        
        if response_time and domain:
            self.response_time.labels(spider=spider, domain=domain).observe(response_time)
    
    def record_item(self, spider: str, item_type: str):
        """Record scraped item"""
        self.items_scraped_total.labels(spider=spider, item_type=item_type).inc()
    
    def record_error(self, spider: str, error_type: str):
        """Record error"""
        self.errors_total.labels(spider=spider, error_type=error_type).inc()
    
    def update_queue_size(self, spider: str, size: int):
        """Update queue size"""
        self.queue_size.labels(spider=spider).set(size)
    
    def update_active_requests(self, spider: str, count: int):
        """Update active requests count"""
        self.active_requests.labels(spider=spider).set(count)
    
    def update_memory_usage(self, spider: str, bytes_used: int):
        """Update memory usage"""
        self.memory_usage.labels(spider=spider).set(bytes_used)
    
    def push_metrics(self, job_name: str):
        """Push metrics to Pushgateway"""
        if self.pushgateway_url:
            try:
                push_to_gateway(
                    self.pushgateway_url, 
                    job=job_name, 
                    registry=self.registry
                )
            except Exception as e:
                logging.error(f"Failed to push metrics: {e}")

def monitor_performance(metrics: ScrapingMetrics, spider_name: str):
    """Decorator to monitor function performance"""
    def decorator(func: Callable) -> Callable:
        @wraps(func)
        def wrapper(*args, **kwargs):
            start_time = time.time()
            try:
                result = func(*args, **kwargs)
                return result
            except Exception as e:
                metrics.record_error(spider_name, type(e).__name__)
                raise
            finally:
                duration = time.time() - start_time
                metrics.item_processing_time.labels(
                    spider=spider_name, 
                    pipeline=func.__name__
                ).observe(duration)
        return wrapper
    return decorator

# Integration with Scrapy
class PrometheusStatsCollector:
    """Collect Scrapy stats and export to Prometheus"""
    
    def __init__(self, crawler):
        self.crawler = crawler
        self.metrics = ScrapingMetrics(
            pushgateway_url=crawler.settings.get('PROMETHEUS_PUSHGATEWAY_URL')
        )
        self.spider_name = None
    
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)
    
    def spider_opened(self, spider):
        self.spider_name = spider.name
        spider.logger.info(f"Prometheus metrics enabled for spider: {spider.name}")
    
    def spider_closed(self, spider, reason):
        # Push final metrics
        if self.metrics.pushgateway_url:
            self.metrics.push_metrics(f"scrapy_{spider.name}")
        
        # Log final stats
        stats = self.crawler.stats.get_stats()
        spider.logger.info(f"Final stats: {stats}")
    
    def request_scheduled(self, request, spider):
        self.metrics.update_active_requests(
            spider.name, 
            self.crawler.stats.get_value('scheduler/enqueued', 0)
        )
    
    def response_received(self, response, request, spider):
        # Record response metrics
        status = str(response.status)
        domain = response.url.split('/')[2] if '://' in response.url else 'unknown'
        
        self.metrics.record_request(
            spider=spider.name,
            status=status,
            domain=domain
        )
    
    def item_scraped(self, item, response, spider):
        # Record item metrics
        item_type = type(item).__name__
        self.metrics.record_item(spider.name, item_type)

Grafana Dashboard Configuration

{
  "dashboard": {
    "id": null,
    "title": "Scrapy Monitoring Dashboard",
    "description": "Comprehensive monitoring for Scrapy spiders",
    "tags": ["scrapy", "monitoring"],
    "timezone": "browser",
    "panels": [
      {
        "id": 1,
        "title": "Requests per Second",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(scrapy_requests_total[5m])",
            "legendFormat": "{{spider}} - {{status}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
        "yAxes": [
          {"label": "Requests/sec", "min": 0}
        ]
      },
      {
        "id": 2,
        "title": "Items Scraped per Hour",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(scrapy_items_scraped_total[1h])*3600",
            "legendFormat": "{{spider}} - {{item_type}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
      },
      {
        "id": 3,
        "title": "Error Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(scrapy_errors_total[5m]) / rate(scrapy_requests_total[5m]) * 100",
            "legendFormat": "{{spider}} error rate %"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
        "alert": {
          "conditions": [
            {
              "query": {"queryType": "", "refId": "A"},
              "reducer": {"type": "last", "params": []},
              "evaluator": {"params": [5], "type": "gt"}
            }
          ],
          "executionErrorState": "alerting",
          "frequency": "10s",
          "handler": 1,
          "name": "High Error Rate",
          "noDataState": "no_data"
        }
      },
      {
        "id": 4,
        "title": "Response Time Distribution",
        "type": "heatmap",
        "targets": [
          {
            "expr": "scrapy_response_time_seconds_bucket",
            "legendFormat": "{{le}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
      },
      {
        "id": 5,
        "title": "Memory Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "scrapy_memory_usage_bytes / 1024 / 1024",
            "legendFormat": "{{spider}} Memory (MB)"
          }
        ],
        "gridPos": {"h": 8, "w": 24, "x": 0, "y": 16}
      },
      {
        "id": 6,
        "title": "Queue Size",
        "type": "graph",
        "targets": [
          {
            "expr": "scrapy_queue_size",
            "legendFormat": "{{spider}} Queue"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 24}
      },
      {
        "id": 7,
        "title": "Active Requests",
        "type": "stat",
        "targets": [
          {
            "expr": "sum(scrapy_active_requests)",
            "legendFormat": "Total Active"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 24}
      }
    ],
    "time": {"from": "now-1h", "to": "now"},
    "refresh": "5s"
  }
}

Security and Compliance

Security Hardening

# webscraper/security/security_middleware.py
import hmac
import hashlib
import time
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
from scrapy.exceptions import IgnoreRequest
import logging

class SecurityMiddleware:
    """Security middleware for production environments"""
    
    def __init__(self, settings):
        self.settings = settings
        self.logger = logging.getLogger(__name__)
        
        # Rate limiting
        self.rate_limits = {}
        self.last_request_time = {}
        
        # Security headers
        self.required_headers = {
            'User-Agent': True,
            'Accept': True,
            'Accept-Language': True
        }
        
        # Blocked patterns
        self.blocked_patterns = [
            r'.*\.exe$',
            r'.*\.zip$',
            r'.*admin.*',
            r'.*login.*',
            r'.*private.*'
        ]
    
    def process_request(self, request, spider):
        # Validate request security
        if not self._validate_request_security(request):
            raise IgnoreRequest("Request blocked by security policy")
        
        # Apply rate limiting
        self._apply_rate_limiting(request, spider)
        
        # Add security headers
        self._add_security_headers(request)
        
        return None
    
    def _validate_request_security(self, request):
        """Validate request against security policies"""
        url = request.url.lower()
        
        # Check blocked patterns
        import re
        for pattern in self.blocked_patterns:
            if re.match(pattern, url):
                self.logger.warning(f"Blocked request to: {request.url}")
                return False
        
        # Check for required headers
        for header, required in self.required_headers.items():
            if required and header not in request.headers:
                self.logger.warning(f"Missing required header: {header}")
                return False
        
        return True
    
    def _apply_rate_limiting(self, request, spider):
        """Apply rate limiting per domain"""
        from urllib.parse import urlparse
        domain = urlparse(request.url).netloc
        
        current_time = time.time()
        min_delay = self.settings.getfloat('SECURITY_MIN_DELAY', 1.0)
        
        if domain in self.last_request_time:
            time_since_last = current_time - self.last_request_time[domain]
            if time_since_last < min_delay:
                sleep_time = min_delay - time_since_last
                spider.logger.debug(f"Security rate limiting: {sleep_time:.2f}s for {domain}")
                time.sleep(sleep_time)
        
        self.last_request_time[domain] = time.time()
    
    def _add_security_headers(self, request):
        """Add security headers to requests"""
        # Add timestamp for request validation
        timestamp = str(int(time.time()))
        request.headers['X-Request-Timestamp'] = timestamp
        
        # Add security token if configured
        secret_key = self.settings.get('SECURITY_SECRET_KEY')
        if secret_key:
            signature = hmac.new(
                secret_key.encode(),
                f"{request.url}{timestamp}".encode(),
                hashlib.sha256
            ).hexdigest()
            request.headers['X-Security-Signature'] = signature

class DataEncryptionPipeline:
    """Encrypt sensitive data before storage"""
    
    def __init__(self, encryption_key):
        self.encryption_key = encryption_key
        self.sensitive_fields = ['email', 'phone', 'address', 'personal_id']
    
    @classmethod
    def from_crawler(cls, crawler):
        encryption_key = crawler.settings.get('ENCRYPTION_KEY')
        if not encryption_key:
            raise ValueError("ENCRYPTION_KEY setting is required")
        return cls(encryption_key)
    
    def process_item(self, item, spider):
        from cryptography.fernet import Fernet
        
        fernet = Fernet(self.encryption_key.encode())
        
        for field in self.sensitive_fields:
            if field in item and item[field]:
                # Encrypt sensitive data
                encrypted_data = fernet.encrypt(str(item[field]).encode())
                item[f"{field}_encrypted"] = encrypted_data.decode()
                # Remove original field
                del item[field]
        
        return item

# scripts/security_audit.py
import subprocess
import json
import sys
from typing import Dict, List

class SecurityAudit:
    """Security audit for production deployment"""
    
    def __init__(self):
        self.issues = []
    
    def audit_dependencies(self) -> bool:
        """Audit Python dependencies for known vulnerabilities"""
        try:
            result = subprocess.run(
                ['safety', 'check', '--json'],
                capture_output=True,
                text=True
            )
            
            if result.returncode != 0:
                vulnerabilities = json.loads(result.stdout)
                for vuln in vulnerabilities:
                    self.issues.append({
                        'type': 'dependency_vulnerability',
                        'severity': 'high',
                        'package': vuln['package_name'],
                        'vulnerability': vuln['vulnerability_id'],
                        'description': vuln['advisory']
                    })
                return False
            
            return True
            
        except Exception as e:
            self.issues.append({
                'type': 'audit_error',
                'severity': 'medium',
                'description': f"Failed to audit dependencies: {e}"
            })
            return False
    
    def audit_docker_image(self, image_name: str) -> bool:
        """Audit Docker image for security issues"""
        try:
            result = subprocess.run(
                ['trivy', 'image', '--format', 'json', image_name],
                capture_output=True,
                text=True
            )
            
            if result.returncode == 0:
                scan_results = json.loads(result.stdout)
                
                for result in scan_results.get('Results', []):
                    for vuln in result.get('Vulnerabilities', []):
                        if vuln.get('Severity') in ['HIGH', 'CRITICAL']:
                            self.issues.append({
                                'type': 'container_vulnerability',
                                'severity': vuln['Severity'].lower(),
                                'package': vuln.get('PkgName'),
                                'vulnerability': vuln.get('VulnerabilityID'),
                                'description': vuln.get('Description', '')
                            })
            
            return len([i for i in self.issues if i['type'] == 'container_vulnerability']) == 0
            
        except Exception as e:
            self.issues.append({
                'type': 'audit_error',
                'severity': 'medium',
                'description': f"Failed to audit Docker image: {e}"
            })
            return False
    
    def audit_kubernetes_config(self, config_path: str) -> bool:
        """Audit Kubernetes configuration"""
        security_checks = [
            self._check_non_root_user,
            self._check_resource_limits,
            self._check_security_context,
            self._check_network_policies
        ]
        
        passed = True
        for check in security_checks:
            if not check(config_path):
                passed = False
        
        return passed
    
    def _check_non_root_user(self, config_path: str) -> bool:
        """Check if containers run as non-root"""
        # Implementation for checking non-root user
        return True
    
    def _check_resource_limits(self, config_path: str) -> bool:
        """Check if resource limits are set"""
        # Implementation for checking resource limits
        return True
    
    def _check_security_context(self, config_path: str) -> bool:
        """Check security context configuration"""
        # Implementation for checking security context
        return True
    
    def _check_network_policies(self, config_path: str) -> bool:
        """Check network policies"""
        # Implementation for checking network policies
        return True
    
    def generate_report(self) -> Dict:
        """Generate security audit report"""
        severity_counts = {}
        for issue in self.issues:
            severity = issue['severity']
            severity_counts[severity] = severity_counts.get(severity, 0) + 1
        
        return {
            'total_issues': len(self.issues),
            'severity_breakdown': severity_counts,
            'issues': self.issues,
            'passed': len(self.issues) == 0
        }
    
    def run_full_audit(self, image_name: str = None, config_path: str = None) -> bool:
        """Run complete security audit"""
        print("🔒 Running security audit...")
        
        # Audit dependencies
        deps_ok = self.audit_dependencies()
        print(f"Dependencies: {'✅' if deps_ok else '❌'}")
        
        # Audit Docker image if provided
        if image_name:
            image_ok = self.audit_docker_image(image_name)
            print(f"Docker image: {'✅' if image_ok else '❌'}")
        
        # Audit Kubernetes config if provided
        if config_path:
            k8s_ok = self.audit_kubernetes_config(config_path)
            print(f"Kubernetes config: {'✅' if k8s_ok else '❌'}")
        
        # Generate report
        report = self.generate_report()
        
        if report['passed']:
            print("🎉 Security audit passed!")
        else:
            print(f"💥 Security audit failed with {report['total_issues']} issues")
            for issue in self.issues:
                print(f"  - {issue['severity'].upper()}: {issue['description']}")
        
        return report['passed']

if __name__ == "__main__":
    import argparse
    
    parser = argparse.ArgumentParser(description='Run security audit')
    parser.add_argument('--image', help='Docker image to audit')
    parser.add_argument('--config', help='Kubernetes config path to audit')
    args = parser.parse_args()
    
    audit = SecurityAudit()
    success = audit.run_full_audit(args.image, args.config)
    
    sys.exit(0 if success else 1)

Wrapping Up

That is the end of the series: setup and first spiders in Part 1, JavaScript rendering and forms in Part 2, anti-detection and distributed scaling in Part 3, data pipelines and storage in Part 4, and deployment here. When something breaks in production, and it will, the official Scrapy documentation is still the reference I reach for first.

Scrapy, part 4: cleaning and storing the data

Tue, 24 Dec 2024 00:00:00 GMT

Scrapy Part 4: Data Processing and Storage

Scraped data arrives messy: inconsistent price formats, broken URLs, duplicates, missing fields. This part builds the pipeline layer that fixes that. You will write a validation and cleaning pipeline, an enrichment pipeline for computed fields, and storage backends for MongoDB and PostgreSQL with deduplication, batch upserts, and analytics queries.

Data Cleaning and Validation

Validation Pipeline

# webscraper/pipelines.py
import re
import hashlib
import json
from datetime import datetime, timedelta
from typing import Dict, List, Optional, Any
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
import logging

class AdvancedValidationPipeline:
    """Comprehensive data validation and cleaning pipeline"""
    
    def __init__(self):
        self.logger = logging.getLogger(__name__)
        self.validation_stats = {
            'total_items': 0,
            'valid_items': 0,
            'dropped_items': 0,
            'field_fixes': 0,
            'validation_errors': {}
        }
        
        # Validation rules configuration
        self.validation_rules = {
            'required_fields': ['name', 'url'],
            'field_types': {
                'price': (int, float),
                'rating': (int, float),
                'review_count': int,
                'in_stock': bool
            },
            'field_ranges': {
                'price': (0, 1000000),
                'rating': (0, 5),
                'review_count': (0, float('inf'))
            },
            'string_patterns': {
                'email': r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$',
                'url': r'^https?:\/\/[^\s/$.?#].[^\s]*$',
                'phone': r'^[\+]?[1-9][\d]{0,15}$'
            },
            'max_lengths': {
                'name': 200,
                'description': 5000,
                'brand': 100
            }
        }
    
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        self.validation_stats['total_items'] += 1
        
        try:
            # Clean and validate all fields
            self._clean_item_fields(adapter)
            self._validate_required_fields(adapter)
            self._validate_field_types(adapter)
            self._validate_field_ranges(adapter)
            self._validate_string_patterns(adapter)
            self._validate_field_lengths(adapter)
            
            # Additional business logic validation
            self._validate_business_rules(adapter)
            
            # Generate quality score
            adapter['data_quality_score'] = self._calculate_quality_score(adapter)
            
            # Add validation metadata
            adapter['validation_timestamp'] = datetime.now().isoformat()
            adapter['validation_passed'] = True
            
            self.validation_stats['valid_items'] += 1
            return item
            
        except ValidationError as e:
            self.validation_stats['dropped_items'] += 1
            error_type = type(e).__name__
            self.validation_stats['validation_errors'][error_type] = \
                self.validation_stats['validation_errors'].get(error_type, 0) + 1
            
            self.logger.warning(f"Validation failed for item: {e}")
            raise DropItem(f"Validation error: {e}")
    
    def _clean_item_fields(self, adapter):
        """Clean and normalize field values"""
        # Clean text fields
        text_fields = ['name', 'description', 'brand', 'category']
        for field in text_fields:
            if adapter.get(field):
                cleaned = self._clean_text(adapter[field])
                if cleaned != adapter[field]:
                    adapter[field] = cleaned
                    self.validation_stats['field_fixes'] += 1
        
        # Clean price field
        if adapter.get('price'):
            cleaned_price = self._clean_price(adapter['price'])
            if cleaned_price != adapter['price']:
                adapter['price'] = cleaned_price
                self.validation_stats['field_fixes'] += 1
        
        # Clean URL fields
        url_fields = ['url', 'image_url']
        for field in url_fields:
            if adapter.get(field):
                cleaned_url = self._clean_url(adapter[field])
                if cleaned_url != adapter[field]:
                    adapter[field] = cleaned_url
                    self.validation_stats['field_fixes'] += 1
        
        # Normalize boolean fields
        bool_fields = ['in_stock', 'featured', 'on_sale']
        for field in bool_fields:
            if field in adapter:
                adapter[field] = self._normalize_boolean(adapter[field])
    
    def _clean_text(self, text: str) -> str:
        """Clean and normalize text content"""
        if not isinstance(text, str):
            text = str(text)
        
        # Remove excessive whitespace
        text = re.sub(r'\s+', ' ', text.strip())
        
        # Remove HTML tags
        text = re.sub(r'<[^>]+>', '', text)
        
        # Remove special characters that might cause issues
        text = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\xff]', '', text)
        
        # Normalize quotes
        text = text.replace('"', '"').replace('"', '"')
        text = text.replace(''', "'").replace(''', "'")
        
        return text
    
    def _clean_price(self, price: Any) -> Optional[float]:
        """Clean and convert price to float"""
        if price is None:
            return None
        
        if isinstance(price, (int, float)):
            return float(price)
        
        if isinstance(price, str):
            # Remove currency symbols and whitespace
            price_clean = re.sub(r'[^\d.,]', '', price)
            
            # Handle different decimal separators
            if ',' in price_clean and '.' in price_clean:
                # Assume comma is thousands separator
                price_clean = price_clean.replace(',', '')
            elif ',' in price_clean:
                # Could be decimal separator or thousands
                if price_clean.count(',') == 1 and len(price_clean.split(',')[1]) <= 2:
                    price_clean = price_clean.replace(',', '.')
                else:
                    price_clean = price_clean.replace(',', '')
            
            try:
                return float(price_clean)
            except ValueError:
                return None
        
        return None
    
    def _clean_url(self, url: str) -> str:
        """Clean and normalize URLs"""
        if not isinstance(url, str):
            return str(url)
        
        url = url.strip()
        
        # Add protocol if missing
        if url.startswith('//'):
            url = 'https:' + url
        elif not url.startswith(('http://', 'https://')):
            url = 'https://' + url
        
        return url
    
    def _normalize_boolean(self, value: Any) -> bool:
        """Normalize various boolean representations"""
        if isinstance(value, bool):
            return value
        
        if isinstance(value, str):
            value = value.lower().strip()
            return value in ('true', 'yes', '1', 'on', 'available', 'in stock')
        
        if isinstance(value, (int, float)):
            return bool(value)
        
        return False
    
    def _validate_required_fields(self, adapter):
        """Validate that required fields are present"""
        for field in self.validation_rules['required_fields']:
            if not adapter.get(field):
                raise ValidationError(f"Required field missing: {field}")
    
    def _validate_field_types(self, adapter):
        """Validate field data types"""
        for field, expected_types in self.validation_rules['field_types'].items():
            if field in adapter and adapter[field] is not None:
                if not isinstance(adapter[field], expected_types):
                    raise ValidationError(
                        f"Field {field} has invalid type: "
                        f"expected {expected_types}, got {type(adapter[field])}"
                    )
    
    def _validate_field_ranges(self, adapter):
        """Validate numeric field ranges"""
        for field, (min_val, max_val) in self.validation_rules['field_ranges'].items():
            if field in adapter and adapter[field] is not None:
                value = adapter[field]
                if not (min_val <= value <= max_val):
                    raise ValidationError(
                        f"Field {field} value {value} outside valid range [{min_val}, {max_val}]"
                    )
    
    def _validate_string_patterns(self, adapter):
        """Validate string fields against regex patterns"""
        for field, pattern in self.validation_rules['string_patterns'].items():
            if field in adapter and adapter[field] is not None:
                if not re.match(pattern, str(adapter[field])):
                    raise ValidationError(f"Field {field} doesn't match required pattern")
    
    def _validate_field_lengths(self, adapter):
        """Validate string field lengths"""
        for field, max_length in self.validation_rules['max_lengths'].items():
            if field in adapter and adapter[field] is not None:
                if len(str(adapter[field])) > max_length:
                    raise ValidationError(
                        f"Field {field} exceeds maximum length of {max_length}"
                    )
    
    def _validate_business_rules(self, adapter):
        """Validate business-specific rules"""
        # Example business rules
        
        # Price must be positive for in-stock items
        if adapter.get('in_stock') and adapter.get('price', 0) <= 0:
            raise ValidationError("In-stock items must have positive price")
        
        # Rating must be present if review_count > 0
        if adapter.get('review_count', 0) > 0 and not adapter.get('rating'):
            raise ValidationError("Items with reviews must have rating")
        
        # Discount validation
        original_price = adapter.get('original_price')
        current_price = adapter.get('price')
        if original_price and current_price:
            if current_price > original_price:
                raise ValidationError("Current price cannot exceed original price")
    
    def _calculate_quality_score(self, adapter) -> float:
        """Calculate data quality score (0-1)"""
        score = 0.0
        max_score = 0.0
        
        # Completeness score (40% of total)
        important_fields = ['name', 'price', 'description', 'brand', 'category']
        present_fields = sum(1 for field in important_fields if adapter.get(field))
        completeness = present_fields / len(important_fields)
        score += completeness * 0.4
        max_score += 0.4
        
        # Accuracy score (30% of total)
        accuracy = 1.0  # Assume high accuracy if validation passed
        score += accuracy * 0.3
        max_score += 0.3
        
        # Consistency score (20% of total)
        consistency = self._check_consistency(adapter)
        score += consistency * 0.2
        max_score += 0.2
        
        # Freshness score (10% of total)
        freshness = 1.0  # Assume fresh data
        score += freshness * 0.1
        max_score += 0.1
        
        return score / max_score if max_score > 0 else 0.0
    
    def _check_consistency(self, adapter) -> float:
        """Check internal consistency of data"""
        consistency_score = 1.0
        
        # Check if price and in_stock are consistent
        if adapter.get('price', 0) <= 0 and adapter.get('in_stock', False):
            consistency_score -= 0.3
        
        # Check if rating and review_count are consistent
        rating = adapter.get('rating')
        review_count = adapter.get('review_count', 0)
        if rating and rating > 0 and review_count == 0:
            consistency_score -= 0.2
        
        return max(0.0, consistency_score)
    
    def close_spider(self, spider):
        """Log validation statistics when spider closes"""
        stats = self.validation_stats
        total = stats['total_items']
        
        self.logger.info("=== Validation Pipeline Statistics ===")
        self.logger.info(f"Total items processed: {total}")
        self.logger.info(f"Valid items: {stats['valid_items']} ({stats['valid_items']/total*100:.1f}%)")
        self.logger.info(f"Dropped items: {stats['dropped_items']} ({stats['dropped_items']/total*100:.1f}%)")
        self.logger.info(f"Field fixes applied: {stats['field_fixes']}")
        
        if stats['validation_errors']:
            self.logger.info("Validation errors by type:")
            for error_type, count in stats['validation_errors'].items():
                self.logger.info(f"  {error_type}: {count}")

class ValidationError(Exception):
    """Custom exception for validation errors"""
    pass

class DataEnrichmentPipeline:
    """Enrich items with additional computed fields"""
    
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        
        # Add computed fields
        self._add_price_analysis(adapter)
        self._add_text_analysis(adapter)
        self._add_category_classification(adapter)
        self._add_uniqueness_hash(adapter)
        
        return item
    
    def _add_price_analysis(self, adapter):
        """Add price-related computed fields"""
        current_price = adapter.get('price')
        original_price = adapter.get('original_price')
        
        if current_price and original_price:
            discount = original_price - current_price
            discount_percent = (discount / original_price) * 100
            
            adapter['discount_amount'] = discount
            adapter['discount_percentage'] = round(discount_percent, 2)
            adapter['is_on_sale'] = discount > 0
        
        # Price tier classification
        if current_price:
            if current_price < 50:
                adapter['price_tier'] = 'budget'
            elif current_price < 200:
                adapter['price_tier'] = 'mid-range'
            elif current_price < 500:
                adapter['price_tier'] = 'premium'
            else:
                adapter['price_tier'] = 'luxury'
    
    def _add_text_analysis(self, adapter):
        """Add text analysis metrics"""
        description = adapter.get('description', '')
        
        if description:
            words = description.split()
            adapter['description_word_count'] = len(words)
            adapter['description_char_count'] = len(description)
            
            # Simple sentiment analysis (in production, use proper NLP)
            positive_words = ['excellent', 'great', 'amazing', 'perfect', 'wonderful']
            negative_words = ['bad', 'terrible', 'awful', 'poor', 'disappointing']
            
            pos_count = sum(1 for word in words if word.lower() in positive_words)
            neg_count = sum(1 for word in words if word.lower() in negative_words)
            
            if pos_count > neg_count:
                adapter['sentiment'] = 'positive'
            elif neg_count > pos_count:
                adapter['sentiment'] = 'negative'
            else:
                adapter['sentiment'] = 'neutral'
    
    def _add_category_classification(self, adapter):
        """Add automatic category classification"""
        name = adapter.get('name', '').lower()
        description = adapter.get('description', '').lower()
        text = f"{name} {description}"
        
        # Simple keyword-based classification
        categories = {
            'electronics': ['phone', 'laptop', 'computer', 'tablet', 'tv', 'camera'],
            'clothing': ['shirt', 'pants', 'dress', 'shoes', 'jacket', 'jeans'],
            'home': ['furniture', 'kitchen', 'bedroom', 'bathroom', 'decor'],
            'books': ['book', 'novel', 'textbook', 'manual', 'guide'],
            'sports': ['fitness', 'exercise', 'sports', 'outdoor', 'gym']
        }
        
        scores = {}
        for category, keywords in categories.items():
            score = sum(1 for keyword in keywords if keyword in text)
            if score > 0:
                scores[category] = score
        
        if scores:
            adapter['auto_category'] = max(scores, key=scores.get)
            adapter['category_confidence'] = max(scores.values()) / len(categories[adapter['auto_category']])
    
    def _add_uniqueness_hash(self, adapter):
        """Add uniqueness hash for deduplication"""
        # Create hash from key identifying fields
        key_fields = ['name', 'brand', 'sku']
        hash_input = ''.join(str(adapter.get(field, '')) for field in key_fields)
        
        if hash_input.strip():
            adapter['uniqueness_hash'] = hashlib.md5(
                hash_input.lower().encode('utf-8')
            ).hexdigest()

Multiple Storage Backends

MongoDB Storage Pipeline

# webscraper/storage/mongodb_pipeline.py
import pymongo
from pymongo import MongoClient, UpdateOne
from itemadapter import ItemAdapter
from datetime import datetime
import logging
from typing import Dict, List

class MongoDBPipeline:
    """Advanced MongoDB storage pipeline with indexing and aggregation"""
    
    def __init__(self, mongo_uri, mongo_db, collection_name):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db
        self.collection_name = collection_name
        self.client = None
        self.db = None
        self.collection = None
        self.logger = logging.getLogger(__name__)
        
        # Batch processing
        self.batch_size = 100
        self.batch_items = []
        
        # Statistics
        self.stats = {
            'inserted': 0,
            'updated': 0,
            'duplicates': 0,
            'errors': 0
        }
    
    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get("MONGO_URI", "mongodb://localhost:27017"),
            mongo_db=crawler.settings.get("MONGO_DATABASE", "scrapy"),
            collection_name=crawler.settings.get("MONGO_COLLECTION", "items")
        )
    
    def open_spider(self, spider):
        """Initialize MongoDB connection and setup indexes"""
        self.client = MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]
        self.collection = self.db[self.collection_name]
        
        # Create indexes for performance
        self._create_indexes()
        
        self.logger.info(f"Connected to MongoDB: {self.mongo_uri}/{self.mongo_db}")
    
    def _create_indexes(self):
        """Create database indexes for optimal performance"""
        indexes = [
            # Unique index for deduplication
            ("uniqueness_hash", pymongo.ASCENDING),
            
            # Search indexes
            ("name", pymongo.TEXT),
            ("description", pymongo.TEXT),
            ("brand", pymongo.ASCENDING),
            ("category", pymongo.ASCENDING),
            
            # Filter indexes
            ("price", pymongo.ASCENDING),
            ("rating", pymongo.DESCENDING),
            ("in_stock", pymongo.ASCENDING),
            
            # Time-based indexes
            ("scraped_at", pymongo.DESCENDING),
            ("updated_at", pymongo.DESCENDING),
            
            # Compound indexes
            (("category", pymongo.ASCENDING), ("price", pymongo.ASCENDING)),
            (("brand", pymongo.ASCENDING), ("rating", pymongo.DESCENDING)),
        ]
        
        for index in indexes:
            try:
                if isinstance(index, tuple):
                    self.collection.create_index([index])
                else:
                    self.collection.create_index(index)
            except Exception as e:
                self.logger.warning(f"Failed to create index {index}: {e}")
        
        # Create text index for search
        try:
            self.collection.create_index([
                ("name", pymongo.TEXT),
                ("description", pymongo.TEXT),
                ("brand", pymongo.TEXT)
            ], name="text_search")
        except Exception as e:
            self.logger.warning(f"Failed to create text index: {e}")
    
    def process_item(self, item, spider):
        """Process item and add to batch"""
        adapter = ItemAdapter(item)
        
        # Add MongoDB-specific fields
        adapter['_scraped_at'] = datetime.utcnow()
        adapter['_spider_name'] = spider.name
        
        # Add to batch
        self.batch_items.append(dict(adapter))
        
        # Process batch if full
        if len(self.batch_items) >= self.batch_size:
            self._process_batch()
        
        return item
    
    def _process_batch(self):
        """Process batch of items with upsert operations"""
        if not self.batch_items:
            return
        
        try:
            operations = []
            
            for item in self.batch_items:
                uniqueness_hash = item.get('uniqueness_hash')
                
                if uniqueness_hash:
                    # Upsert based on uniqueness hash
                    operation = UpdateOne(
                        {'uniqueness_hash': uniqueness_hash},
                        {
                            '$set': item,
                            '$setOnInsert': {'_created_at': datetime.utcnow()},
                            '$currentDate': {'_updated_at': True}
                        },
                        upsert=True
                    )
                    operations.append(operation)
                else:
                    # Insert without duplicate check
                    self.collection.insert_one(item)
                    self.stats['inserted'] += 1
            
            if operations:
                result = self.collection.bulk_write(operations, ordered=False)
                self.stats['inserted'] += result.upserted_count
                self.stats['updated'] += result.modified_count
                self.stats['duplicates'] += len(operations) - result.upserted_count - result.modified_count
            
        except Exception as e:
            self.stats['errors'] += len(self.batch_items)
            self.logger.error(f"Error processing batch: {e}")
        
        finally:
            self.batch_items.clear()
    
    def close_spider(self, spider):
        """Process remaining items and close connection"""
        # Process remaining batch
        self._process_batch()
        
        # Log statistics
        self.logger.info("=== MongoDB Pipeline Statistics ===")
        for stat, count in self.stats.items():
            self.logger.info(f"{stat.capitalize()}: {count}")
        
        # Create aggregation pipelines for analytics
        self._create_analytics_views()
        
        # Close connection
        if self.client:
            self.client.close()
    
    def _create_analytics_views(self):
        """Create MongoDB views for analytics"""
        try:
            # Category statistics view
            self.db.create_collection("category_stats", viewOn=self.collection_name, pipeline=[
                {"$group": {
                    "_id": "$category",
                    "total_products": {"$sum": 1},
                    "avg_price": {"$avg": "$price"},
                    "avg_rating": {"$avg": "$rating"},
                    "in_stock_count": {"$sum": {"$cond": ["$in_stock", 1, 0]}}
                }},
                {"$sort": {"total_products": -1}}
            ])
            
            # Brand statistics view
            self.db.create_collection("brand_stats", viewOn=self.collection_name, pipeline=[
                {"$group": {
                    "_id": "$brand",
                    "total_products": {"$sum": 1},
                    "avg_price": {"$avg": "$price"},
                    "price_range": {
                        "min": {"$min": "$price"},
                        "max": {"$max": "$price"}
                    }
                }},
                {"$sort": {"total_products": -1}}
            ])
            
            # Daily scraping statistics
            self.db.create_collection("daily_stats", viewOn=self.collection_name, pipeline=[
                {"$group": {
                    "_id": {
                        "$dateToString": {
                            "format": "%Y-%m-%d",
                            "date": "$_scraped_at"
                        }
                    },
                    "items_scraped": {"$sum": 1},
                    "unique_brands": {"$addToSet": "$brand"},
                    "unique_categories": {"$addToSet": "$category"}
                }},
                {"$addFields": {
                    "unique_brand_count": {"$size": "$unique_brands"},
                    "unique_category_count": {"$size": "$unique_categories"}
                }},
                {"$sort": {"_id": -1}}
            ])
            
        except Exception as e:
            self.logger.warning(f"Failed to create analytics views: {e}")

class MongoDBAnalytics:
    """Advanced analytics queries for MongoDB"""
    
    def __init__(self, mongo_uri, mongo_db, collection_name):
        self.client = MongoClient(mongo_uri)
        self.db = self.client[mongo_db]
        self.collection = self.db[collection_name]
    
    def get_price_trends(self, days: int = 30) -> List[Dict]:
        """Get price trends over time"""
        pipeline = [
            {
                "$match": {
                    "_scraped_at": {
                        "$gte": datetime.utcnow() - timedelta(days=days)
                    }
                }
            },
            {
                "$group": {
                    "_id": {
                        "date": {"$dateToString": {"format": "%Y-%m-%d", "date": "$_scraped_at"}},
                        "category": "$category"
                    },
                    "avg_price": {"$avg": "$price"},
                    "min_price": {"$min": "$price"},
                    "max_price": {"$max": "$price"},
                    "product_count": {"$sum": 1}
                }
            },
            {"$sort": {"_id.date": 1}}
        ]
        
        return list(self.collection.aggregate(pipeline))
    
    def get_top_products(self, category: str = None, limit: int = 10) -> List[Dict]:
        """Get top-rated products"""
        match_stage = {"rating": {"$exists": True, "$gte": 4.0}}
        if category:
            match_stage["category"] = category
        
        pipeline = [
            {"$match": match_stage},
            {"$sort": {"rating": -1, "review_count": -1}},
            {"$limit": limit},
            {
                "$project": {
                    "name": 1,
                    "brand": 1,
                    "price": 1,
                    "rating": 1,
                    "review_count": 1,
                    "url": 1
                }
            }
        ]
        
        return list(self.collection.aggregate(pipeline))
    
    def get_inventory_alerts(self) -> List[Dict]:
        """Get products that went out of stock recently"""
        pipeline = [
            {
                "$match": {
                    "in_stock": False,
                    "_scraped_at": {"$gte": datetime.utcnow() - timedelta(days=1)}
                }
            },
            {
                "$project": {
                    "name": 1,
                    "brand": 1,
                    "price": 1,
                    "category": 1,
                    "_scraped_at": 1
                }
            },
            {"$sort": {"_scraped_at": -1}}
        ]
        
        return list(self.collection.aggregate(pipeline))

PostgreSQL Storage Pipeline

# webscraper/storage/postgresql_pipeline.py
import psycopg2
import psycopg2.extras
from itemadapter import ItemAdapter
from datetime import datetime
import logging
import json
from typing import Dict, List, Optional

class PostgreSQLPipeline:
    """Advanced PostgreSQL storage with JSONB and full-text search"""
    
    def __init__(self, postgres_settings):
        self.postgres_settings = postgres_settings
        self.connection = None
        self.cursor = None
        self.logger = logging.getLogger(__name__)
        
        # Batch processing
        self.batch_size = 50
        self.batch_items = []
        
        # Statistics
        self.stats = {
            'inserted': 0,
            'updated': 0,
            'errors': 0
        }
    
    @classmethod
    def from_crawler(cls, crawler):
        postgres_settings = crawler.settings.getdict("POSTGRES_SETTINGS")
        return cls(postgres_settings)
    
    def open_spider(self, spider):
        """Initialize PostgreSQL connection and setup tables"""
        try:
            self.connection = psycopg2.connect(**self.postgres_settings)
            self.cursor = self.connection.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
            
            # Create tables and indexes
            self._create_tables()
            self._create_indexes()
            
            self.logger.info("Connected to PostgreSQL successfully")
            
        except Exception as e:
            self.logger.error(f"Error connecting to PostgreSQL: {e}")
            raise
    
    def _create_tables(self):
        """Create database tables"""
        create_tables_sql = """
        -- Main products table
        CREATE TABLE IF NOT EXISTS products (
            id SERIAL PRIMARY KEY,
            uniqueness_hash VARCHAR(32) UNIQUE,
            name VARCHAR(500) NOT NULL,
            brand VARCHAR(100),
            category VARCHAR(100),
            price DECIMAL(10,2),
            original_price DECIMAL(10,2),
            currency VARCHAR(3) DEFAULT 'USD',
            rating DECIMAL(3,2),
            review_count INTEGER,
            in_stock BOOLEAN DEFAULT true,
            description TEXT,
            specifications JSONB,
            images JSONB,
            url VARCHAR(1000),
            source_spider VARCHAR(50),
            data_quality_score DECIMAL(3,2),
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
            updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
            raw_data JSONB
        );
        
        -- Price history table
        CREATE TABLE IF NOT EXISTS price_history (
            id SERIAL PRIMARY KEY,
            product_id INTEGER REFERENCES products(id),
            price DECIMAL(10,2) NOT NULL,
            original_price DECIMAL(10,2),
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );
        
        -- Categories table
        CREATE TABLE IF NOT EXISTS categories (
            id SERIAL PRIMARY KEY,
            name VARCHAR(100) UNIQUE NOT NULL,
            parent_category VARCHAR(100),
            description TEXT
        );
        
        -- Brands table
        CREATE TABLE IF NOT EXISTS brands (
            id SERIAL PRIMARY KEY,
            name VARCHAR(100) UNIQUE NOT NULL,
            website VARCHAR(255),
            description TEXT
        );
        
        -- Search statistics table
        CREATE TABLE IF NOT EXISTS search_stats (
            id SERIAL PRIMARY KEY,
            search_term VARCHAR(255),
            results_count INTEGER,
            avg_price DECIMAL(10,2),
            searched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );
        """
        
        try:
            self.cursor.execute(create_tables_sql)
            self.connection.commit()
            self.logger.info("Database tables created successfully")
        except Exception as e:
            self.logger.error(f"Error creating tables: {e}")
            self.connection.rollback()
    
    def _create_indexes(self):
        """Create database indexes"""
        indexes_sql = [
            "CREATE INDEX IF NOT EXISTS idx_products_name ON products USING gin(to_tsvector('english', name));",
            "CREATE INDEX IF NOT EXISTS idx_products_brand ON products(brand);",
            "CREATE INDEX IF NOT EXISTS idx_products_category ON products(category);",
            "CREATE INDEX IF NOT EXISTS idx_products_price ON products(price);",
            "CREATE INDEX IF NOT EXISTS idx_products_rating ON products(rating DESC);",
            "CREATE INDEX IF NOT EXISTS idx_products_scraped_at ON products(scraped_at DESC);",
            "CREATE INDEX IF NOT EXISTS idx_products_in_stock ON products(in_stock);",
            "CREATE INDEX IF NOT EXISTS idx_products_data_quality ON products(data_quality_score DESC);",
            "CREATE INDEX IF NOT EXISTS idx_price_history_product_id ON price_history(product_id);",
            "CREATE INDEX IF NOT EXISTS idx_price_history_scraped_at ON price_history(scraped_at DESC);",
            
            # JSONB indexes
            "CREATE INDEX IF NOT EXISTS idx_products_specs ON products USING gin(specifications);",
            "CREATE INDEX IF NOT EXISTS idx_products_raw_data ON products USING gin(raw_data);",
            
            # Full-text search index
            "CREATE INDEX IF NOT EXISTS idx_products_fulltext ON products USING gin(to_tsvector('english', name || ' ' || COALESCE(description, '')));"
        ]
        
        for index_sql in indexes_sql:
            try:
                self.cursor.execute(index_sql)
                self.connection.commit()
            except Exception as e:
                self.logger.warning(f"Error creating index: {e}")
                self.connection.rollback()
    
    def process_item(self, item, spider):
        """Process item and add to batch"""
        adapter = ItemAdapter(item)
        
        # Prepare item for PostgreSQL
        postgres_item = self._prepare_item(adapter, spider)
        self.batch_items.append(postgres_item)
        
        # Process batch if full
        if len(self.batch_items) >= self.batch_size:
            self._process_batch()
        
        return item
    
    def _prepare_item(self, adapter: ItemAdapter, spider) -> Dict:
        """Prepare item for PostgreSQL storage"""
        return {
            'uniqueness_hash': adapter.get('uniqueness_hash'),
            'name': adapter.get('name'),
            'brand': adapter.get('brand'),
            'category': adapter.get('category'),
            'price': adapter.get('price'),
            'original_price': adapter.get('original_price'),
            'currency': adapter.get('currency', 'USD'),
            'rating': adapter.get('rating'),
            'review_count': adapter.get('review_count'),
            'in_stock': adapter.get('in_stock', True),
            'description': adapter.get('description'),
            'specifications': json.dumps(adapter.get('specifications', {})),
            'images': json.dumps(adapter.get('images', [])),
            'url': adapter.get('url'),
            'source_spider': spider.name,
            'data_quality_score': adapter.get('data_quality_score'),
            'raw_data': json.dumps(dict(adapter))
        }
    
    def _process_batch(self):
        """Process batch of items with upsert"""
        if not self.batch_items:
            return
        
        upsert_sql = """
        INSERT INTO products (
            uniqueness_hash, name, brand, category, price, original_price,
            currency, rating, review_count, in_stock, description,
            specifications, images, url, source_spider, data_quality_score, raw_data
        ) VALUES (
            %(uniqueness_hash)s, %(name)s, %(brand)s, %(category)s, %(price)s, %(original_price)s,
            %(currency)s, %(rating)s, %(review_count)s, %(in_stock)s, %(description)s,
            %(specifications)s, %(images)s, %(url)s, %(source_spider)s, %(data_quality_score)s, %(raw_data)s
        )
        ON CONFLICT (uniqueness_hash) DO UPDATE SET
            name = EXCLUDED.name,
            brand = EXCLUDED.brand,
            category = EXCLUDED.category,
            price = EXCLUDED.price,
            original_price = EXCLUDED.original_price,
            rating = EXCLUDED.rating,
            review_count = EXCLUDED.review_count,
            in_stock = EXCLUDED.in_stock,
            description = EXCLUDED.description,
            specifications = EXCLUDED.specifications,
            images = EXCLUDED.images,
            updated_at = CURRENT_TIMESTAMP,
            raw_data = EXCLUDED.raw_data
        RETURNING id, (xmax = 0) AS inserted;
        """
        
        try:
            # Execute batch upsert
            results = []
            for item in self.batch_items:
                self.cursor.execute(upsert_sql, item)
                result = self.cursor.fetchone()
                results.append(result)
            
            # Insert price history
            self._insert_price_history(results)
            
            self.connection.commit()
            
            # Update statistics
            for result in results:
                if result['inserted']:
                    self.stats['inserted'] += 1
                else:
                    self.stats['updated'] += 1
                    
        except Exception as e:
            self.stats['errors'] += len(self.batch_items)
            self.logger.error(f"Error processing batch: {e}")
            self.connection.rollback()
        
        finally:
            self.batch_items.clear()
    
    def _insert_price_history(self, results: List[Dict]):
        """Insert price history records"""
        price_history_sql = """
        INSERT INTO price_history (product_id, price, original_price)
        VALUES (%s, %s, %s)
        """
        
        price_history_data = []
        for i, result in enumerate(results):
            item = self.batch_items[i]
            if item.get('price'):
                price_history_data.append((
                    result['id'],
                    item['price'],
                    item.get('original_price')
                ))
        
        if price_history_data:
            self.cursor.executemany(price_history_sql, price_history_data)
    
    def close_spider(self, spider):
        """Process remaining items and close connection"""
        # Process remaining batch
        self._process_batch()
        
        # Update categories and brands tables
        self._update_reference_tables()
        
        # Log statistics
        self.logger.info("=== PostgreSQL Pipeline Statistics ===")
        for stat, count in self.stats.items():
            self.logger.info(f"{stat.capitalize()}: {count}")
        
        # Close connection
        if self.cursor:
            self.cursor.close()
        if self.connection:
            self.connection.close()
    
    def _update_reference_tables(self):
        """Update categories and brands reference tables"""
        try:
            # Update categories
            self.cursor.execute("""
                INSERT INTO categories (name)
                SELECT DISTINCT category
                FROM products
                WHERE category IS NOT NULL
                ON CONFLICT (name) DO NOTHING;
            """)
            
            # Update brands
            self.cursor.execute("""
                INSERT INTO brands (name)
                SELECT DISTINCT brand
                FROM products
                WHERE brand IS NOT NULL
                ON CONFLICT (name) DO NOTHING;
            """)
            
            self.connection.commit()
            
        except Exception as e:
            self.logger.error(f"Error updating reference tables: {e}")
            self.connection.rollback()

class PostgreSQLAnalytics:
    """Advanced analytics queries for PostgreSQL"""
    
    def __init__(self, postgres_settings):
        self.connection = psycopg2.connect(**postgres_settings)
        self.cursor = self.connection.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
    
    def search_products(self, query: str, limit: int = 50) -> List[Dict]:
        """Full-text search for products"""
        search_sql = """
        SELECT id, name, brand, category, price, rating, review_count, url,
               ts_rank(to_tsvector('english', name || ' ' || COALESCE(description, '')), 
                      plainto_tsquery('english', %s)) AS rank
        FROM products
        WHERE to_tsvector('english', name || ' ' || COALESCE(description, '')) 
              @@ plainto_tsquery('english', %s)
        ORDER BY rank DESC, rating DESC NULLS LAST
        LIMIT %s;
        """
        
        self.cursor.execute(search_sql, (query, query, limit))
        return self.cursor.fetchall()
    
    def get_price_trends(self, product_id: int, days: int = 30) -> List[Dict]:
        """Get price trends for a specific product"""
        trends_sql = """
        SELECT DATE(scraped_at) as date, 
               AVG(price) as avg_price,
               MIN(price) as min_price,
               MAX(price) as max_price,
               COUNT(*) as data_points
        FROM price_history
        WHERE product_id = %s 
          AND scraped_at >= CURRENT_DATE - INTERVAL '%s days'
        GROUP BY DATE(scraped_at)
        ORDER BY date;
        """
        
        self.cursor.execute(trends_sql, (product_id, days))
        return self.cursor.fetchall()
    
    def get_category_insights(self) -> List[Dict]:
        """Get insights by category"""
        insights_sql = """
        SELECT category,
               COUNT(*) as product_count,
               AVG(price) as avg_price,
               MIN(price) as min_price,
               MAX(price) as max_price,
               AVG(rating) as avg_rating,
               COUNT(CASE WHEN in_stock THEN 1 END) as in_stock_count,
               AVG(data_quality_score) as avg_quality_score
        FROM products
        WHERE category IS NOT NULL
        GROUP BY category
        ORDER BY product_count DESC;
        """
        
        self.cursor.execute(insights_sql)
        return self.cursor.fetchall()

Next Steps

That covers the data layer: validation, enrichment, and two storage backends with deduplication and batch upserts. Before moving on, a useful exercise is to wire both pipelines into the spider from Part 3 and check the quality scores it produces on real data. Part 5: Production Deployment covers Docker, Kubernetes, CI/CD, and monitoring.

Scrapy, part 3: staying undetected at scale

Mon, 23 Dec 2024 00:00:00 GMT

Anti-detection and scaling

This part is about running crawls that survive contact with real anti-bot systems: fingerprint-consistent headers, proxy rotation with health checks, distributed workers on Scrapy-Redis, and the monitoring and compliance layers around all of it.

What this part covers

Advanced anti-detection and stealth techniques
Distributed scraping with Scrapy-Redis
IP rotation and proxy management
Browser fingerprinting avoidance
Rate limiting and adaptive throttling
Monitoring, alerting, and health checks
Legal compliance and ethical scraping
Performance optimization and scaling

Anti-Detection Techniques

Detection systems look at far more than your user agent: request timing, header order, and fingerprint consistency all get checked. Here's how to stay under the radar:

Stealth Configuration

# webscraper/settings.py - Advanced Stealth Settings

# Respect robots.txt but with custom delay
ROBOTSTXT_OBEY = True
ROBOTSTXT_USER_AGENT = '*'

# Realistic browser behavior
DOWNLOAD_DELAY = 3  # Base delay between requests
RANDOMIZE_DOWNLOAD_DELAY = True  # 0.5 * to 1.5 * DOWNLOAD_DELAY
DOWNLOAD_TIMEOUT = 30
DOWNLOAD_MAXSIZE = 1073741824  # 1GB
DOWNLOAD_WARNSIZE = 33554432   # 32MB

# Advanced AutoThrottle settings
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0  # Conservative for stealth
AUTOTHROTTLE_DEBUG = False

# Realistic request patterns
CONCURRENT_REQUESTS = 8
CONCURRENT_REQUESTS_PER_DOMAIN = 2

# Advanced cookie handling
COOKIES_ENABLED = True
COOKIES_DEBUG = False

# Memory and resource management
MEMUSAGE_ENABLED = True
MEMUSAGE_LIMIT_MB = 2048
MEMUSAGE_WARNING_MB = 1024

# DNS timeout
DNSCACHE_ENABLED = True
DNSCACHE_SIZE = 10000
DNS_TIMEOUT = 60

# Enable telnet console for debugging (disable in production)
TELNETCONSOLE_ENABLED = False

# Custom middleware stack for stealth
DOWNLOADER_MIDDLEWARES = {
    'webscraper.middlewares.StealthUserAgentMiddleware': 400,
    'webscraper.middlewares.ProxyRotationMiddleware': 410,
    'webscraper.middlewares.HeaderSpoofingMiddleware': 420,
    'webscraper.middlewares.CookiePersistenceMiddleware': 430,
    'webscraper.middlewares.RequestTimingMiddleware': 440,
    'webscraper.middlewares.FingerprintResistanceMiddleware': 450,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 500,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,  # Disabled
}

# Anti-detection headers
DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1',
    'Cache-Control': 'max-age=0',
}

Advanced Stealth Middleware

# webscraper/middlewares.py
import random
import time
import json
import hashlib
from datetime import datetime, timedelta
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
from scrapy import signals
from scrapy.exceptions import IgnoreRequest

class StealthUserAgentMiddleware(UserAgentMiddleware):
    """Advanced user agent rotation with browser consistency"""
    
    def __init__(self):
        # Browser families with consistent headers
        self.browser_profiles = {
            'chrome': {
                'user_agents': [
                    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
                    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
                ],
                'sec_ch_ua': '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
                'sec_ch_ua_mobile': '?0',
                'sec_ch_ua_platform': '"Windows"',
            },
            'firefox': {
                'user_agents': [
                    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0',
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0',
                    'Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0',
                ],
            },
            'safari': {
                'user_agents': [
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15',
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15',
                ],
            }
        }
        self.current_profile = None
        self.session_ua = None
    
    def process_request(self, request, spider):
        # Maintain consistent user agent per session
        if not self.session_ua:
            browser = random.choice(list(self.browser_profiles.keys()))
            self.current_profile = self.browser_profiles[browser]
            self.session_ua = random.choice(self.current_profile['user_agents'])
        
        request.headers['User-Agent'] = self.session_ua
        
        # Add browser-specific headers
        if 'chrome' in self.session_ua.lower():
            profile = self.browser_profiles['chrome']
            request.headers['sec-ch-ua'] = profile['sec_ch_ua']
            request.headers['sec-ch-ua-mobile'] = profile['sec_ch_ua_mobile']
            request.headers['sec-ch-ua-platform'] = profile['sec_ch_ua_platform']

class HeaderSpoofingMiddleware:
    """Spoof headers to mimic real browser behavior"""
    
    def __init__(self):
        self.session_id = hashlib.md5(str(time.time()).encode()).hexdigest()[:16]
        self.request_count = 0
    
    def process_request(self, request, spider):
        self.request_count += 1
        
        # Add realistic timing headers
        request.headers['X-Request-ID'] = f"{self.session_id}-{self.request_count}"
        
        # Randomize header order
        headers_to_randomize = [
            'Accept-Encoding',
            'Accept-Language', 
            'Cache-Control',
            'Connection',
            'DNT',
            'Upgrade-Insecure-Requests'
        ]
        
        # Occasionally omit some headers to appear more human
        if random.random() < 0.1:  # 10% chance
            header_to_remove = random.choice(headers_to_randomize)
            if header_to_remove in request.headers:
                del request.headers[header_to_remove]
        
        # Add referer for non-start URLs
        if not request.meta.get('is_start_url', False):
            if 'Referer' not in request.headers:
                # Use the domain as referer
                domain = request.url.split('/')[2]
                request.headers['Referer'] = f"https://{domain}/"

class RequestTimingMiddleware:
    """Implement realistic request timing patterns"""
    
    def __init__(self):
        self.last_request_time = {}
        self.human_patterns = [
            # Reading pattern: quick succession then pause
            [0.5, 0.3, 0.8, 5.0],
            # Browsing pattern: varied intervals
            [1.2, 2.1, 0.7, 3.5, 1.8],
            # Search pattern: quick then slow
            [0.4, 0.6, 8.0, 2.0],
        ]
        self.current_pattern = []
        self.pattern_index = 0
    
    def process_request(self, request, spider):
        domain = request.url.split('/')[2]
        current_time = time.time()
        
        # Get or initialize timing for this domain
        if domain not in self.last_request_time:
            self.last_request_time[domain] = current_time
            return
        
        # Calculate time since last request to this domain
        time_since_last = current_time - self.last_request_time[domain]
        
        # Choose a human-like delay pattern
        if not self.current_pattern:
            self.current_pattern = random.choice(self.human_patterns).copy()
            self.pattern_index = 0
        
        target_delay = self.current_pattern[self.pattern_index]
        self.pattern_index = (self.pattern_index + 1) % len(self.current_pattern)
        
        # Apply delay if needed
        if time_since_last < target_delay:
            sleep_time = target_delay - time_since_last
            spider.logger.debug(f'Applying human-like delay: {sleep_time:.2f}s for {domain}')
            time.sleep(sleep_time)
        
        self.last_request_time[domain] = time.time()

class FingerprintResistanceMiddleware:
    """Resist browser fingerprinting attempts"""
    
    def process_request(self, request, spider):
        # Randomize Accept header slightly
        accept_variations = [
            'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
            'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        ]
        
        if random.random() < 0.3:  # 30% chance to vary
            request.headers['Accept'] = random.choice(accept_variations)
        
        # Randomize Accept-Language
        lang_variations = [
            'en-US,en;q=0.9',
            'en-US,en;q=0.8',
            'en-US,en;q=0.9,es;q=0.8',
            'en-US,en;q=0.5',
        ]
        
        if random.random() < 0.2:  # 20% chance to vary
            request.headers['Accept-Language'] = random.choice(lang_variations)

class CookiePersistenceMiddleware:
    """Advanced cookie management for session consistency"""
    
    def __init__(self):
        self.session_cookies = {}
    
    def process_response(self, request, response, spider):
        # Store cookies for this domain
        domain = request.url.split('/')[2]
        cookies = response.headers.getlist('Set-Cookie')
        
        if cookies:
            if domain not in self.session_cookies:
                self.session_cookies[domain] = {}
            
            for cookie in cookies:
                cookie_str = cookie.decode('utf-8')
                if '=' in cookie_str:
                    name, value = cookie_str.split('=', 1)
                    # Store only the name=value part
                    value = value.split(';')[0]
                    self.session_cookies[domain][name] = value
        
        return response
    
    def process_request(self, request, spider):
        domain = request.url.split('/')[2]
        
        # Add stored cookies for this domain
        if domain in self.session_cookies:
            cookie_header = '; '.join([
                f"{name}={value}" 
                for name, value in self.session_cookies[domain].items()
            ])
            if cookie_header:
                request.headers['Cookie'] = cookie_header

Proxy Rotation System

# webscraper/proxy_manager.py
import random
import time
import requests
from typing import List, Dict, Optional
import threading
from dataclasses import dataclass
from enum import Enum

class ProxyStatus(Enum):
    ACTIVE = "active"
    FAILED = "failed"
    RATE_LIMITED = "rate_limited"
    TESTING = "testing"

@dataclass
class ProxyInfo:
    host: str
    port: int
    username: Optional[str] = None
    password: Optional[str] = None
    protocol: str = "http"
    status: ProxyStatus = ProxyStatus.TESTING
    success_count: int = 0
    failure_count: int = 0
    last_used: Optional[float] = None
    response_time: Optional[float] = None
    rate_limit_until: Optional[float] = None
    
    @property
    def url(self) -> str:
        if self.username and self.password:
            return f"{self.protocol}://{self.username}:{self.password}@{self.host}:{self.port}"
        return f"{self.protocol}://{self.host}:{self.port}"
    
    @property
    def success_rate(self) -> float:
        total = self.success_count + self.failure_count
        return self.success_count / total if total > 0 else 0.0
    
    def is_available(self) -> bool:
        if self.status == ProxyStatus.FAILED:
            return False
        if self.rate_limit_until and time.time() < self.rate_limit_until:
            return False
        return True

class ProxyManager:
    """Advanced proxy rotation and health management"""
    
    def __init__(self, proxy_list: List[Dict], health_check_interval: int = 300):
        self.proxies = [ProxyInfo(**proxy) for proxy in proxy_list]
        self.health_check_interval = health_check_interval
        self.last_health_check = 0
        self.lock = threading.Lock()
        
        # Start health checking thread
        self.health_thread = threading.Thread(target=self._health_check_loop, daemon=True)
        self.health_thread.start()
    
    def get_proxy(self) -> Optional[ProxyInfo]:
        """Get the best available proxy"""
        with self.lock:
            available_proxies = [p for p in self.proxies if p.is_available()]
            
            if not available_proxies:
                return None
            
            # Sort by success rate and response time
            available_proxies.sort(
                key=lambda p: (p.success_rate, -p.response_time or 0),
                reverse=True
            )
            
            # Weighted selection favoring better proxies
            weights = [max(0.1, p.success_rate) for p in available_proxies]
            proxy = random.choices(available_proxies, weights=weights)[0]
            
            proxy.last_used = time.time()
            return proxy
    
    def report_success(self, proxy: ProxyInfo, response_time: float = None):
        """Report successful proxy usage"""
        with self.lock:
            proxy.success_count += 1
            proxy.status = ProxyStatus.ACTIVE
            if response_time:
                proxy.response_time = response_time
    
    def report_failure(self, proxy: ProxyInfo, is_rate_limit: bool = False):
        """Report proxy failure"""
        with self.lock:
            proxy.failure_count += 1
            
            if is_rate_limit:
                proxy.status = ProxyStatus.RATE_LIMITED
                proxy.rate_limit_until = time.time() + 300  # 5 minute cooldown
            elif proxy.failure_count > 5:
                proxy.status = ProxyStatus.FAILED
    
    def _health_check_loop(self):
        """Background thread for proxy health checking"""
        while True:
            time.sleep(self.health_check_interval)
            self._perform_health_checks()
    
    def _perform_health_checks(self):
        """Check health of all proxies"""
        test_url = "http://httpbin.org/ip"
        
        for proxy in self.proxies:
            if proxy.status == ProxyStatus.FAILED:
                continue
                
            try:
                start_time = time.time()
                response = requests.get(
                    test_url,
                    proxies={"http": proxy.url, "https": proxy.url},
                    timeout=10
                )
                response_time = time.time() - start_time
                
                if response.status_code == 200:
                    self.report_success(proxy, response_time)
                else:
                    self.report_failure(proxy)
                    
            except Exception:
                self.report_failure(proxy)
    
    def get_stats(self) -> Dict:
        """Get proxy pool statistics"""
        with self.lock:
            total = len(self.proxies)
            active = sum(1 for p in self.proxies if p.status == ProxyStatus.ACTIVE)
            failed = sum(1 for p in self.proxies if p.status == ProxyStatus.FAILED)
            rate_limited = sum(1 for p in self.proxies if p.status == ProxyStatus.RATE_LIMITED)
            
            return {
                'total': total,
                'active': active,
                'failed': failed,
                'rate_limited': rate_limited,
                'success_rate': sum(p.success_rate for p in self.proxies) / total if total > 0 else 0
            }

# Proxy rotation middleware
class AdvancedProxyRotationMiddleware:
    """Advanced proxy rotation with health management"""
    
    def __init__(self, proxy_list):
        self.proxy_manager = ProxyManager(proxy_list)
    
    @classmethod
    def from_crawler(cls, crawler):
        proxy_list = crawler.settings.get('PROXY_LIST', [])
        return cls(proxy_list)
    
    def process_request(self, request, spider):
        proxy = self.proxy_manager.get_proxy()
        
        if proxy:
            request.meta['proxy'] = proxy.url
            request.meta['proxy_info'] = proxy
            spider.logger.debug(f'Using proxy: {proxy.host}:{proxy.port}')
        else:
            spider.logger.warning('No available proxies')
    
    def process_response(self, request, response, spider):
        proxy_info = request.meta.get('proxy_info')
        
        if proxy_info:
            if response.status == 200:
                self.proxy_manager.report_success(proxy_info)
            elif response.status == 429:  # Rate limited
                self.proxy_manager.report_failure(proxy_info, is_rate_limit=True)
            elif response.status >= 400:
                self.proxy_manager.report_failure(proxy_info)
        
        return response
    
    def process_exception(self, request, exception, spider):
        proxy_info = request.meta.get('proxy_info')
        
        if proxy_info:
            self.proxy_manager.report_failure(proxy_info)

Distributed Scraping with Scrapy-Redis

When one machine isn't enough, Scrapy-Redis coordinates multiple workers through a shared queue:

Scrapy-Redis Setup

# Install Scrapy-Redis
pip install scrapy-redis

# Start Redis server
redis-server

# Or using Docker
docker run -d -p 6379:6379 redis:alpine

Distributed Spider Configuration

# webscraper/settings.py - Redis Configuration

# Enable Scrapy-Redis
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 300,
    'webscraper.pipelines.ValidationPipeline': 400,
}

# Redis connection
REDIS_URL = 'redis://localhost:6379'

# Or with detailed configuration
REDIS_PARAMS = {
    'host': 'localhost',
    'port': 6379,
    'db': 0,
    'password': None,
}

# Scheduler configuration
SCHEDULER_PERSIST = True  # Keep scheduler data after spider closes
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.PriorityQueue'

# Request serialization
SCHEDULER_SERIALIZER = "scrapy_redis.picklecompat"

Distributed Spider

# webscraper/spiders/distributed_spider.py
from scrapy_redis.spiders import RedisSpider
from webscraper.items import ProductItem
from scrapy.loader import ItemLoader
import json

class DistributedEcommerceSpider(RedisSpider):
    """Distributed spider using Redis for coordination"""
    
    name = 'distributed_ecommerce'
    redis_key = 'distributed_ecommerce:start_urls'
    
    custom_settings = {
        'CONCURRENT_REQUESTS': 32,
        'CONCURRENT_REQUESTS_PER_DOMAIN': 16,
        'DOWNLOAD_DELAY': 1,
        'SCHEDULER_IDLE_BEFORE_CLOSE': 10,  # Wait 10 seconds before closing
    }
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.processed_count = 0
        self.error_count = 0
    
    def parse(self, response):
        """Parse category pages and product listings"""
        self.logger.info(f'Worker {self.name} processing: {response.url}')
        
        # Extract product links
        product_links = response.css('.product-item a::attr(href)').getall()
        
        for link in product_links:
            product_url = response.urljoin(link)
            yield response.follow(
                product_url,
                callback=self.parse_product,
                meta={'category_url': response.url}
            )
        
        # Extract pagination
        next_page = response.css('.pagination .next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)
        
        # Add more category pages to Redis queue
        self.discover_more_urls(response)
    
    def parse_product(self, response):
        """Parse individual product pages"""
        try:
            loader = ItemLoader(item=ProductItem(), response=response)
            
            loader.add_css('name', 'h1.product-title::text')
            loader.add_css('price', '.price-current::text')
            loader.add_css('description', '.product-description::text')
            loader.add_css('brand', '.brand-name::text')
            loader.add_css('rating', '.rating-value::text')
            
            # Add metadata
            loader.add_value('url', response.url)
            loader.add_value('category_url', response.meta.get('category_url'))
            loader.add_value('worker_id', self.name)
            loader.add_value('scraped_at', self.get_current_time())
            
            item = loader.load_item()
            self.processed_count += 1
            
            if self.processed_count % 100 == 0:
                self.logger.info(f'Worker {self.name} processed {self.processed_count} products')
            
            yield item
            
        except Exception as e:
            self.error_count += 1
            self.logger.error(f'Error parsing product {response.url}: {e}')
    
    def discover_more_urls(self, response):
        """Discover and add more URLs to the Redis queue"""
        # Find category links
        category_links = response.css('.category-nav a::attr(href)').getall()
        
        for link in category_links:
            category_url = response.urljoin(link)
            # Add to Redis queue for other workers
            self.server.lpush(
                f'{self.redis_key}:discovered',
                json.dumps({'url': category_url, 'priority': 1})
            )
    
    def get_current_time(self):
        from datetime import datetime
        return datetime.now().isoformat()
    
    def closed(self, reason):
        """Spider closing callback"""
        self.logger.info(f'Worker {self.name} closing: {reason}')
        self.logger.info(f'Processed: {self.processed_count}, Errors: {self.error_count}')

# Master coordinator spider
class CoordinatorSpider(RedisSpider):
    """Coordinator spider that feeds URLs to workers"""
    
    name = 'coordinator'
    redis_key = 'distributed_ecommerce:start_urls'
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.seed_urls()
    
    def seed_urls(self):
        """Seed initial URLs into Redis"""
        initial_urls = [
            'https://example-store.com/categories/electronics',
            'https://example-store.com/categories/clothing',
            'https://example-store.com/categories/home',
            'https://example-store.com/categories/books',
        ]
        
        for url in initial_urls:
            self.server.lpush(self.redis_key, url)
        
        self.logger.info(f'Seeded {len(initial_urls)} URLs')
    
    def parse(self, response):
        """This spider doesn't parse, it just coordinates"""
        pass

Worker Management Script

# scripts/worker_manager.py
import subprocess
import sys
import time
import signal
import json
import redis
from typing import List, Dict

class WorkerManager:
    """Manage distributed scraping workers"""
    
    def __init__(self, redis_url: str = 'redis://localhost:6379'):
        self.redis_client = redis.from_url(redis_url)
        self.workers: List[subprocess.Popen] = []
        self.running = True
        
        # Register signal handlers
        signal.signal(signal.SIGINT, self.shutdown_handler)
        signal.signal(signal.SIGTERM, self.shutdown_handler)
    
    def start_workers(self, spider_name: str, num_workers: int = 4):
        """Start multiple worker processes"""
        print(f"Starting {num_workers} workers for spider '{spider_name}'")
        
        for i in range(num_workers):
            worker_id = f"{spider_name}_worker_{i}"
            cmd = [
                'scrapy', 'crawl', spider_name,
                '-s', f'BOT_NAME={worker_id}',
                '-L', 'INFO'
            ]
            
            worker = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
                text=True
            )
            
            self.workers.append(worker)
            print(f"Started worker {worker_id} (PID: {worker.pid})")
    
    def monitor_workers(self):
        """Monitor worker health and restart if needed"""
        while self.running:
            for i, worker in enumerate(self.workers):
                if worker.poll() is not None:  # Worker has terminated
                    print(f"Worker {i} terminated with code {worker.returncode}")
                    
                    # Restart worker if it crashed
                    if worker.returncode != 0:
                        print(f"Restarting worker {i}")
                        # Implementation for restarting worker
                        pass
            
            # Monitor Redis queue sizes
            self.print_queue_stats()
            
            time.sleep(30)  # Check every 30 seconds
    
    def print_queue_stats(self):
        """Print Redis queue statistics"""
        try:
            queue_size = self.redis_client.llen('distributed_ecommerce:start_urls')
            processing = self.redis_client.scard('distributed_ecommerce:dupefilter')
            
            print(f"Queue size: {queue_size}, Processed: {processing}")
            
        except Exception as e:
            print(f"Error getting Redis stats: {e}")
    
    def shutdown_handler(self, signum, frame):
        """Handle shutdown signals"""
        print("\nShutting down workers...")
        self.running = False
        
        for i, worker in enumerate(self.workers):
            print(f"Terminating worker {i}")
            worker.terminate()
            
            # Wait for graceful shutdown
            try:
                worker.wait(timeout=10)
            except subprocess.TimeoutExpired:
                print(f"Force killing worker {i}")
                worker.kill()
        
        print("All workers shut down")
        sys.exit(0)
    
    def run(self, spider_name: str, num_workers: int = 4):
        """Main execution method"""
        self.start_workers(spider_name, num_workers)
        self.monitor_workers()

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python worker_manager.py <spider_name> [num_workers]")
        sys.exit(1)
    
    spider_name = sys.argv[1]
    num_workers = int(sys.argv[2]) if len(sys.argv) > 2 else 4
    
    manager = WorkerManager()
    manager.run(spider_name, num_workers)

Monitoring and Alerting

Monitoring System

# webscraper/monitoring.py
import time
import json
import smtplib
from datetime import datetime, timedelta
from email.mime.text import MimeText
from email.mime.multipart import MimeMultipart
from typing import Dict, List, Optional
import psutil
import redis
from dataclasses import dataclass, asdict

@dataclass
class ScrapingMetrics:
    timestamp: str
    spider_name: str
    pages_scraped: int
    items_extracted: int
    errors: int
    response_time_avg: float
    memory_usage_mb: float
    cpu_usage_percent: float
    active_requests: int
    queue_size: int
    success_rate: float

class PerformanceMonitor:
    """Monitor scraping performance and system resources"""
    
    def __init__(self, redis_url: str = 'redis://localhost:6379'):
        self.redis_client = redis.from_url(redis_url)
        self.metrics_history: List[ScrapingMetrics] = []
        self.alert_thresholds = {
            'error_rate': 0.1,      # 10% error rate
            'memory_usage': 2048,    # 2GB memory usage
            'cpu_usage': 80,         # 80% CPU usage
            'response_time': 10,     # 10 second average response time
            'queue_stall': 300,      # 5 minutes without queue progress
        }
        self.last_queue_size = 0
        self.queue_stall_start = None
    
    def collect_metrics(self, spider_name: str) -> ScrapingMetrics:
        """Collect current performance metrics"""
        # Get system metrics
        memory_usage = psutil.virtual_memory().used / 1024 / 1024  # MB
        cpu_usage = psutil.cpu_percent(interval=1)
        
        # Get Redis metrics
        try:
            queue_size = self.redis_client.llen(f'{spider_name}:start_urls')
            processed_count = self.redis_client.scard(f'{spider_name}:dupefilter')
            
            # Detect queue stall
            if queue_size == self.last_queue_size:
                if not self.queue_stall_start:
                    self.queue_stall_start = time.time()
            else:
                self.queue_stall_start = None
            
            self.last_queue_size = queue_size
            
        except Exception:
            queue_size = 0
            processed_count = 0
        
        # Create metrics object (simplified - in real implementation, 
        # you'd collect these from your spider's stats)
        metrics = ScrapingMetrics(
            timestamp=datetime.now().isoformat(),
            spider_name=spider_name,
            pages_scraped=processed_count,
            items_extracted=processed_count * 0.8,  # Estimate
            errors=processed_count * 0.05,  # Estimate
            response_time_avg=2.5,  # Would be collected from actual stats
            memory_usage_mb=memory_usage,
            cpu_usage_percent=cpu_usage,
            active_requests=queue_size,
            queue_size=queue_size,
            success_rate=0.95  # Would be calculated from actual stats
        )
        
        self.metrics_history.append(metrics)
        
        # Keep only last 24 hours of metrics
        cutoff_time = datetime.now() - timedelta(hours=24)
        self.metrics_history = [
            m for m in self.metrics_history 
            if datetime.fromisoformat(m.timestamp) > cutoff_time
        ]
        
        return metrics
    
    def check_alerts(self, metrics: ScrapingMetrics) -> List[Dict]:
        """Check metrics against alert thresholds"""
        alerts = []
        
        # Error rate alert
        error_rate = metrics.errors / max(1, metrics.pages_scraped)
        if error_rate > self.alert_thresholds['error_rate']:
            alerts.append({
                'type': 'error_rate',
                'severity': 'high',
                'message': f'High error rate: {error_rate:.2%}',
                'value': error_rate,
                'threshold': self.alert_thresholds['error_rate']
            })
        
        # Memory usage alert
        if metrics.memory_usage_mb > self.alert_thresholds['memory_usage']:
            alerts.append({
                'type': 'memory_usage',
                'severity': 'medium',
                'message': f'High memory usage: {metrics.memory_usage_mb:.1f}MB',
                'value': metrics.memory_usage_mb,
                'threshold': self.alert_thresholds['memory_usage']
            })
        
        # CPU usage alert
        if metrics.cpu_usage_percent > self.alert_thresholds['cpu_usage']:
            alerts.append({
                'type': 'cpu_usage',
                'severity': 'medium',
                'message': f'High CPU usage: {metrics.cpu_usage_percent:.1f}%',
                'value': metrics.cpu_usage_percent,
                'threshold': self.alert_thresholds['cpu_usage']
            })
        
        # Response time alert
        if metrics.response_time_avg > self.alert_thresholds['response_time']:
            alerts.append({
                'type': 'response_time',
                'severity': 'medium',
                'message': f'Slow response time: {metrics.response_time_avg:.1f}s',
                'value': metrics.response_time_avg,
                'threshold': self.alert_thresholds['response_time']
            })
        
        # Queue stall alert
        if self.queue_stall_start:
            stall_duration = time.time() - self.queue_stall_start
            if stall_duration > self.alert_thresholds['queue_stall']:
                alerts.append({
                    'type': 'queue_stall',
                    'severity': 'high',
                    'message': f'Queue stalled for {stall_duration/60:.1f} minutes',
                    'value': stall_duration,
                    'threshold': self.alert_thresholds['queue_stall']
                })
        
        return alerts
    
    def generate_report(self, hours: int = 24) -> Dict:
        """Generate performance report"""
        cutoff_time = datetime.now() - timedelta(hours=hours)
        recent_metrics = [
            m for m in self.metrics_history 
            if datetime.fromisoformat(m.timestamp) > cutoff_time
        ]
        
        if not recent_metrics:
            return {'error': 'No metrics available'}
        
        # Calculate aggregated statistics
        total_pages = sum(m.pages_scraped for m in recent_metrics)
        total_items = sum(m.items_extracted for m in recent_metrics)
        total_errors = sum(m.errors for m in recent_metrics)
        avg_response_time = sum(m.response_time_avg for m in recent_metrics) / len(recent_metrics)
        avg_memory = sum(m.memory_usage_mb for m in recent_metrics) / len(recent_metrics)
        avg_cpu = sum(m.cpu_usage_percent for m in recent_metrics) / len(recent_metrics)
        
        return {
            'period': f'Last {hours} hours',
            'total_pages_scraped': total_pages,
            'total_items_extracted': total_items,
            'total_errors': total_errors,
            'error_rate': total_errors / max(1, total_pages),
            'average_response_time': avg_response_time,
            'average_memory_usage_mb': avg_memory,
            'average_cpu_usage_percent': avg_cpu,
            'current_queue_size': recent_metrics[-1].queue_size if recent_metrics else 0,
            'metrics_count': len(recent_metrics)
        }

class AlertManager:
    """Manage and send alerts"""
    
    def __init__(self, config: Dict):
        self.config = config
        self.sent_alerts = {}  # Track sent alerts to avoid spam
        self.cooldown_period = 300  # 5 minutes cooldown between same alerts
    
    def send_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Send alert via configured channels"""
        alert_key = f"{alert['type']}_{alert.get('severity', 'medium')}"
        current_time = time.time()
        
        # Check cooldown
        if alert_key in self.sent_alerts:
            if current_time - self.sent_alerts[alert_key] < self.cooldown_period:
                return  # Skip sending due to cooldown
        
        # Send via email
        if self.config.get('email'):
            self._send_email_alert(alert, metrics)
        
        # Send via webhook
        if self.config.get('webhook'):
            self._send_webhook_alert(alert, metrics)
        
        # Log to file
        self._log_alert(alert, metrics)
        
        self.sent_alerts[alert_key] = current_time
    
    def _send_email_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Send alert via email"""
        try:
            smtp_config = self.config['email']
            
            msg = MimeMultipart()
            msg['From'] = smtp_config['from']
            msg['To'] = smtp_config['to']
            msg['Subject'] = f"Scraping Alert: {alert['type']} - {alert['severity'].upper()}"
            
            body = f"""
            Alert: {alert['message']}
            
            Spider: {metrics.spider_name}
            Time: {metrics.timestamp}
            Current Value: {alert['value']}
            Threshold: {alert['threshold']}
            
            Current Metrics:
            - Pages Scraped: {metrics.pages_scraped}
            - Items Extracted: {metrics.items_extracted}
            - Error Rate: {metrics.errors / max(1, metrics.pages_scraped):.2%}
            - Memory Usage: {metrics.memory_usage_mb:.1f}MB
            - CPU Usage: {metrics.cpu_usage_percent:.1f}%
            - Queue Size: {metrics.queue_size}
            """
            
            msg.attach(MimeText(body, 'plain'))
            
            with smtplib.SMTP(smtp_config['smtp_server'], smtp_config['smtp_port']) as server:
                if smtp_config.get('use_tls'):
                    server.starttls()
                if smtp_config.get('username'):
                    server.login(smtp_config['username'], smtp_config['password'])
                server.send_message(msg)
                
        except Exception as e:
            print(f"Failed to send email alert: {e}")
    
    def _send_webhook_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Send alert via webhook"""
        import requests
        
        try:
            webhook_config = self.config['webhook']
            
            payload = {
                'alert': alert,
                'metrics': asdict(metrics),
                'timestamp': metrics.timestamp
            }
            
            response = requests.post(
                webhook_config['url'],
                json=payload,
                headers=webhook_config.get('headers', {}),
                timeout=10
            )
            response.raise_for_status()
            
        except Exception as e:
            print(f"Failed to send webhook alert: {e}")
    
    def _log_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Log alert to file"""
        import logging
        
        logging.basicConfig(
            filename='scraping_alerts.log',
            level=logging.INFO,
            format='%(asctime)s - %(levelname)s - %(message)s'
        )
        
        logging.info(
            f"ALERT: {alert['type']} - {alert['message']} "
            f"(Spider: {metrics.spider_name}, Value: {alert['value']})"
        )

Monitoring Dashboard Script

# scripts/monitoring_dashboard.py
import time
import json
from webscraper.monitoring import PerformanceMonitor, AlertManager

def main():
    """Main monitoring loop"""
    
    # Configuration
    config = {
        'email': {
            'smtp_server': 'smtp.gmail.com',
            'smtp_port': 587,
            'use_tls': True,
            'username': 'your-email@gmail.com',
            'password': 'your-app-password',
            'from': 'your-email@gmail.com',
            'to': 'alerts@yourcompany.com'
        },
        'webhook': {
            'url': 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK',
            'headers': {'Content-Type': 'application/json'}
        }
    }
    
    monitor = PerformanceMonitor()
    alert_manager = AlertManager(config)
    
    spider_name = 'distributed_ecommerce'
    
    print("Starting monitoring dashboard...")
    print("Press Ctrl+C to stop")
    
    try:
        while True:
            # Collect metrics
            metrics = monitor.collect_metrics(spider_name)
            
            # Check for alerts
            alerts = monitor.check_alerts(metrics)
            
            # Send alerts
            for alert in alerts:
                alert_manager.send_alert(alert, metrics)
                print(f"ALERT: {alert['message']}")
            
            # Print current status
            print(f"\n[{metrics.timestamp}] Spider: {spider_name}")
            print(f"Pages: {metrics.pages_scraped}, Items: {metrics.items_extracted}")
            print(f"Errors: {metrics.errors}, Queue: {metrics.queue_size}")
            print(f"Memory: {metrics.memory_usage_mb:.1f}MB, CPU: {metrics.cpu_usage_percent:.1f}%")
            
            if alerts:
                print(f"Active alerts: {len(alerts)}")
            
            # Generate hourly report
            if int(time.time()) % 3600 == 0:  # Every hour
                report = monitor.generate_report(hours=1)
                print("\n--- Hourly Report ---")
                print(json.dumps(report, indent=2))
            
            time.sleep(60)  # Check every minute
            
    except KeyboardInterrupt:
        print("\nMonitoring stopped")

if __name__ == "__main__":
    main()

Legal Compliance and Ethics

Compliance Framework

# webscraper/compliance.py
import time
import requests
from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin, urlparse
from typing import Dict, List, Optional
import logging

class ComplianceManager:
    """Ensure legal and ethical scraping compliance"""
    
    def __init__(self):
        self.robots_cache = {}
        self.rate_limits = {}
        self.compliance_rules = {
            'respect_robots_txt': True,
            'rate_limit_default': 1.0,  # 1 second between requests
            'rate_limit_per_domain': {},
            'user_agent_required': True,
            'contact_info': 'your-email@company.com',
            'personal_data_handling': 'exclude',
            'copyright_respect': True,
        }
        
        self.logger = logging.getLogger(__name__)
    
    def check_robots_txt(self, url: str, user_agent: str = '*') -> bool:
        """Check if URL is allowed by robots.txt"""
        domain = urlparse(url).netloc
        
        if domain not in self.robots_cache:
            robots_url = urljoin(f"https://{domain}", "/robots.txt")
            
            try:
                rp = RobotFileParser()
                rp.set_url(robots_url)
                rp.read()
                self.robots_cache[domain] = rp
                
                # Extract crawl delay if specified
                delay = rp.crawl_delay(user_agent)
                if delay:
                    self.rate_limits[domain] = float(delay)
                
            except Exception as e:
                self.logger.warning(f"Could not fetch robots.txt for {domain}: {e}")
                return True  # Allow if robots.txt can't be fetched
        
        robots_parser = self.robots_cache.get(domain)
        if robots_parser:
            return robots_parser.can_fetch(user_agent, url)
        
        return True
    
    def get_rate_limit(self, domain: str) -> float:
        """Get appropriate rate limit for domain"""
        # Check domain-specific rate limit
        if domain in self.compliance_rules['rate_limit_per_domain']:
            return self.compliance_rules['rate_limit_per_domain'][domain]
        
        # Check robots.txt specified delay
        if domain in self.rate_limits:
            return self.rate_limits[domain]
        
        # Use default
        return self.compliance_rules['rate_limit_default']
    
    def validate_request(self, request) -> bool:
        """Validate request against compliance rules"""
        url = request.url
        domain = urlparse(url).netloc
        user_agent = request.headers.get('User-Agent', '')
        
        # Check robots.txt
        if self.compliance_rules['respect_robots_txt']:
            if not self.check_robots_txt(url, user_agent):
                self.logger.warning(f"Blocked by robots.txt: {url}")
                return False
        
        # Check user agent requirement
        if self.compliance_rules['user_agent_required']:
            if not user_agent or 'bot' not in user_agent.lower():
                self.logger.warning(f"Invalid user agent for compliance: {user_agent}")
                return False
        
        return True
    
    def filter_personal_data(self, item: Dict) -> Dict:
        """Filter out personal data from scraped items"""
        if self.compliance_rules['personal_data_handling'] == 'exclude':
            sensitive_fields = [
                'email', 'phone', 'address', 'ssn', 'credit_card',
                'personal_id', 'passport', 'driver_license'
            ]
            
            filtered_item = {}
            for key, value in item.items():
                if key.lower() not in sensitive_fields:
                    # Additional check for values that look like personal data
                    if not self.looks_like_personal_data(str(value)):
                        filtered_item[key] = value
                    else:
                        self.logger.info(f"Filtered potential personal data: {key}")
            
            return filtered_item
        
        return item
    
    def looks_like_personal_data(self, text: str) -> bool:
        """Heuristic check for personal data patterns"""
        import re
        
        patterns = [
            r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',  # Email
            r'\b\d{3}-\d{2}-\d{4}\b',  # SSN
            r'\b\d{3}-\d{3}-\d{4}\b',  # Phone
            r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b',  # Credit card
        ]
        
        for pattern in patterns:
            if re.search(pattern, text):
                return True
        
        return False
    
    def generate_compliance_report(self) -> Dict:
        """Generate compliance status report"""
        return {
            'robots_txt_domains_checked': len(self.robots_cache),
            'custom_rate_limits': len(self.rate_limits),
            'compliance_rules': self.compliance_rules,
            'timestamp': time.time()
        }

# Compliance middleware
class ComplianceMiddleware:
    """Scrapy middleware for compliance enforcement"""
    
    def __init__(self):
        self.compliance_manager = ComplianceManager()
        self.last_request_time = {}
    
    def process_request(self, request, spider):
        # Validate compliance
        if not self.compliance_manager.validate_request(request):
            raise IgnoreRequest(f"Request blocked by compliance rules: {request.url}")
        
        # Enforce rate limiting
        domain = urlparse(request.url).netloc
        rate_limit = self.compliance_manager.get_rate_limit(domain)
        
        if domain in self.last_request_time:
            time_since_last = time.time() - self.last_request_time[domain]
            if time_since_last < rate_limit:
                sleep_time = rate_limit - time_since_last
                spider.logger.debug(f"Rate limiting: sleeping {sleep_time:.2f}s for {domain}")
                time.sleep(sleep_time)
        
        self.last_request_time[domain] = time.time()
    
    def process_item(self, item, spider):
        # Filter personal data
        filtered_item = self.compliance_manager.filter_personal_data(dict(item))
        return filtered_item

Summary

This part covered the operational side: stealth middleware and fingerprint consistency, proxy pools with health checks, distributed workers on Scrapy-Redis, monitoring with alerting, and a compliance layer that enforces robots.txt and rate limits.

What's Next?

Part 4: Data Processing and Storage covers:

Advanced data cleaning and validation
Multiple storage backends (MongoDB, PostgreSQL, Elasticsearch)
Real-time data pipelines
Data quality monitoring
ETL processes and data warehousing

Practice Exercise

Build a distributed scraper that:

Uses anti-detection techniques to scrape a major e-commerce site
Scales across multiple workers with Redis coordination
Implements monitoring and alerting
Ensures legal compliance and ethical scraping
Handles millions of products with proper rate limiting

Scrapy, part 2: JavaScript, forms, and dynamic pages

Sun, 22 Dec 2024 00:00:00 GMT

JavaScript, forms, and dynamic pages

Part 1 dealt with sites that hand you clean HTML. This part is about the ones that do not: pages that render with JavaScript, data hidden behind a login, and content that only appears after a form or a scroll.

All the code is on GitHub: scrapy-tutorial/part-2-dynamic-pages. Same as before, the spiders point at placeholder sites so the focus stays on the technique. Swap the domain and selectors to run them for real.

First, decide how to get the data

When a page renders with JavaScript, most people jump straight to a headless browser. That is usually the slowest option. There are three routes, and I try them in this order.

Reach for a browser last, not first. If the data arrives over an API call, hitting that endpoint is faster and far more stable than rendering the whole page. So before writing any Splash code, open the network tab and look for the request that returns the data.

Rendering with Splash

Full file: webscraper/spiders/spa_spider.py

When the data really is built in the browser, Splash runs the JavaScript for you. It takes a small Lua script that says what to do on the page: load it, wait for the content, scroll, click. You send a SplashRequest instead of a normal one.

yield SplashRequest(
    url=url,
    callback=self.parse,
    args={'lua_source': LUA_SOURCE, 'timeout': 30, 'wait': 5},
)

The Lua part is where you wait for the right element and interact with the page:

splash:go(args.url)
splash:wait(3)
splash:runjs("window.scrollTo(0, document.body.scrollHeight);")

local load_more = splash:select('.load-more-btn')
if load_more then
    load_more:click()
    splash:wait(3)
end

Splash needs Docker running: docker run -p 8050:8050 scrapinghub/splash. The full file has the complete Lua script and the settings block to wire it in.

Logging in and holding the session

Full file: webscraper/spiders/login_spider.py

The trick with login forms is FormRequest.from_response. It reads the existing form off the page, so hidden fields and CSRF tokens come along for free. You only add the fields you care about.

return FormRequest.from_response(
    response,
    formdata={'username': 'you', 'password': 'secret', 'csrf_token': csrf_token},
    callback=self.after_login,
)

Scrapy keeps the session cookies after that, so the pages you visit next are already logged in. Always confirm the login actually worked before scraping, by checking for something that only shows up when signed in.

Filling a search form

Full file: webscraper/spiders/form_spider.py

Search forms are the same idea. Carry over every field the page already has, then set your own search terms on top. Copying the existing fields matters, because servers often reject a submission that is missing a hidden token they expected.

Going straight to the API

Full file: webscraper/spiders/ajax_spider.py

This is the route I reach for most. If the page loads its products from a JSON endpoint, you can skip the page and request that endpoint yourself. It returns clean structured data, no parsing of messy HTML, and it is fast.

params = {'page': page, 'limit': 20, 'format': 'json'}
yield scrapy.Request(
    url=f'https://api-example.com/api/products?{urlencode(params)}',
    callback=self.parse_api_response,
    headers={'Accept': 'application/json', 'X-Requested-With': 'XMLHttpRequest'},
)

The one header that matters most is often X-Requested-With: XMLHttpRequest, since some APIs check for it. Copy the headers the real page sends and you are usually in.

Infinite scroll

Full file: webscraper/spiders/infinite_scroll_spider.py

Infinite scroll is just more of the same two options. Either drive Splash to scroll and click until the end marker appears, or find the pagination API behind the scroll and page through it directly. The second is almost always the better deal.

Middleware: rotate and retry

Full file: webscraper/middlewares.py

Middleware lets you change every request in one place. Two are worth having early: one that rotates the user agent, and one that backs off and retries when a site starts returning 429s or 503s.

def process_response(self, request, response, spider):
    if response.status in (429, 502, 503, 504):
        retries = request.meta.get('retry_times', 0)
        if retries < self.max_retry_times:
            time.sleep(self.initial_delay * (2 ** retries))
            retry = request.copy()
            retry.meta['retry_times'] = retries + 1
            retry.dont_filter = True
            return retry
    return response

The delay doubles each attempt, so a struggling site gets room to breathe instead of a second wave of requests. The file also has proxy rotation and header rotation.

Handy selector helpers

Full file: webscraper/utils/selectors.py

A few small helpers save repetition: one that tries a list of selectors and returns the first hit, one that turns an HTML table into a list of dicts, and one that digs a JSON object out of a script tag.

What I reach for first

Check the network tab before writing a line of Splash. Nine times out of ten the data is sitting in a JSON response, and reading that is faster to write, faster to run, and far less likely to break. Save the headless browser for the sites that genuinely build their content in the page and expose no API.

What is next

Part 3 is where sites start fighting back: anti-detection, distributed crawls with Scrapy-Redis, and the monitoring you need once a crawl runs across many machines.

Scrapy, part 1: fundamentals and your first spider

Sat, 21 Dec 2024 00:00:00 GMT

Scrapy fundamentals and your first spider

This is the first of five posts on Scrapy. By the end you will have a small project that works: a spider that walks an e-commerce catalog, item loaders that clean the data on the way in, and pipelines that check it and save it.

All the code is on GitHub: scrapy-tutorial/part-1-fundamentals. I link the exact file at the top of each section, so you can read the finished version any time the post gets into the weeds.

One thing worth saying up front. The spider points at example-store.com, which is a placeholder. The selectors show the shape of the work, but that site does not exist, so a plain scrapy crawl will not return rows. Swap the domain and selectors for a site you actually want, and it runs. The tests use sample HTML, so they pass with no network at all.

Here is the whole flow before we build any of it. A request goes in at the top, and clean data comes out the bottom.

The series

What scraping actually is

A scraper is a program that fetches web pages and pulls out the specific fields you care about. Nothing more mysterious than that. You point it at a page, tell it "the price lives in this element," and it hands you the number.

People reach for it in a few common places. Online stores want to track competitor prices, watch stock levels, or build a product catalog from several sources. Real estate teams collect listings and watch how prices move. Newsrooms aggregate stories and follow coverage of a topic. Researchers gather papers or public posts to study at scale. The technique is the same every time. Only the fields change.

The rules before the code

I have had this bite people, so read this part even though it is not code.

Check robots.txt and read the site's terms before you start. Scrape slowly enough that you are not a burden on the server, because a scraper that hammers a site is both rude and easy to block. Be careful with anything personal, and stick to data that is public. When you plan a large crawl, it is often worth a short email to the site owner first. None of this is legal advice, but it keeps you on the right side of most trouble.

Scrapy helps here. It reads robots.txt for you when you ask it to, and it can throttle itself so you do not have to babysit the request rate.

Setting up

Start with a clean virtual environment so this project's packages do not leak into anything else.

python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install --upgrade pip
pip install scrapy

Then let Scrapy lay out the project for you and add a couple of folders for output.

scrapy startproject webscraper
cd webscraper
mkdir -p data/raw data/processed logs

What Scrapy generates looks like this. The files that matter for now are items.py, pipelines.py, settings.py, and the spiders folder.

webscraper/
├── scrapy.cfg
└── webscraper/
    ├── items.py          # what a record looks like
    ├── pipelines.py      # what happens after extraction
    ├── settings.py       # how the crawl behaves
    └── spiders/          # the crawlers themselves

Say what you want: items

Full file: webscraper/items.py

An item is the shape of one record. Defining it up front keeps your fields consistent and gives you one place to clean each value. The nice part of Scrapy items is the processors. MapCompose runs a cleaning function on the way in, and TakeFirst grabs the first non-empty match so you do not carry lists around.

import scrapy
from itemloaders.processors import TakeFirst, MapCompose
from w3lib.html import remove_tags


def clean_price(value):
    if value:
        cleaned = ''.join(c for c in value if c.isdigit() or c == '.')
        try:
            return float(cleaned)
        except ValueError:
            return None
    return None


class ProductItem(scrapy.Item):
    name = scrapy.Field(output_processor=TakeFirst())
    price = scrapy.Field(
        input_processor=MapCompose(clean_price),
        output_processor=TakeFirst(),
    )
    url = scrapy.Field(output_processor=TakeFirst())

The full file carries the rest of the fields, like brand, rating, images, and a timestamp. The pattern repeats, so once you have seen three fields you have seen them all.

The spider

Full file: webscraper/spiders/ecommerce_spider.py

The spider does two jobs. It reads a category page and collects the links to each product, and it reads a product page and pulls out the fields. Splitting those into two methods keeps each one easy to follow.

def parse(self, response):
    for link in response.css('.product-item a::attr(href)').getall():
        yield response.follow(link, callback=self.parse_product)

    next_page = response.css('.pagination .next::attr(href)').get()
    if next_page:
        yield response.follow(next_page, callback=self.parse)

parse follows every product link, then looks for a "next" link and follows that too. That second yield is the whole of pagination. Scrapy keeps calling parse on each new page until there is no next link left.

The product page is where the extraction happens. Two habits here have saved me more times than I can count.

The first is fallback selectors. Sites change their markup, and often more than one layout is live at once. So I give the loader more than one selector for the important fields and let it take the first one that hits.

loader = ItemLoader(item=ProductItem(), response=response)
loader.add_css('name', 'h1.product-title::text')
loader.add_css('name', '.product-name::text')     # fallback
loader.add_css('price', '.price-current::text')
loader.add_css('price', '.current-price::text')   # fallback

The second is JSON-LD. A lot of stores ship a clean block of structured data in the page for search engines. When it is there, it is far more reliable than reading text out of divs, so the spider reads it and fills in anything the selectors missed.

scripts = response.xpath('//script[@type="application/ld+json"]/text()').getall()

Clean it on the way in: item loaders

The loader is the bridge between the page and the item. You feed it selectors, and the item's processors clean each value as it arrives. That means your spider stays about "where is the data" and the cleaning lives in one place instead of being smeared across the parse method. By the time you call loader.load_item(), the record is already tidy.

Pipelines: check, dedupe, store

Full file: webscraper/pipelines.py

Every item the spider yields runs through the pipelines in order. This is where you catch bad records before they reach your files. The project has four small pipelines, each with one job.

class ValidationPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        for field in ('name', 'url'):
            if not adapter.get(field):
                raise DropItem(f'Missing required field: {field}')
        return item

The first checks that the required fields are there and drops the record if they are not. The next filters out duplicates using the name and URL. The third writes the results to a JSON file. The last counts what came through and prints a short summary when the crawl ends, which is handy for spotting a run that quietly scraped nothing.

You switch them on in settings.py. The numbers set the order, low to high.

ITEM_PIPELINES = {
    'webscraper.pipelines.ValidationPipeline': 300,
    'webscraper.pipelines.DuplicationFilterPipeline': 400,
    'webscraper.pipelines.JsonWriterPipeline': 500,
    'webscraper.pipelines.StatisticsPipeline': 600,
}

Settings worth knowing

Full file: webscraper/settings.py

Most of the defaults are fine. A few settings are worth setting on purpose from day one.

ROBOTSTXT_OBEY = True
AUTOTHROTTLE_ENABLED = True     # slow down when the site does
DOWNLOAD_DELAY = 1              # a floor between requests
RETRY_TIMES = 3                 # retry the usual transient failures

AutoThrottle is the one I would not skip. It watches how fast the site responds and backs off on its own, which keeps you polite without hand-tuning delays.

Running it

scrapy crawl ecommerce                 # run it
scrapy crawl ecommerce -o products.json  # run and dump to a file
scrapy crawl ecommerce -L DEBUG        # run with loud logging

There is also a small run_spider.py if you would rather start the crawl from Python and pass settings in code.

Test it without hitting the network

Full file: tests/test_ecommerce_spider.py

You do not need a live site to test parsing. Build a fake response from a string of HTML, run it through the spider, and check what comes out. These tests are fast and they do not depend on a site staying up.

def test_parse_product(self):
    html = '<h1 class="product-title">Test Product</h1><span class="price-current">$99.99</span>'
    response = HtmlResponse(url='http://example.com/p/1', body=html.encode())
    item = list(self.spider.parse_product(response))[0]
    assert item['name'] == 'Test Product'
    assert item['price'] == 99.99

Notice the price comes out as the number 99.99, not the string $99.99. That is clean_price from the item doing its job before the value ever lands in the record.

A few things I learned the hard way

Give your important fields a fallback selector. The extra line costs you nothing and saves a 2am fix when the site ships a redesign.

Read JSON-LD when it is there. It is cleaner than scraping visible text, and it tends to survive layout changes that break your selectors.

Turn on AutoThrottle and keep a delay. A fast scraper that gets you blocked is slower than a polite one that keeps running.

Watch the item count at the end of a run. A crawl that finishes with zero items is usually a changed selector, not an empty site.

What is next

Part 2 moves past the basics: pages that need JavaScript, logging in, forms, and the AJAX calls behind dynamic content.

If you want to try this yourself first, clone the repo, point the spider at a simple store you like, and get the name and price out. That one exercise teaches more than reading ever will.