From a scraper to a monitored data pipeline

Most scraping projects end the same way. The script works on delivery day, the client is happy, and three weeks later someone notices the dashboard hasn’t changed since the 4th. Nobody got an error. The scraper ran, found nothing, and exited cleanly.

The scraper wasn’t the problem. Everything around it was missing. This post covers the parts I add to every scraper before I call it done: scheduling, alerting, deduplication, change history, and a clean handoff.

Why scrapers die in week two

Scrapers rarely crash loudly. They break in a few quiet ways:

Layout changes. A class name changes, the selector matches nothing, and the run returns zero rows with exit code 0.
Soft blocks. Cloudflare or DataDome serves a challenge page with HTTP 200. Your parser reads the challenge page, finds no products, and moves on.
Burned proxies. A proxy pool that worked in testing gets flagged. Success rate drops from 98% to 40%, and the dataset quietly shrinks.
Duplicates. The same listing gets scraped from page 3 today and page 4 tomorrow. Counts go up, but the data doesn’t.

None of these throw an exception. So “it didn’t error” tells you nothing about whether the data is right.

Scheduling: cron is fine until it isn’t

For one scraper on one server, cron is fine. Don’t reach for Airflow on day one.

# every day at 02:15, log output, never overlap runs
15 2 * * * flock -n /tmp/prices.lock /app/run.sh >> /var/log/prices.log 2>&1

flock -n matters more than it looks. Without it, a slow run overlaps the next one and you get two scrapers fighting over the same proxy pool.

Cron stops being enough when you need:

Retries with backoff for one failed source, without rerunning all of them
Backfills, such as “rerun last Tuesday for source B”
Dependencies, such as “load to the warehouse only after all 12 sources finish”
A place to see history of what ran, how long it took and what failed

That’s the point to move to an orchestrator like Airflow. Moving is easy if each source is already its own command with a date argument, like run.sh --source b --date 2026-09-30. Design for that from day one, even on cron.

Alert on zero rows, not on exceptions

Exceptions are the failures you already handle. The dangerous ones are silent, so check the output, not the exit code.

After every run I record a few numbers per source and compare them with recent history:

def check_run(source: str, rows: int, recent: list[int]) -> list[str]:
    problems = []
    if rows == 0:
        problems.append(f"{source}: zero rows")
    elif recent and rows < 0.5 * (sum(recent) / len(recent)):
        problems.append(f"{source}: {rows} rows, under half the recent average")
    return problems

Add three more checks and you catch most real failures:

Freshness: the newest record is older than expected.
Nulls: a required field (price, title, URL) is suddenly empty on many rows. That usually means a selector broke.
Block rate: the share of responses that were challenge pages or 403s. Rising block rate is the early warning before rows drop.

Send problems to a place people actually read, like Slack or email. Send nothing when everything is fine. An alert channel that posts “all good” every day gets muted within a week.

Deduplication and change history in PostgreSQL

Scraped data needs two things most scripts don’t give you: one row per real-world thing, and a record of how it changed.

Pick a natural key per source. That’s usually the site’s own ID or a normalised URL, never the scrape timestamp. Then upsert:

create table listings (
    source      text not null,
    external_id text not null,
    title       text,
    price       numeric,
    first_seen  timestamptz not null default now(),
    last_seen   timestamptz not null default now(),
    primary key (source, external_id)
);

create table listing_changes (
    source      text not null,
    external_id text not null,
    field       text not null,
    old_value   text,
    new_value   text,
    changed_at  timestamptz not null default now()
);

insert into listings (source, external_id, title, price)
values ($1, $2, $3, $4)
on conflict (source, external_id) do update
set title = excluded.title,
    price = excluded.price,
    last_seen = now();

Before the upsert, compare the incoming row with the stored one and write any differences to listing_changes. Now “what was the price on the 12th?” and “which listings disappeared this week?” are simple queries. You don’t have to rescrape.

Two things this gives you for free:

Reruns are safe. Running the same day twice updates rows instead of doubling them, so a retry can never corrupt the dataset.
Disappearances show up. A listing whose last_seen stops moving was removed or sold. For price and inventory monitoring, that’s often the most valuable signal in the data.

Handoff: the client owns the code and the accounts

A pipeline that only I can run isn’t a deliverable. At handoff the client gets:

The code in their repository, with a README that says how to run one source for one date
Proxy, server and alerting accounts in their name, or clear instructions to move them
A short video walkthrough of where the data lands, what the alerts mean, and what to do when one fires
A list of known risks, such as which sources are fragile and what changes most often

Then they choose. They run it themselves, or I keep it running as a monitored data feed, and a site change becomes my problem instead of theirs. Both are fine. What isn’t fine is a scraper nobody watches.

FAQ

Do I need Airflow for a single scraper? No. Cron with flock and output checks is enough for a few sources. Move to an orchestrator when you need retries per source, backfills or dependencies.

What should I alert on first? Zero rows and sudden drops in row count. They catch layout changes, soft blocks and proxy problems before anyone downstream notices.

How do I stop duplicate records? Use a natural key from the source (site ID or normalised URL) as the primary key, and upsert instead of insert. Reruns then update rows instead of adding new ones.

How often do scrapers need fixing? It depends on the site. Heavily protected or frequently redesigned sites need attention more often than stable ones. That’s why monitoring matters more than the first version of the code.