Scrapy articles

Web scraping at scale — what production actually takes

What it takes to scrape millions of pages a day: the architecture, the anti-bot arms race, and the failures that taught me each part.