Blog
#airflow#antibot#data-engineering#data-extraction#devops#evals#llm#monitoring#playwright#proxies#python#reverse-engineering#scrapy#web-scraping
2026
- From a scraper to a monitored data pipeline
· data-engineering, web-scraping, monitoring
- LLM extraction that never invents a value
· llm, data-extraction, evals
- What web scraping really costs per 1,000 pages
· web-scraping, proxies, data-engineering
2025
- Scheduling Scrapy with Airflow: retries, backfills, alerts
· data-engineering, scrapy, airflow, python
- Scrapy vs Playwright vs the hidden API: how to choose
· web-scraping, scrapy, playwright, python
- How antibot systems fingerprint your scraper
· antibot, reverse-engineering, web-scraping
2024
- Scrapy, part 5: deploying to production
· web-scraping, scrapy, devops
- Scrapy, part 4: cleaning and storing the data
· web-scraping, scrapy, data-engineering
- Scrapy, part 3: staying undetected at scale
· web-scraping, scrapy, antibot, python
- Scrapy, part 2: JavaScript, forms, and dynamic pages
· web-scraping, scrapy, python
- Scrapy, part 1: fundamentals and your first spider
· web-scraping, scrapy, python