Surendra Tamang

Scrapy, part 5: deploying to production

Wed, 25 Dec 2024 00:00:00 GMT

Scrapy Part 5: Production Deployment

Final part of the series. The scraper works; now it has to run unattended. This part covers packaging it with a multi-stage Docker build, running it on Kubernetes with autoscaling, wiring up a CI/CD pipeline with automated tests, monitoring it with Prometheus and Grafana, and hardening it for production.

Docker Containerization

Multi-Stage Dockerfile

# webscraper/Dockerfile
# Multi-stage build for optimized production image

# Build stage
FROM python:3.11-slim as builder

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    libxml2-dev \
    libxslt-dev \
    libffi-dev \
    libssl-dev \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Create and activate virtual environment
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install Python dependencies
COPY requirements.txt .
RUN pip install --upgrade pip && \
    pip install -r requirements.txt

# Production stage
FROM python:3.11-slim as production

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PATH="/opt/venv/bin:$PATH"

# Create non-root user
RUN groupadd -r scrapy && useradd -r -g scrapy scrapy

# Install runtime dependencies
RUN apt-get update && apt-get install -y \
    curl \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

# Copy virtual environment from builder stage
COPY --from=builder /opt/venv /opt/venv

# Create application directory
WORKDIR /app

# Copy application code
COPY --chown=scrapy:scrapy . .

# Create necessary directories
RUN mkdir -p /app/logs /app/data /app/exports && \
    chown -R scrapy:scrapy /app

# Switch to non-root user
USER scrapy

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
    CMD curl -f http://localhost:6023/ || exit 1

# Default command
CMD ["scrapy", "list"]

# Development stage
FROM production as development

USER root

# Install development dependencies
RUN pip install pytest pytest-cov black flake8 mypy

# Install debugging tools
RUN apt-get update && apt-get install -y \
    vim \
    htop \
    net-tools \
    && rm -rf /var/lib/apt/lists/*

USER scrapy

# Override default command for development
CMD ["tail", "-f", "/dev/null"]

Docker Compose for Development

# docker-compose.yml
version: '3.8'

services:
  # Main scraper service
  scraper:
    build:
      context: .
      target: development
      dockerfile: Dockerfile
    volumes:
      - .:/app
      - ./data:/app/data
      - ./logs:/app/logs
    environment:
      - SCRAPY_SETTINGS_MODULE=webscraper.settings.development
      - REDIS_URL=redis://redis:6379/0
      - MONGO_URI=mongodb://mongo:27017/scrapy_dev
      - POSTGRES_HOST=postgres
      - POSTGRES_DB=scrapy_dev
      - POSTGRES_USER=scrapy
      - POSTGRES_PASSWORD=scrapy_password
    depends_on:
      - redis
      - mongo
      - postgres
    networks:
      - scrapy-network

  # Redis for distributed scraping
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    command: redis-server --appendonly yes
    networks:
      - scrapy-network

  # MongoDB for document storage
  mongo:
    image: mongo:6
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db
    environment:
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin_password
    networks:
      - scrapy-network

  # PostgreSQL for structured data
  postgres:
    image: postgres:15
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=scrapy_dev
      - POSTGRES_USER=scrapy
      - POSTGRES_PASSWORD=scrapy_password
    networks:
      - scrapy-network

  # Elasticsearch for search and analytics
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.0
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    networks:
      - scrapy-network

  # Kibana for data visualization
  kibana:
    image: docker.elastic.co/kibana/kibana:8.8.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - scrapy-network

  # Prometheus for monitoring
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
    networks:
      - scrapy-network

  # Grafana for dashboards
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
      - ./monitoring/grafana:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus
    networks:
      - scrapy-network

  # Splash for JavaScript rendering
  splash:
    image: scrapinghub/splash:latest
    ports:
      - "8050:8050"
    command: --max-timeout=3600 --slots=5
    networks:
      - scrapy-network

volumes:
  redis-data:
  mongo-data:
  postgres-data:
  es-data:
  prometheus-data:
  grafana-data:

networks:
  scrapy-network:
    driver: bridge

Production Docker Compose

# docker-compose.prod.yml
version: '3.8'

services:
  # Load balancer
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/ssl:/etc/nginx/ssl
    depends_on:
      - scraper-coordinator
    networks:
      - scrapy-network

  # Coordinator service
  scraper-coordinator:
    build:
      context: .
      target: production
    command: ["python", "scripts/coordinator.py"]
    environment:
      - SCRAPY_SETTINGS_MODULE=webscraper.settings.production
      - REDIS_URL=redis://redis-cluster:6379/0
      - SENTRY_DSN=${SENTRY_DSN}
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
    depends_on:
      - redis-cluster
    networks:
      - scrapy-network

  # Worker services
  scraper-worker:
    build:
      context: .
      target: production
    command: ["python", "scripts/worker.py"]
    environment:
      - SCRAPY_SETTINGS_MODULE=webscraper.settings.production
      - REDIS_URL=redis://redis-cluster:6379/0
      - WORKER_ID=${HOSTNAME}
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M
    depends_on:
      - redis-cluster
      - scraper-coordinator
    networks:
      - scrapy-network

  # Redis cluster
  redis-cluster:
    image: redis:7-alpine
    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
    volumes:
      - redis-prod-data:/data
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    networks:
      - scrapy-network

  # Monitoring stack
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./monitoring/prometheus.prod.yml:/etc/prometheus/prometheus.yml
      - prometheus-prod-data:/prometheus
    deploy:
      resources:
        limits:
          cpus: '0.3'
          memory: 256M
    networks:
      - scrapy-network

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana-prod-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    deploy:
      resources:
        limits:
          cpus: '0.3'
          memory: 256M
    networks:
      - scrapy-network

volumes:
  redis-prod-data:
  prometheus-prod-data:
  grafana-prod-data:

networks:
  scrapy-network:
    driver: overlay
    attachable: true

Kubernetes Deployment

Kubernetes Manifests

# k8s/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: scrapy-production
  labels:
    name: scrapy-production

---
# k8s/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: scrapy-config
  namespace: scrapy-production
data:
  SCRAPY_SETTINGS_MODULE: "webscraper.settings.production"
  REDIS_URL: "redis://redis-service:6379/0"
  LOG_LEVEL: "INFO"
  CONCURRENT_REQUESTS: "16"
  DOWNLOAD_DELAY: "1"

---
# k8s/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: scrapy-secrets
  namespace: scrapy-production
type: Opaque
data:
  # Base64 encoded values
  POSTGRES_PASSWORD: c2NyYXB5X3Bhc3N3b3Jk  # scrapy_password
  MONGO_PASSWORD: bW9uZ29fcGFzc3dvcmQ=      # mongo_password
  SENTRY_DSN: aHR0cHM6Ly9zZW50cnkuaW8=    # https://sentry.io

---
# k8s/redis-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: scrapy-production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        ports:
        - containerPort: 6379
        command: ["redis-server"]
        args: ["--appendonly", "yes", "--maxmemory", "256mb"]
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        volumeMounts:
        - name: redis-storage
          mountPath: /data
      volumes:
      - name: redis-storage
        persistentVolumeClaim:
          claimName: redis-pvc

---
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  namespace: scrapy-production
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379
  type: ClusterIP

---
# k8s/coordinator-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scraper-coordinator
  namespace: scrapy-production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: scraper-coordinator
  template:
    metadata:
      labels:
        app: scraper-coordinator
    spec:
      containers:
      - name: coordinator
        image: your-registry/webscraper:latest
        command: ["python", "scripts/coordinator.py"]
        envFrom:
        - configMapRef:
            name: scrapy-config
        - secretRef:
            name: scrapy-secrets
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

---
# k8s/worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: scraper-worker
  namespace: scrapy-production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: scraper-worker
  template:
    metadata:
      labels:
        app: scraper-worker
    spec:
      containers:
      - name: worker
        image: your-registry/webscraper:latest
        command: ["python", "scripts/worker.py"]
        envFrom:
        - configMapRef:
            name: scrapy-config
        - secretRef:
            name: scrapy-secrets
        env:
        - name: WORKER_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
          requests:
            memory: "512Mi"
            cpu: "300m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        ports:
        - containerPort: 8081
        livenessProbe:
          httpGet:
            path: /health
            port: 8081
          initialDelaySeconds: 60
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 8081
          initialDelaySeconds: 10
          periodSeconds: 10

---
# k8s/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: scraper-worker-hpa
  namespace: scrapy-production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: scraper-worker
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60

---
# k8s/cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: scraper-job
  namespace: scrapy-production
spec:
  schedule: "0 */6 * * *"  # Every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: scraper
            image: your-registry/webscraper:latest
            command: ["scrapy", "crawl", "ecommerce"]
            envFrom:
            - configMapRef:
                name: scrapy-config
            - secretRef:
                name: scrapy-secrets
            resources:
              requests:
                memory: "1Gi"
                cpu: "500m"
              limits:
                memory: "2Gi"
                cpu: "1000m"
          restartPolicy: OnFailure
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

---
# k8s/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc
  namespace: scrapy-production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

CI/CD Pipeline

GitHub Actions Workflow

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  # Test and quality checks
  test:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379

    steps:
    - uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Cache dependencies
      uses: actions/cache@v3
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install -r requirements-dev.txt

    - name: Code formatting check
      run: |
        black --check --diff .

    - name: Linting
      run: |
        flake8 webscraper tests

    - name: Type checking
      run: |
        mypy webscraper

    - name: Security scan
      run: |
        bandit -r webscraper

    - name: Run tests
      run: |
        pytest tests/ --cov=webscraper --cov-report=xml --cov-report=html
      env:
        REDIS_URL: redis://localhost:6379/0

    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        file: ./coverage.xml

  # Build and push Docker image
  build:
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
    - uses: actions/checkout@v4

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Login to Container Registry
      uses: docker/login-action@v3
      with:
        registry: ${{ env.REGISTRY }}
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Extract metadata
      id: meta
      uses: docker/metadata-action@v5
      with:
        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
        tags: |
          type=ref,event=branch
          type=ref,event=pr
          type=sha,prefix={{branch}}-
          type=raw,value=latest,enable={{is_default_branch}}

    - name: Build and push
      uses: docker/build-push-action@v5
      with:
        context: .
        target: production
        push: true
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        cache-from: type=gha
        cache-to: type=gha,mode=max

  # Deploy to staging
  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/develop'
    environment: staging

    steps:
    - uses: actions/checkout@v4

    - name: Configure kubectl
      uses: azure/k8s-set-context@v3
      with:
        method: kubeconfig
        kubeconfig: ${{ secrets.KUBE_CONFIG_STAGING }}

    - name: Deploy to staging
      run: |
        # Update image in k8s manifests
        sed -i "s|your-registry/webscraper:latest|${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:develop|g" k8s/*.yaml
        
        # Apply manifests
        kubectl apply -f k8s/ -n scrapy-staging
        
        # Wait for rollout
        kubectl rollout status deployment/scraper-coordinator -n scrapy-staging
        kubectl rollout status deployment/scraper-worker -n scrapy-staging

    - name: Run smoke tests
      run: |
        # Wait for services to be ready
        kubectl wait --for=condition=ready pod -l app=scraper-coordinator -n scrapy-staging --timeout=300s
        
        # Run basic smoke tests
        python scripts/smoke_tests.py --environment=staging

  # Deploy to production
  deploy-production:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    environment: production

    steps:
    - uses: actions/checkout@v4

    - name: Configure kubectl
      uses: azure/k8s-set-context@v3
      with:
        method: kubeconfig
        kubeconfig: ${{ secrets.KUBE_CONFIG_PRODUCTION }}

    - name: Deploy to production
      run: |
        # Update image in k8s manifests
        sed -i "s|your-registry/webscraper:latest|${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest|g" k8s/*.yaml
        
        # Apply manifests with rolling update
        kubectl apply -f k8s/ -n scrapy-production
        
        # Wait for rollout
        kubectl rollout status deployment/scraper-coordinator -n scrapy-production --timeout=600s
        kubectl rollout status deployment/scraper-worker -n scrapy-production --timeout=600s

    - name: Verify deployment
      run: |
        # Check pod health
        kubectl get pods -n scrapy-production
        
        # Run production health checks
        python scripts/health_check.py --environment=production

    - name: Notify deployment
      uses: 8398a7/action-slack@v3
      with:
        status: ${{ job.status }}
        channel: '#deployments'
        webhook_url: ${{ secrets.SLACK_WEBHOOK }}
      if: always()

Testing Framework

# tests/test_spiders.py
import pytest
import responses
from scrapy.http import HtmlResponse, Request
from webscraper.spiders.ecommerce_spider import EcommerceSpider
from webscraper.items import ProductItem

class TestEcommerceSpider:
    
    @pytest.fixture
    def spider(self):
        return EcommerceSpider()
    
    @pytest.fixture
    def sample_product_html(self):
        return """
        <html>
        <body>
            <h1 class="product-title">Test Product</h1>
            <span class="price-current">$99.99</span>
            <div class="product-description">
                <p>This is a test product description</p>
            </div>
            <span class="brand-name">TestBrand</span>
            <span class="rating-value">4.5</span>
            <div class="stock-status">In Stock</div>
        </body>
        </html>
        """
    
    def create_response(self, html, url="http://test.com"):
        request = Request(url=url)
        return HtmlResponse(url=url, request=request, body=html.encode('utf-8'))
    
    def test_parse_product_basic(self, spider, sample_product_html):
        """Test basic product parsing"""
        response = self.create_response(sample_product_html)
        
        items = list(spider.parse_product(response))
        
        assert len(items) == 1
        item = items[0]
        
        assert item['name'] == 'Test Product'
        assert item['price'] == 99.99
        assert item['brand'] == 'TestBrand'
        assert item['rating'] == 4.5
    
    def test_parse_product_missing_fields(self, spider):
        """Test handling of missing fields"""
        html = "<html><body><h1>Product</h1></body></html>"
        response = self.create_response(html)
        
        items = list(spider.parse_product(response))
        
        assert len(items) == 1
        item = items[0]
        assert item['name'] == 'Product'
        assert 'price' not in item or item['price'] is None
    
    @responses.activate
    def test_api_integration(self, spider):
        """Test API integration"""
        # Mock API response
        responses.add(
            responses.GET,
            'http://api.test.com/products',
            json={'products': [{'id': 1, 'name': 'API Product'}]},
            status=200
        )
        
        # Test API call logic
        import requests
        response = requests.get('http://api.test.com/products')
        assert response.status_code == 200
        assert response.json()['products'][0]['name'] == 'API Product'

# tests/test_pipelines.py
import pytest
from itemadapter import ItemAdapter
from webscraper.pipelines import AdvancedValidationPipeline, ValidationError
from webscraper.items import ProductItem

class TestValidationPipeline:
    
    @pytest.fixture
    def pipeline(self):
        return AdvancedValidationPipeline()
    
    @pytest.fixture
    def valid_item(self):
        return ProductItem({
            'name': 'Test Product',
            'url': 'https://test.com/product',
            'price': 99.99,
            'rating': 4.5,
            'in_stock': True
        })
    
    def test_valid_item_passes(self, pipeline, valid_item):
        """Test that valid item passes validation"""
        result = pipeline.process_item(valid_item, None)
        assert result is not None
        assert result['validation_passed'] is True
    
    def test_missing_required_field_fails(self, pipeline):
        """Test that missing required fields cause validation failure"""
        item = ProductItem({'price': 99.99})
        
        with pytest.raises(ValidationError):
            pipeline.process_item(item, None)
    
    def test_invalid_price_type_fails(self, pipeline):
        """Test that invalid price type causes validation failure"""
        item = ProductItem({
            'name': 'Test Product',
            'url': 'https://test.com/product',
            'price': 'invalid'
        })
        
        with pytest.raises(ValidationError):
            pipeline.process_item(item, None)
    
    def test_price_cleaning(self, pipeline):
        """Test price cleaning functionality"""
        price_tests = [
            ('$99.99', 99.99),
            ('€1,234.56', 1234.56),
            ('1.234,56', 1234.56),
            ('FREE', None),
            ('', None)
        ]
        
        for input_price, expected in price_tests:
            cleaned = pipeline._clean_price(input_price)
            assert cleaned == expected

# scripts/smoke_tests.py
import requests
import time
import sys
import argparse
from typing import Dict, List

class SmokeTests:
    """Basic smoke tests for deployed environment"""
    
    def __init__(self, environment: str):
        self.environment = environment
        self.base_urls = {
            'staging': 'https://staging-scraper.yourcompany.com',
            'production': 'https://scraper.yourcompany.com'
        }
        self.base_url = self.base_urls[environment]
    
    def test_health_endpoints(self) -> bool:
        """Test health endpoints"""
        endpoints = ['/health', '/ready', '/metrics']
        
        for endpoint in endpoints:
            try:
                response = requests.get(f"{self.base_url}{endpoint}", timeout=10)
                if response.status_code != 200:
                    print(f"❌ Health check failed for {endpoint}: {response.status_code}")
                    return False
                print(f"✅ Health check passed for {endpoint}")
            except Exception as e:
                print(f"❌ Health check failed for {endpoint}: {e}")
                return False
        
        return True
    
    def test_basic_functionality(self) -> bool:
        """Test basic scraping functionality"""
        try:
            # Trigger a test spider run
            response = requests.post(
                f"{self.base_url}/api/spiders/test/start",
                json={'test_mode': True},
                timeout=30
            )
            
            if response.status_code != 200:
                print(f"❌ Failed to start test spider: {response.status_code}")
                return False
            
            job_id = response.json().get('job_id')
            
            # Check job status
            for _ in range(10):  # Wait up to 50 seconds
                status_response = requests.get(
                    f"{self.base_url}/api/jobs/{job_id}/status",
                    timeout=10
                )
                
                if status_response.status_code == 200:
                    status = status_response.json().get('status')
                    if status == 'completed':
                        print("✅ Basic functionality test passed")
                        return True
                    elif status == 'failed':
                        print("❌ Basic functionality test failed")
                        return False
                
                time.sleep(5)
            
            print("❌ Basic functionality test timed out")
            return False
            
        except Exception as e:
            print(f"❌ Basic functionality test failed: {e}")
            return False
    
    def run_all_tests(self) -> bool:
        """Run all smoke tests"""
        print(f"🚀 Running smoke tests for {self.environment} environment")
        
        tests = [
            self.test_health_endpoints,
            self.test_basic_functionality
        ]
        
        results = []
        for test in tests:
            results.append(test())
        
        success = all(results)
        
        if success:
            print("🎉 All smoke tests passed!")
        else:
            print("💥 Some smoke tests failed!")
        
        return success

def main():
    parser = argparse.ArgumentParser(description='Run smoke tests')
    parser.add_argument('--environment', required=True, 
                       choices=['staging', 'production'],
                       help='Target environment')
    args = parser.parse_args()
    
    tests = SmokeTests(args.environment)
    success = tests.run_all_tests()
    
    sys.exit(0 if success else 1)

if __name__ == "__main__":
    main()

Production Monitoring and Observability

Prometheus Metrics

# webscraper/monitoring/metrics.py
from prometheus_client import Counter, Histogram, Gauge, CollectorRegistry, push_to_gateway
import time
from functools import wraps
from typing import Callable
import logging

class ScrapingMetrics:
    """Prometheus metrics for scraping operations"""
    
    def __init__(self, pushgateway_url: str = None):
        self.registry = CollectorRegistry()
        self.pushgateway_url = pushgateway_url
        
        # Counters
        self.requests_total = Counter(
            'scrapy_requests_total',
            'Total number of requests made',
            ['spider', 'status'],
            registry=self.registry
        )
        
        self.items_scraped_total = Counter(
            'scrapy_items_scraped_total',
            'Total number of items scraped',
            ['spider', 'item_type'],
            registry=self.registry
        )
        
        self.errors_total = Counter(
            'scrapy_errors_total',
            'Total number of errors',
            ['spider', 'error_type'],
            registry=self.registry
        )
        
        # Histograms
        self.response_time = Histogram(
            'scrapy_response_time_seconds',
            'Response time for requests',
            ['spider', 'domain'],
            registry=self.registry
        )
        
        self.item_processing_time = Histogram(
            'scrapy_item_processing_time_seconds',
            'Time to process items',
            ['spider', 'pipeline'],
            registry=self.registry
        )
        
        # Gauges
        self.active_requests = Gauge(
            'scrapy_active_requests',
            'Number of active requests',
            ['spider'],
            registry=self.registry
        )
        
        self.queue_size = Gauge(
            'scrapy_queue_size',
            'Size of request queue',
            ['spider'],
            registry=self.registry
        )
        
        self.memory_usage = Gauge(
            'scrapy_memory_usage_bytes',
            'Memory usage in bytes',
            ['spider'],
            registry=self.registry
        )
    
    def record_request(self, spider: str, status: str, response_time: float = None, domain: str = None):
        """Record request metrics"""
        self.requests_total.labels(spider=spider, status=status).inc()
        
        if response_time and domain:
            self.response_time.labels(spider=spider, domain=domain).observe(response_time)
    
    def record_item(self, spider: str, item_type: str):
        """Record scraped item"""
        self.items_scraped_total.labels(spider=spider, item_type=item_type).inc()
    
    def record_error(self, spider: str, error_type: str):
        """Record error"""
        self.errors_total.labels(spider=spider, error_type=error_type).inc()
    
    def update_queue_size(self, spider: str, size: int):
        """Update queue size"""
        self.queue_size.labels(spider=spider).set(size)
    
    def update_active_requests(self, spider: str, count: int):
        """Update active requests count"""
        self.active_requests.labels(spider=spider).set(count)
    
    def update_memory_usage(self, spider: str, bytes_used: int):
        """Update memory usage"""
        self.memory_usage.labels(spider=spider).set(bytes_used)
    
    def push_metrics(self, job_name: str):
        """Push metrics to Pushgateway"""
        if self.pushgateway_url:
            try:
                push_to_gateway(
                    self.pushgateway_url, 
                    job=job_name, 
                    registry=self.registry
                )
            except Exception as e:
                logging.error(f"Failed to push metrics: {e}")

def monitor_performance(metrics: ScrapingMetrics, spider_name: str):
    """Decorator to monitor function performance"""
    def decorator(func: Callable) -> Callable:
        @wraps(func)
        def wrapper(*args, **kwargs):
            start_time = time.time()
            try:
                result = func(*args, **kwargs)
                return result
            except Exception as e:
                metrics.record_error(spider_name, type(e).__name__)
                raise
            finally:
                duration = time.time() - start_time
                metrics.item_processing_time.labels(
                    spider=spider_name, 
                    pipeline=func.__name__
                ).observe(duration)
        return wrapper
    return decorator

# Integration with Scrapy
class PrometheusStatsCollector:
    """Collect Scrapy stats and export to Prometheus"""
    
    def __init__(self, crawler):
        self.crawler = crawler
        self.metrics = ScrapingMetrics(
            pushgateway_url=crawler.settings.get('PROMETHEUS_PUSHGATEWAY_URL')
        )
        self.spider_name = None
    
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)
    
    def spider_opened(self, spider):
        self.spider_name = spider.name
        spider.logger.info(f"Prometheus metrics enabled for spider: {spider.name}")
    
    def spider_closed(self, spider, reason):
        # Push final metrics
        if self.metrics.pushgateway_url:
            self.metrics.push_metrics(f"scrapy_{spider.name}")
        
        # Log final stats
        stats = self.crawler.stats.get_stats()
        spider.logger.info(f"Final stats: {stats}")
    
    def request_scheduled(self, request, spider):
        self.metrics.update_active_requests(
            spider.name, 
            self.crawler.stats.get_value('scheduler/enqueued', 0)
        )
    
    def response_received(self, response, request, spider):
        # Record response metrics
        status = str(response.status)
        domain = response.url.split('/')[2] if '://' in response.url else 'unknown'
        
        self.metrics.record_request(
            spider=spider.name,
            status=status,
            domain=domain
        )
    
    def item_scraped(self, item, response, spider):
        # Record item metrics
        item_type = type(item).__name__
        self.metrics.record_item(spider.name, item_type)

Grafana Dashboard Configuration

{
  "dashboard": {
    "id": null,
    "title": "Scrapy Monitoring Dashboard",
    "description": "Comprehensive monitoring for Scrapy spiders",
    "tags": ["scrapy", "monitoring"],
    "timezone": "browser",
    "panels": [
      {
        "id": 1,
        "title": "Requests per Second",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(scrapy_requests_total[5m])",
            "legendFormat": "{{spider}} - {{status}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
        "yAxes": [
          {"label": "Requests/sec", "min": 0}
        ]
      },
      {
        "id": 2,
        "title": "Items Scraped per Hour",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(scrapy_items_scraped_total[1h])*3600",
            "legendFormat": "{{spider}} - {{item_type}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0}
      },
      {
        "id": 3,
        "title": "Error Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(scrapy_errors_total[5m]) / rate(scrapy_requests_total[5m]) * 100",
            "legendFormat": "{{spider}} error rate %"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
        "alert": {
          "conditions": [
            {
              "query": {"queryType": "", "refId": "A"},
              "reducer": {"type": "last", "params": []},
              "evaluator": {"params": [5], "type": "gt"}
            }
          ],
          "executionErrorState": "alerting",
          "frequency": "10s",
          "handler": 1,
          "name": "High Error Rate",
          "noDataState": "no_data"
        }
      },
      {
        "id": 4,
        "title": "Response Time Distribution",
        "type": "heatmap",
        "targets": [
          {
            "expr": "scrapy_response_time_seconds_bucket",
            "legendFormat": "{{le}}"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 8}
      },
      {
        "id": 5,
        "title": "Memory Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "scrapy_memory_usage_bytes / 1024 / 1024",
            "legendFormat": "{{spider}} Memory (MB)"
          }
        ],
        "gridPos": {"h": 8, "w": 24, "x": 0, "y": 16}
      },
      {
        "id": 6,
        "title": "Queue Size",
        "type": "graph",
        "targets": [
          {
            "expr": "scrapy_queue_size",
            "legendFormat": "{{spider}} Queue"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 24}
      },
      {
        "id": 7,
        "title": "Active Requests",
        "type": "stat",
        "targets": [
          {
            "expr": "sum(scrapy_active_requests)",
            "legendFormat": "Total Active"
          }
        ],
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 24}
      }
    ],
    "time": {"from": "now-1h", "to": "now"},
    "refresh": "5s"
  }
}

Security and Compliance

Security Hardening

# webscraper/security/security_middleware.py
import hmac
import hashlib
import time
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
from scrapy.exceptions import IgnoreRequest
import logging

class SecurityMiddleware:
    """Security middleware for production environments"""
    
    def __init__(self, settings):
        self.settings = settings
        self.logger = logging.getLogger(__name__)
        
        # Rate limiting
        self.rate_limits = {}
        self.last_request_time = {}
        
        # Security headers
        self.required_headers = {
            'User-Agent': True,
            'Accept': True,
            'Accept-Language': True
        }
        
        # Blocked patterns
        self.blocked_patterns = [
            r'.*\.exe$',
            r'.*\.zip$',
            r'.*admin.*',
            r'.*login.*',
            r'.*private.*'
        ]
    
    def process_request(self, request, spider):
        # Validate request security
        if not self._validate_request_security(request):
            raise IgnoreRequest("Request blocked by security policy")
        
        # Apply rate limiting
        self._apply_rate_limiting(request, spider)
        
        # Add security headers
        self._add_security_headers(request)
        
        return None
    
    def _validate_request_security(self, request):
        """Validate request against security policies"""
        url = request.url.lower()
        
        # Check blocked patterns
        import re
        for pattern in self.blocked_patterns:
            if re.match(pattern, url):
                self.logger.warning(f"Blocked request to: {request.url}")
                return False
        
        # Check for required headers
        for header, required in self.required_headers.items():
            if required and header not in request.headers:
                self.logger.warning(f"Missing required header: {header}")
                return False
        
        return True
    
    def _apply_rate_limiting(self, request, spider):
        """Apply rate limiting per domain"""
        from urllib.parse import urlparse
        domain = urlparse(request.url).netloc
        
        current_time = time.time()
        min_delay = self.settings.getfloat('SECURITY_MIN_DELAY', 1.0)
        
        if domain in self.last_request_time:
            time_since_last = current_time - self.last_request_time[domain]
            if time_since_last < min_delay:
                sleep_time = min_delay - time_since_last
                spider.logger.debug(f"Security rate limiting: {sleep_time:.2f}s for {domain}")
                time.sleep(sleep_time)
        
        self.last_request_time[domain] = time.time()
    
    def _add_security_headers(self, request):
        """Add security headers to requests"""
        # Add timestamp for request validation
        timestamp = str(int(time.time()))
        request.headers['X-Request-Timestamp'] = timestamp
        
        # Add security token if configured
        secret_key = self.settings.get('SECURITY_SECRET_KEY')
        if secret_key:
            signature = hmac.new(
                secret_key.encode(),
                f"{request.url}{timestamp}".encode(),
                hashlib.sha256
            ).hexdigest()
            request.headers['X-Security-Signature'] = signature

class DataEncryptionPipeline:
    """Encrypt sensitive data before storage"""
    
    def __init__(self, encryption_key):
        self.encryption_key = encryption_key
        self.sensitive_fields = ['email', 'phone', 'address', 'personal_id']
    
    @classmethod
    def from_crawler(cls, crawler):
        encryption_key = crawler.settings.get('ENCRYPTION_KEY')
        if not encryption_key:
            raise ValueError("ENCRYPTION_KEY setting is required")
        return cls(encryption_key)
    
    def process_item(self, item, spider):
        from cryptography.fernet import Fernet
        
        fernet = Fernet(self.encryption_key.encode())
        
        for field in self.sensitive_fields:
            if field in item and item[field]:
                # Encrypt sensitive data
                encrypted_data = fernet.encrypt(str(item[field]).encode())
                item[f"{field}_encrypted"] = encrypted_data.decode()
                # Remove original field
                del item[field]
        
        return item

# scripts/security_audit.py
import subprocess
import json
import sys
from typing import Dict, List

class SecurityAudit:
    """Security audit for production deployment"""
    
    def __init__(self):
        self.issues = []
    
    def audit_dependencies(self) -> bool:
        """Audit Python dependencies for known vulnerabilities"""
        try:
            result = subprocess.run(
                ['safety', 'check', '--json'],
                capture_output=True,
                text=True
            )
            
            if result.returncode != 0:
                vulnerabilities = json.loads(result.stdout)
                for vuln in vulnerabilities:
                    self.issues.append({
                        'type': 'dependency_vulnerability',
                        'severity': 'high',
                        'package': vuln['package_name'],
                        'vulnerability': vuln['vulnerability_id'],
                        'description': vuln['advisory']
                    })
                return False
            
            return True
            
        except Exception as e:
            self.issues.append({
                'type': 'audit_error',
                'severity': 'medium',
                'description': f"Failed to audit dependencies: {e}"
            })
            return False
    
    def audit_docker_image(self, image_name: str) -> bool:
        """Audit Docker image for security issues"""
        try:
            result = subprocess.run(
                ['trivy', 'image', '--format', 'json', image_name],
                capture_output=True,
                text=True
            )
            
            if result.returncode == 0:
                scan_results = json.loads(result.stdout)
                
                for result in scan_results.get('Results', []):
                    for vuln in result.get('Vulnerabilities', []):
                        if vuln.get('Severity') in ['HIGH', 'CRITICAL']:
                            self.issues.append({
                                'type': 'container_vulnerability',
                                'severity': vuln['Severity'].lower(),
                                'package': vuln.get('PkgName'),
                                'vulnerability': vuln.get('VulnerabilityID'),
                                'description': vuln.get('Description', '')
                            })
            
            return len([i for i in self.issues if i['type'] == 'container_vulnerability']) == 0
            
        except Exception as e:
            self.issues.append({
                'type': 'audit_error',
                'severity': 'medium',
                'description': f"Failed to audit Docker image: {e}"
            })
            return False
    
    def audit_kubernetes_config(self, config_path: str) -> bool:
        """Audit Kubernetes configuration"""
        security_checks = [
            self._check_non_root_user,
            self._check_resource_limits,
            self._check_security_context,
            self._check_network_policies
        ]
        
        passed = True
        for check in security_checks:
            if not check(config_path):
                passed = False
        
        return passed
    
    def _check_non_root_user(self, config_path: str) -> bool:
        """Check if containers run as non-root"""
        # Implementation for checking non-root user
        return True
    
    def _check_resource_limits(self, config_path: str) -> bool:
        """Check if resource limits are set"""
        # Implementation for checking resource limits
        return True
    
    def _check_security_context(self, config_path: str) -> bool:
        """Check security context configuration"""
        # Implementation for checking security context
        return True
    
    def _check_network_policies(self, config_path: str) -> bool:
        """Check network policies"""
        # Implementation for checking network policies
        return True
    
    def generate_report(self) -> Dict:
        """Generate security audit report"""
        severity_counts = {}
        for issue in self.issues:
            severity = issue['severity']
            severity_counts[severity] = severity_counts.get(severity, 0) + 1
        
        return {
            'total_issues': len(self.issues),
            'severity_breakdown': severity_counts,
            'issues': self.issues,
            'passed': len(self.issues) == 0
        }
    
    def run_full_audit(self, image_name: str = None, config_path: str = None) -> bool:
        """Run complete security audit"""
        print("🔒 Running security audit...")
        
        # Audit dependencies
        deps_ok = self.audit_dependencies()
        print(f"Dependencies: {'✅' if deps_ok else '❌'}")
        
        # Audit Docker image if provided
        if image_name:
            image_ok = self.audit_docker_image(image_name)
            print(f"Docker image: {'✅' if image_ok else '❌'}")
        
        # Audit Kubernetes config if provided
        if config_path:
            k8s_ok = self.audit_kubernetes_config(config_path)
            print(f"Kubernetes config: {'✅' if k8s_ok else '❌'}")
        
        # Generate report
        report = self.generate_report()
        
        if report['passed']:
            print("🎉 Security audit passed!")
        else:
            print(f"💥 Security audit failed with {report['total_issues']} issues")
            for issue in self.issues:
                print(f"  - {issue['severity'].upper()}: {issue['description']}")
        
        return report['passed']

if __name__ == "__main__":
    import argparse
    
    parser = argparse.ArgumentParser(description='Run security audit')
    parser.add_argument('--image', help='Docker image to audit')
    parser.add_argument('--config', help='Kubernetes config path to audit')
    args = parser.parse_args()
    
    audit = SecurityAudit()
    success = audit.run_full_audit(args.image, args.config)
    
    sys.exit(0 if success else 1)

Wrapping Up

That is the end of the series: setup and first spiders in Part 1, JavaScript rendering and forms in Part 2, anti-detection and distributed scaling in Part 3, data pipelines and storage in Part 4, and deployment here. When something breaks in production, and it will, the official Scrapy documentation is still the reference I reach for first.

Scrapy, part 4: cleaning and storing the data

Tue, 24 Dec 2024 00:00:00 GMT

Scrapy Part 4: Data Processing and Storage

Scraped data arrives messy: inconsistent price formats, broken URLs, duplicates, missing fields. This part builds the pipeline layer that fixes that. You will write a validation and cleaning pipeline, an enrichment pipeline for computed fields, and storage backends for MongoDB and PostgreSQL with deduplication, batch upserts, and analytics queries.

Data Cleaning and Validation

Validation Pipeline

# webscraper/pipelines.py
import re
import hashlib
import json
from datetime import datetime, timedelta
from typing import Dict, List, Optional, Any
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
import logging

class AdvancedValidationPipeline:
    """Comprehensive data validation and cleaning pipeline"""
    
    def __init__(self):
        self.logger = logging.getLogger(__name__)
        self.validation_stats = {
            'total_items': 0,
            'valid_items': 0,
            'dropped_items': 0,
            'field_fixes': 0,
            'validation_errors': {}
        }
        
        # Validation rules configuration
        self.validation_rules = {
            'required_fields': ['name', 'url'],
            'field_types': {
                'price': (int, float),
                'rating': (int, float),
                'review_count': int,
                'in_stock': bool
            },
            'field_ranges': {
                'price': (0, 1000000),
                'rating': (0, 5),
                'review_count': (0, float('inf'))
            },
            'string_patterns': {
                'email': r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$',
                'url': r'^https?:\/\/[^\s/$.?#].[^\s]*$',
                'phone': r'^[\+]?[1-9][\d]{0,15}$'
            },
            'max_lengths': {
                'name': 200,
                'description': 5000,
                'brand': 100
            }
        }
    
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        self.validation_stats['total_items'] += 1
        
        try:
            # Clean and validate all fields
            self._clean_item_fields(adapter)
            self._validate_required_fields(adapter)
            self._validate_field_types(adapter)
            self._validate_field_ranges(adapter)
            self._validate_string_patterns(adapter)
            self._validate_field_lengths(adapter)
            
            # Additional business logic validation
            self._validate_business_rules(adapter)
            
            # Generate quality score
            adapter['data_quality_score'] = self._calculate_quality_score(adapter)
            
            # Add validation metadata
            adapter['validation_timestamp'] = datetime.now().isoformat()
            adapter['validation_passed'] = True
            
            self.validation_stats['valid_items'] += 1
            return item
            
        except ValidationError as e:
            self.validation_stats['dropped_items'] += 1
            error_type = type(e).__name__
            self.validation_stats['validation_errors'][error_type] = \
                self.validation_stats['validation_errors'].get(error_type, 0) + 1
            
            self.logger.warning(f"Validation failed for item: {e}")
            raise DropItem(f"Validation error: {e}")
    
    def _clean_item_fields(self, adapter):
        """Clean and normalize field values"""
        # Clean text fields
        text_fields = ['name', 'description', 'brand', 'category']
        for field in text_fields:
            if adapter.get(field):
                cleaned = self._clean_text(adapter[field])
                if cleaned != adapter[field]:
                    adapter[field] = cleaned
                    self.validation_stats['field_fixes'] += 1
        
        # Clean price field
        if adapter.get('price'):
            cleaned_price = self._clean_price(adapter['price'])
            if cleaned_price != adapter['price']:
                adapter['price'] = cleaned_price
                self.validation_stats['field_fixes'] += 1
        
        # Clean URL fields
        url_fields = ['url', 'image_url']
        for field in url_fields:
            if adapter.get(field):
                cleaned_url = self._clean_url(adapter[field])
                if cleaned_url != adapter[field]:
                    adapter[field] = cleaned_url
                    self.validation_stats['field_fixes'] += 1
        
        # Normalize boolean fields
        bool_fields = ['in_stock', 'featured', 'on_sale']
        for field in bool_fields:
            if field in adapter:
                adapter[field] = self._normalize_boolean(adapter[field])
    
    def _clean_text(self, text: str) -> str:
        """Clean and normalize text content"""
        if not isinstance(text, str):
            text = str(text)
        
        # Remove excessive whitespace
        text = re.sub(r'\s+', ' ', text.strip())
        
        # Remove HTML tags
        text = re.sub(r'<[^>]+>', '', text)
        
        # Remove special characters that might cause issues
        text = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f-\xff]', '', text)
        
        # Normalize quotes
        text = text.replace('"', '"').replace('"', '"')
        text = text.replace(''', "'").replace(''', "'")
        
        return text
    
    def _clean_price(self, price: Any) -> Optional[float]:
        """Clean and convert price to float"""
        if price is None:
            return None
        
        if isinstance(price, (int, float)):
            return float(price)
        
        if isinstance(price, str):
            # Remove currency symbols and whitespace
            price_clean = re.sub(r'[^\d.,]', '', price)
            
            # Handle different decimal separators
            if ',' in price_clean and '.' in price_clean:
                # Assume comma is thousands separator
                price_clean = price_clean.replace(',', '')
            elif ',' in price_clean:
                # Could be decimal separator or thousands
                if price_clean.count(',') == 1 and len(price_clean.split(',')[1]) <= 2:
                    price_clean = price_clean.replace(',', '.')
                else:
                    price_clean = price_clean.replace(',', '')
            
            try:
                return float(price_clean)
            except ValueError:
                return None
        
        return None
    
    def _clean_url(self, url: str) -> str:
        """Clean and normalize URLs"""
        if not isinstance(url, str):
            return str(url)
        
        url = url.strip()
        
        # Add protocol if missing
        if url.startswith('//'):
            url = 'https:' + url
        elif not url.startswith(('http://', 'https://')):
            url = 'https://' + url
        
        return url
    
    def _normalize_boolean(self, value: Any) -> bool:
        """Normalize various boolean representations"""
        if isinstance(value, bool):
            return value
        
        if isinstance(value, str):
            value = value.lower().strip()
            return value in ('true', 'yes', '1', 'on', 'available', 'in stock')
        
        if isinstance(value, (int, float)):
            return bool(value)
        
        return False
    
    def _validate_required_fields(self, adapter):
        """Validate that required fields are present"""
        for field in self.validation_rules['required_fields']:
            if not adapter.get(field):
                raise ValidationError(f"Required field missing: {field}")
    
    def _validate_field_types(self, adapter):
        """Validate field data types"""
        for field, expected_types in self.validation_rules['field_types'].items():
            if field in adapter and adapter[field] is not None:
                if not isinstance(adapter[field], expected_types):
                    raise ValidationError(
                        f"Field {field} has invalid type: "
                        f"expected {expected_types}, got {type(adapter[field])}"
                    )
    
    def _validate_field_ranges(self, adapter):
        """Validate numeric field ranges"""
        for field, (min_val, max_val) in self.validation_rules['field_ranges'].items():
            if field in adapter and adapter[field] is not None:
                value = adapter[field]
                if not (min_val <= value <= max_val):
                    raise ValidationError(
                        f"Field {field} value {value} outside valid range [{min_val}, {max_val}]"
                    )
    
    def _validate_string_patterns(self, adapter):
        """Validate string fields against regex patterns"""
        for field, pattern in self.validation_rules['string_patterns'].items():
            if field in adapter and adapter[field] is not None:
                if not re.match(pattern, str(adapter[field])):
                    raise ValidationError(f"Field {field} doesn't match required pattern")
    
    def _validate_field_lengths(self, adapter):
        """Validate string field lengths"""
        for field, max_length in self.validation_rules['max_lengths'].items():
            if field in adapter and adapter[field] is not None:
                if len(str(adapter[field])) > max_length:
                    raise ValidationError(
                        f"Field {field} exceeds maximum length of {max_length}"
                    )
    
    def _validate_business_rules(self, adapter):
        """Validate business-specific rules"""
        # Example business rules
        
        # Price must be positive for in-stock items
        if adapter.get('in_stock') and adapter.get('price', 0) <= 0:
            raise ValidationError("In-stock items must have positive price")
        
        # Rating must be present if review_count > 0
        if adapter.get('review_count', 0) > 0 and not adapter.get('rating'):
            raise ValidationError("Items with reviews must have rating")
        
        # Discount validation
        original_price = adapter.get('original_price')
        current_price = adapter.get('price')
        if original_price and current_price:
            if current_price > original_price:
                raise ValidationError("Current price cannot exceed original price")
    
    def _calculate_quality_score(self, adapter) -> float:
        """Calculate data quality score (0-1)"""
        score = 0.0
        max_score = 0.0
        
        # Completeness score (40% of total)
        important_fields = ['name', 'price', 'description', 'brand', 'category']
        present_fields = sum(1 for field in important_fields if adapter.get(field))
        completeness = present_fields / len(important_fields)
        score += completeness * 0.4
        max_score += 0.4
        
        # Accuracy score (30% of total)
        accuracy = 1.0  # Assume high accuracy if validation passed
        score += accuracy * 0.3
        max_score += 0.3
        
        # Consistency score (20% of total)
        consistency = self._check_consistency(adapter)
        score += consistency * 0.2
        max_score += 0.2
        
        # Freshness score (10% of total)
        freshness = 1.0  # Assume fresh data
        score += freshness * 0.1
        max_score += 0.1
        
        return score / max_score if max_score > 0 else 0.0
    
    def _check_consistency(self, adapter) -> float:
        """Check internal consistency of data"""
        consistency_score = 1.0
        
        # Check if price and in_stock are consistent
        if adapter.get('price', 0) <= 0 and adapter.get('in_stock', False):
            consistency_score -= 0.3
        
        # Check if rating and review_count are consistent
        rating = adapter.get('rating')
        review_count = adapter.get('review_count', 0)
        if rating and rating > 0 and review_count == 0:
            consistency_score -= 0.2
        
        return max(0.0, consistency_score)
    
    def close_spider(self, spider):
        """Log validation statistics when spider closes"""
        stats = self.validation_stats
        total = stats['total_items']
        
        self.logger.info("=== Validation Pipeline Statistics ===")
        self.logger.info(f"Total items processed: {total}")
        self.logger.info(f"Valid items: {stats['valid_items']} ({stats['valid_items']/total*100:.1f}%)")
        self.logger.info(f"Dropped items: {stats['dropped_items']} ({stats['dropped_items']/total*100:.1f}%)")
        self.logger.info(f"Field fixes applied: {stats['field_fixes']}")
        
        if stats['validation_errors']:
            self.logger.info("Validation errors by type:")
            for error_type, count in stats['validation_errors'].items():
                self.logger.info(f"  {error_type}: {count}")

class ValidationError(Exception):
    """Custom exception for validation errors"""
    pass

class DataEnrichmentPipeline:
    """Enrich items with additional computed fields"""
    
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        
        # Add computed fields
        self._add_price_analysis(adapter)
        self._add_text_analysis(adapter)
        self._add_category_classification(adapter)
        self._add_uniqueness_hash(adapter)
        
        return item
    
    def _add_price_analysis(self, adapter):
        """Add price-related computed fields"""
        current_price = adapter.get('price')
        original_price = adapter.get('original_price')
        
        if current_price and original_price:
            discount = original_price - current_price
            discount_percent = (discount / original_price) * 100
            
            adapter['discount_amount'] = discount
            adapter['discount_percentage'] = round(discount_percent, 2)
            adapter['is_on_sale'] = discount > 0
        
        # Price tier classification
        if current_price:
            if current_price < 50:
                adapter['price_tier'] = 'budget'
            elif current_price < 200:
                adapter['price_tier'] = 'mid-range'
            elif current_price < 500:
                adapter['price_tier'] = 'premium'
            else:
                adapter['price_tier'] = 'luxury'
    
    def _add_text_analysis(self, adapter):
        """Add text analysis metrics"""
        description = adapter.get('description', '')
        
        if description:
            words = description.split()
            adapter['description_word_count'] = len(words)
            adapter['description_char_count'] = len(description)
            
            # Simple sentiment analysis (in production, use proper NLP)
            positive_words = ['excellent', 'great', 'amazing', 'perfect', 'wonderful']
            negative_words = ['bad', 'terrible', 'awful', 'poor', 'disappointing']
            
            pos_count = sum(1 for word in words if word.lower() in positive_words)
            neg_count = sum(1 for word in words if word.lower() in negative_words)
            
            if pos_count > neg_count:
                adapter['sentiment'] = 'positive'
            elif neg_count > pos_count:
                adapter['sentiment'] = 'negative'
            else:
                adapter['sentiment'] = 'neutral'
    
    def _add_category_classification(self, adapter):
        """Add automatic category classification"""
        name = adapter.get('name', '').lower()
        description = adapter.get('description', '').lower()
        text = f"{name} {description}"
        
        # Simple keyword-based classification
        categories = {
            'electronics': ['phone', 'laptop', 'computer', 'tablet', 'tv', 'camera'],
            'clothing': ['shirt', 'pants', 'dress', 'shoes', 'jacket', 'jeans'],
            'home': ['furniture', 'kitchen', 'bedroom', 'bathroom', 'decor'],
            'books': ['book', 'novel', 'textbook', 'manual', 'guide'],
            'sports': ['fitness', 'exercise', 'sports', 'outdoor', 'gym']
        }
        
        scores = {}
        for category, keywords in categories.items():
            score = sum(1 for keyword in keywords if keyword in text)
            if score > 0:
                scores[category] = score
        
        if scores:
            adapter['auto_category'] = max(scores, key=scores.get)
            adapter['category_confidence'] = max(scores.values()) / len(categories[adapter['auto_category']])
    
    def _add_uniqueness_hash(self, adapter):
        """Add uniqueness hash for deduplication"""
        # Create hash from key identifying fields
        key_fields = ['name', 'brand', 'sku']
        hash_input = ''.join(str(adapter.get(field, '')) for field in key_fields)
        
        if hash_input.strip():
            adapter['uniqueness_hash'] = hashlib.md5(
                hash_input.lower().encode('utf-8')
            ).hexdigest()

Multiple Storage Backends

MongoDB Storage Pipeline

# webscraper/storage/mongodb_pipeline.py
import pymongo
from pymongo import MongoClient, UpdateOne
from itemadapter import ItemAdapter
from datetime import datetime
import logging
from typing import Dict, List

class MongoDBPipeline:
    """Advanced MongoDB storage pipeline with indexing and aggregation"""
    
    def __init__(self, mongo_uri, mongo_db, collection_name):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db
        self.collection_name = collection_name
        self.client = None
        self.db = None
        self.collection = None
        self.logger = logging.getLogger(__name__)
        
        # Batch processing
        self.batch_size = 100
        self.batch_items = []
        
        # Statistics
        self.stats = {
            'inserted': 0,
            'updated': 0,
            'duplicates': 0,
            'errors': 0
        }
    
    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get("MONGO_URI", "mongodb://localhost:27017"),
            mongo_db=crawler.settings.get("MONGO_DATABASE", "scrapy"),
            collection_name=crawler.settings.get("MONGO_COLLECTION", "items")
        )
    
    def open_spider(self, spider):
        """Initialize MongoDB connection and setup indexes"""
        self.client = MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]
        self.collection = self.db[self.collection_name]
        
        # Create indexes for performance
        self._create_indexes()
        
        self.logger.info(f"Connected to MongoDB: {self.mongo_uri}/{self.mongo_db}")
    
    def _create_indexes(self):
        """Create database indexes for optimal performance"""
        indexes = [
            # Unique index for deduplication
            ("uniqueness_hash", pymongo.ASCENDING),
            
            # Search indexes
            ("name", pymongo.TEXT),
            ("description", pymongo.TEXT),
            ("brand", pymongo.ASCENDING),
            ("category", pymongo.ASCENDING),
            
            # Filter indexes
            ("price", pymongo.ASCENDING),
            ("rating", pymongo.DESCENDING),
            ("in_stock", pymongo.ASCENDING),
            
            # Time-based indexes
            ("scraped_at", pymongo.DESCENDING),
            ("updated_at", pymongo.DESCENDING),
            
            # Compound indexes
            (("category", pymongo.ASCENDING), ("price", pymongo.ASCENDING)),
            (("brand", pymongo.ASCENDING), ("rating", pymongo.DESCENDING)),
        ]
        
        for index in indexes:
            try:
                if isinstance(index, tuple):
                    self.collection.create_index([index])
                else:
                    self.collection.create_index(index)
            except Exception as e:
                self.logger.warning(f"Failed to create index {index}: {e}")
        
        # Create text index for search
        try:
            self.collection.create_index([
                ("name", pymongo.TEXT),
                ("description", pymongo.TEXT),
                ("brand", pymongo.TEXT)
            ], name="text_search")
        except Exception as e:
            self.logger.warning(f"Failed to create text index: {e}")
    
    def process_item(self, item, spider):
        """Process item and add to batch"""
        adapter = ItemAdapter(item)
        
        # Add MongoDB-specific fields
        adapter['_scraped_at'] = datetime.utcnow()
        adapter['_spider_name'] = spider.name
        
        # Add to batch
        self.batch_items.append(dict(adapter))
        
        # Process batch if full
        if len(self.batch_items) >= self.batch_size:
            self._process_batch()
        
        return item
    
    def _process_batch(self):
        """Process batch of items with upsert operations"""
        if not self.batch_items:
            return
        
        try:
            operations = []
            
            for item in self.batch_items:
                uniqueness_hash = item.get('uniqueness_hash')
                
                if uniqueness_hash:
                    # Upsert based on uniqueness hash
                    operation = UpdateOne(
                        {'uniqueness_hash': uniqueness_hash},
                        {
                            '$set': item,
                            '$setOnInsert': {'_created_at': datetime.utcnow()},
                            '$currentDate': {'_updated_at': True}
                        },
                        upsert=True
                    )
                    operations.append(operation)
                else:
                    # Insert without duplicate check
                    self.collection.insert_one(item)
                    self.stats['inserted'] += 1
            
            if operations:
                result = self.collection.bulk_write(operations, ordered=False)
                self.stats['inserted'] += result.upserted_count
                self.stats['updated'] += result.modified_count
                self.stats['duplicates'] += len(operations) - result.upserted_count - result.modified_count
            
        except Exception as e:
            self.stats['errors'] += len(self.batch_items)
            self.logger.error(f"Error processing batch: {e}")
        
        finally:
            self.batch_items.clear()
    
    def close_spider(self, spider):
        """Process remaining items and close connection"""
        # Process remaining batch
        self._process_batch()
        
        # Log statistics
        self.logger.info("=== MongoDB Pipeline Statistics ===")
        for stat, count in self.stats.items():
            self.logger.info(f"{stat.capitalize()}: {count}")
        
        # Create aggregation pipelines for analytics
        self._create_analytics_views()
        
        # Close connection
        if self.client:
            self.client.close()
    
    def _create_analytics_views(self):
        """Create MongoDB views for analytics"""
        try:
            # Category statistics view
            self.db.create_collection("category_stats", viewOn=self.collection_name, pipeline=[
                {"$group": {
                    "_id": "$category",
                    "total_products": {"$sum": 1},
                    "avg_price": {"$avg": "$price"},
                    "avg_rating": {"$avg": "$rating"},
                    "in_stock_count": {"$sum": {"$cond": ["$in_stock", 1, 0]}}
                }},
                {"$sort": {"total_products": -1}}
            ])
            
            # Brand statistics view
            self.db.create_collection("brand_stats", viewOn=self.collection_name, pipeline=[
                {"$group": {
                    "_id": "$brand",
                    "total_products": {"$sum": 1},
                    "avg_price": {"$avg": "$price"},
                    "price_range": {
                        "min": {"$min": "$price"},
                        "max": {"$max": "$price"}
                    }
                }},
                {"$sort": {"total_products": -1}}
            ])
            
            # Daily scraping statistics
            self.db.create_collection("daily_stats", viewOn=self.collection_name, pipeline=[
                {"$group": {
                    "_id": {
                        "$dateToString": {
                            "format": "%Y-%m-%d",
                            "date": "$_scraped_at"
                        }
                    },
                    "items_scraped": {"$sum": 1},
                    "unique_brands": {"$addToSet": "$brand"},
                    "unique_categories": {"$addToSet": "$category"}
                }},
                {"$addFields": {
                    "unique_brand_count": {"$size": "$unique_brands"},
                    "unique_category_count": {"$size": "$unique_categories"}
                }},
                {"$sort": {"_id": -1}}
            ])
            
        except Exception as e:
            self.logger.warning(f"Failed to create analytics views: {e}")

class MongoDBAnalytics:
    """Advanced analytics queries for MongoDB"""
    
    def __init__(self, mongo_uri, mongo_db, collection_name):
        self.client = MongoClient(mongo_uri)
        self.db = self.client[mongo_db]
        self.collection = self.db[collection_name]
    
    def get_price_trends(self, days: int = 30) -> List[Dict]:
        """Get price trends over time"""
        pipeline = [
            {
                "$match": {
                    "_scraped_at": {
                        "$gte": datetime.utcnow() - timedelta(days=days)
                    }
                }
            },
            {
                "$group": {
                    "_id": {
                        "date": {"$dateToString": {"format": "%Y-%m-%d", "date": "$_scraped_at"}},
                        "category": "$category"
                    },
                    "avg_price": {"$avg": "$price"},
                    "min_price": {"$min": "$price"},
                    "max_price": {"$max": "$price"},
                    "product_count": {"$sum": 1}
                }
            },
            {"$sort": {"_id.date": 1}}
        ]
        
        return list(self.collection.aggregate(pipeline))
    
    def get_top_products(self, category: str = None, limit: int = 10) -> List[Dict]:
        """Get top-rated products"""
        match_stage = {"rating": {"$exists": True, "$gte": 4.0}}
        if category:
            match_stage["category"] = category
        
        pipeline = [
            {"$match": match_stage},
            {"$sort": {"rating": -1, "review_count": -1}},
            {"$limit": limit},
            {
                "$project": {
                    "name": 1,
                    "brand": 1,
                    "price": 1,
                    "rating": 1,
                    "review_count": 1,
                    "url": 1
                }
            }
        ]
        
        return list(self.collection.aggregate(pipeline))
    
    def get_inventory_alerts(self) -> List[Dict]:
        """Get products that went out of stock recently"""
        pipeline = [
            {
                "$match": {
                    "in_stock": False,
                    "_scraped_at": {"$gte": datetime.utcnow() - timedelta(days=1)}
                }
            },
            {
                "$project": {
                    "name": 1,
                    "brand": 1,
                    "price": 1,
                    "category": 1,
                    "_scraped_at": 1
                }
            },
            {"$sort": {"_scraped_at": -1}}
        ]
        
        return list(self.collection.aggregate(pipeline))

PostgreSQL Storage Pipeline

# webscraper/storage/postgresql_pipeline.py
import psycopg2
import psycopg2.extras
from itemadapter import ItemAdapter
from datetime import datetime
import logging
import json
from typing import Dict, List, Optional

class PostgreSQLPipeline:
    """Advanced PostgreSQL storage with JSONB and full-text search"""
    
    def __init__(self, postgres_settings):
        self.postgres_settings = postgres_settings
        self.connection = None
        self.cursor = None
        self.logger = logging.getLogger(__name__)
        
        # Batch processing
        self.batch_size = 50
        self.batch_items = []
        
        # Statistics
        self.stats = {
            'inserted': 0,
            'updated': 0,
            'errors': 0
        }
    
    @classmethod
    def from_crawler(cls, crawler):
        postgres_settings = crawler.settings.getdict("POSTGRES_SETTINGS")
        return cls(postgres_settings)
    
    def open_spider(self, spider):
        """Initialize PostgreSQL connection and setup tables"""
        try:
            self.connection = psycopg2.connect(**self.postgres_settings)
            self.cursor = self.connection.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
            
            # Create tables and indexes
            self._create_tables()
            self._create_indexes()
            
            self.logger.info("Connected to PostgreSQL successfully")
            
        except Exception as e:
            self.logger.error(f"Error connecting to PostgreSQL: {e}")
            raise
    
    def _create_tables(self):
        """Create database tables"""
        create_tables_sql = """
        -- Main products table
        CREATE TABLE IF NOT EXISTS products (
            id SERIAL PRIMARY KEY,
            uniqueness_hash VARCHAR(32) UNIQUE,
            name VARCHAR(500) NOT NULL,
            brand VARCHAR(100),
            category VARCHAR(100),
            price DECIMAL(10,2),
            original_price DECIMAL(10,2),
            currency VARCHAR(3) DEFAULT 'USD',
            rating DECIMAL(3,2),
            review_count INTEGER,
            in_stock BOOLEAN DEFAULT true,
            description TEXT,
            specifications JSONB,
            images JSONB,
            url VARCHAR(1000),
            source_spider VARCHAR(50),
            data_quality_score DECIMAL(3,2),
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
            updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
            raw_data JSONB
        );
        
        -- Price history table
        CREATE TABLE IF NOT EXISTS price_history (
            id SERIAL PRIMARY KEY,
            product_id INTEGER REFERENCES products(id),
            price DECIMAL(10,2) NOT NULL,
            original_price DECIMAL(10,2),
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );
        
        -- Categories table
        CREATE TABLE IF NOT EXISTS categories (
            id SERIAL PRIMARY KEY,
            name VARCHAR(100) UNIQUE NOT NULL,
            parent_category VARCHAR(100),
            description TEXT
        );
        
        -- Brands table
        CREATE TABLE IF NOT EXISTS brands (
            id SERIAL PRIMARY KEY,
            name VARCHAR(100) UNIQUE NOT NULL,
            website VARCHAR(255),
            description TEXT
        );
        
        -- Search statistics table
        CREATE TABLE IF NOT EXISTS search_stats (
            id SERIAL PRIMARY KEY,
            search_term VARCHAR(255),
            results_count INTEGER,
            avg_price DECIMAL(10,2),
            searched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );
        """
        
        try:
            self.cursor.execute(create_tables_sql)
            self.connection.commit()
            self.logger.info("Database tables created successfully")
        except Exception as e:
            self.logger.error(f"Error creating tables: {e}")
            self.connection.rollback()
    
    def _create_indexes(self):
        """Create database indexes"""
        indexes_sql = [
            "CREATE INDEX IF NOT EXISTS idx_products_name ON products USING gin(to_tsvector('english', name));",
            "CREATE INDEX IF NOT EXISTS idx_products_brand ON products(brand);",
            "CREATE INDEX IF NOT EXISTS idx_products_category ON products(category);",
            "CREATE INDEX IF NOT EXISTS idx_products_price ON products(price);",
            "CREATE INDEX IF NOT EXISTS idx_products_rating ON products(rating DESC);",
            "CREATE INDEX IF NOT EXISTS idx_products_scraped_at ON products(scraped_at DESC);",
            "CREATE INDEX IF NOT EXISTS idx_products_in_stock ON products(in_stock);",
            "CREATE INDEX IF NOT EXISTS idx_products_data_quality ON products(data_quality_score DESC);",
            "CREATE INDEX IF NOT EXISTS idx_price_history_product_id ON price_history(product_id);",
            "CREATE INDEX IF NOT EXISTS idx_price_history_scraped_at ON price_history(scraped_at DESC);",
            
            # JSONB indexes
            "CREATE INDEX IF NOT EXISTS idx_products_specs ON products USING gin(specifications);",
            "CREATE INDEX IF NOT EXISTS idx_products_raw_data ON products USING gin(raw_data);",
            
            # Full-text search index
            "CREATE INDEX IF NOT EXISTS idx_products_fulltext ON products USING gin(to_tsvector('english', name || ' ' || COALESCE(description, '')));"
        ]
        
        for index_sql in indexes_sql:
            try:
                self.cursor.execute(index_sql)
                self.connection.commit()
            except Exception as e:
                self.logger.warning(f"Error creating index: {e}")
                self.connection.rollback()
    
    def process_item(self, item, spider):
        """Process item and add to batch"""
        adapter = ItemAdapter(item)
        
        # Prepare item for PostgreSQL
        postgres_item = self._prepare_item(adapter, spider)
        self.batch_items.append(postgres_item)
        
        # Process batch if full
        if len(self.batch_items) >= self.batch_size:
            self._process_batch()
        
        return item
    
    def _prepare_item(self, adapter: ItemAdapter, spider) -> Dict:
        """Prepare item for PostgreSQL storage"""
        return {
            'uniqueness_hash': adapter.get('uniqueness_hash'),
            'name': adapter.get('name'),
            'brand': adapter.get('brand'),
            'category': adapter.get('category'),
            'price': adapter.get('price'),
            'original_price': adapter.get('original_price'),
            'currency': adapter.get('currency', 'USD'),
            'rating': adapter.get('rating'),
            'review_count': adapter.get('review_count'),
            'in_stock': adapter.get('in_stock', True),
            'description': adapter.get('description'),
            'specifications': json.dumps(adapter.get('specifications', {})),
            'images': json.dumps(adapter.get('images', [])),
            'url': adapter.get('url'),
            'source_spider': spider.name,
            'data_quality_score': adapter.get('data_quality_score'),
            'raw_data': json.dumps(dict(adapter))
        }
    
    def _process_batch(self):
        """Process batch of items with upsert"""
        if not self.batch_items:
            return
        
        upsert_sql = """
        INSERT INTO products (
            uniqueness_hash, name, brand, category, price, original_price,
            currency, rating, review_count, in_stock, description,
            specifications, images, url, source_spider, data_quality_score, raw_data
        ) VALUES (
            %(uniqueness_hash)s, %(name)s, %(brand)s, %(category)s, %(price)s, %(original_price)s,
            %(currency)s, %(rating)s, %(review_count)s, %(in_stock)s, %(description)s,
            %(specifications)s, %(images)s, %(url)s, %(source_spider)s, %(data_quality_score)s, %(raw_data)s
        )
        ON CONFLICT (uniqueness_hash) DO UPDATE SET
            name = EXCLUDED.name,
            brand = EXCLUDED.brand,
            category = EXCLUDED.category,
            price = EXCLUDED.price,
            original_price = EXCLUDED.original_price,
            rating = EXCLUDED.rating,
            review_count = EXCLUDED.review_count,
            in_stock = EXCLUDED.in_stock,
            description = EXCLUDED.description,
            specifications = EXCLUDED.specifications,
            images = EXCLUDED.images,
            updated_at = CURRENT_TIMESTAMP,
            raw_data = EXCLUDED.raw_data
        RETURNING id, (xmax = 0) AS inserted;
        """
        
        try:
            # Execute batch upsert
            results = []
            for item in self.batch_items:
                self.cursor.execute(upsert_sql, item)
                result = self.cursor.fetchone()
                results.append(result)
            
            # Insert price history
            self._insert_price_history(results)
            
            self.connection.commit()
            
            # Update statistics
            for result in results:
                if result['inserted']:
                    self.stats['inserted'] += 1
                else:
                    self.stats['updated'] += 1
                    
        except Exception as e:
            self.stats['errors'] += len(self.batch_items)
            self.logger.error(f"Error processing batch: {e}")
            self.connection.rollback()
        
        finally:
            self.batch_items.clear()
    
    def _insert_price_history(self, results: List[Dict]):
        """Insert price history records"""
        price_history_sql = """
        INSERT INTO price_history (product_id, price, original_price)
        VALUES (%s, %s, %s)
        """
        
        price_history_data = []
        for i, result in enumerate(results):
            item = self.batch_items[i]
            if item.get('price'):
                price_history_data.append((
                    result['id'],
                    item['price'],
                    item.get('original_price')
                ))
        
        if price_history_data:
            self.cursor.executemany(price_history_sql, price_history_data)
    
    def close_spider(self, spider):
        """Process remaining items and close connection"""
        # Process remaining batch
        self._process_batch()
        
        # Update categories and brands tables
        self._update_reference_tables()
        
        # Log statistics
        self.logger.info("=== PostgreSQL Pipeline Statistics ===")
        for stat, count in self.stats.items():
            self.logger.info(f"{stat.capitalize()}: {count}")
        
        # Close connection
        if self.cursor:
            self.cursor.close()
        if self.connection:
            self.connection.close()
    
    def _update_reference_tables(self):
        """Update categories and brands reference tables"""
        try:
            # Update categories
            self.cursor.execute("""
                INSERT INTO categories (name)
                SELECT DISTINCT category
                FROM products
                WHERE category IS NOT NULL
                ON CONFLICT (name) DO NOTHING;
            """)
            
            # Update brands
            self.cursor.execute("""
                INSERT INTO brands (name)
                SELECT DISTINCT brand
                FROM products
                WHERE brand IS NOT NULL
                ON CONFLICT (name) DO NOTHING;
            """)
            
            self.connection.commit()
            
        except Exception as e:
            self.logger.error(f"Error updating reference tables: {e}")
            self.connection.rollback()

class PostgreSQLAnalytics:
    """Advanced analytics queries for PostgreSQL"""
    
    def __init__(self, postgres_settings):
        self.connection = psycopg2.connect(**postgres_settings)
        self.cursor = self.connection.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
    
    def search_products(self, query: str, limit: int = 50) -> List[Dict]:
        """Full-text search for products"""
        search_sql = """
        SELECT id, name, brand, category, price, rating, review_count, url,
               ts_rank(to_tsvector('english', name || ' ' || COALESCE(description, '')), 
                      plainto_tsquery('english', %s)) AS rank
        FROM products
        WHERE to_tsvector('english', name || ' ' || COALESCE(description, '')) 
              @@ plainto_tsquery('english', %s)
        ORDER BY rank DESC, rating DESC NULLS LAST
        LIMIT %s;
        """
        
        self.cursor.execute(search_sql, (query, query, limit))
        return self.cursor.fetchall()
    
    def get_price_trends(self, product_id: int, days: int = 30) -> List[Dict]:
        """Get price trends for a specific product"""
        trends_sql = """
        SELECT DATE(scraped_at) as date, 
               AVG(price) as avg_price,
               MIN(price) as min_price,
               MAX(price) as max_price,
               COUNT(*) as data_points
        FROM price_history
        WHERE product_id = %s 
          AND scraped_at >= CURRENT_DATE - INTERVAL '%s days'
        GROUP BY DATE(scraped_at)
        ORDER BY date;
        """
        
        self.cursor.execute(trends_sql, (product_id, days))
        return self.cursor.fetchall()
    
    def get_category_insights(self) -> List[Dict]:
        """Get insights by category"""
        insights_sql = """
        SELECT category,
               COUNT(*) as product_count,
               AVG(price) as avg_price,
               MIN(price) as min_price,
               MAX(price) as max_price,
               AVG(rating) as avg_rating,
               COUNT(CASE WHEN in_stock THEN 1 END) as in_stock_count,
               AVG(data_quality_score) as avg_quality_score
        FROM products
        WHERE category IS NOT NULL
        GROUP BY category
        ORDER BY product_count DESC;
        """
        
        self.cursor.execute(insights_sql)
        return self.cursor.fetchall()

Next Steps

That covers the data layer: validation, enrichment, and two storage backends with deduplication and batch upserts. Before moving on, a useful exercise is to wire both pipelines into the spider from Part 3 and check the quality scores it produces on real data. Part 5: Production Deployment covers Docker, Kubernetes, CI/CD, and monitoring.

Scrapy, part 3: staying undetected at scale

Mon, 23 Dec 2024 00:00:00 GMT

Anti-detection and scaling

This part is about running crawls that survive contact with real anti-bot systems: fingerprint-consistent headers, proxy rotation with health checks, distributed workers on Scrapy-Redis, and the monitoring and compliance layers around all of it.

What this part covers

Advanced anti-detection and stealth techniques
Distributed scraping with Scrapy-Redis
IP rotation and proxy management
Browser fingerprinting avoidance
Rate limiting and adaptive throttling
Monitoring, alerting, and health checks
Legal compliance and ethical scraping
Performance optimization and scaling

Anti-Detection Techniques

Detection systems look at far more than your user agent: request timing, header order, and fingerprint consistency all get checked. Here's how to stay under the radar:

Stealth Configuration

# webscraper/settings.py - Advanced Stealth Settings

# Respect robots.txt but with custom delay
ROBOTSTXT_OBEY = True
ROBOTSTXT_USER_AGENT = '*'

# Realistic browser behavior
DOWNLOAD_DELAY = 3  # Base delay between requests
RANDOMIZE_DOWNLOAD_DELAY = True  # 0.5 * to 1.5 * DOWNLOAD_DELAY
DOWNLOAD_TIMEOUT = 30
DOWNLOAD_MAXSIZE = 1073741824  # 1GB
DOWNLOAD_WARNSIZE = 33554432   # 32MB

# Advanced AutoThrottle settings
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 60
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0  # Conservative for stealth
AUTOTHROTTLE_DEBUG = False

# Realistic request patterns
CONCURRENT_REQUESTS = 8
CONCURRENT_REQUESTS_PER_DOMAIN = 2

# Advanced cookie handling
COOKIES_ENABLED = True
COOKIES_DEBUG = False

# Memory and resource management
MEMUSAGE_ENABLED = True
MEMUSAGE_LIMIT_MB = 2048
MEMUSAGE_WARNING_MB = 1024

# DNS timeout
DNSCACHE_ENABLED = True
DNSCACHE_SIZE = 10000
DNS_TIMEOUT = 60

# Enable telnet console for debugging (disable in production)
TELNETCONSOLE_ENABLED = False

# Custom middleware stack for stealth
DOWNLOADER_MIDDLEWARES = {
    'webscraper.middlewares.StealthUserAgentMiddleware': 400,
    'webscraper.middlewares.ProxyRotationMiddleware': 410,
    'webscraper.middlewares.HeaderSpoofingMiddleware': 420,
    'webscraper.middlewares.CookiePersistenceMiddleware': 430,
    'webscraper.middlewares.RequestTimingMiddleware': 440,
    'webscraper.middlewares.FingerprintResistanceMiddleware': 450,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 500,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,  # Disabled
}

# Anti-detection headers
DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1',
    'Cache-Control': 'max-age=0',
}

Advanced Stealth Middleware

# webscraper/middlewares.py
import random
import time
import json
import hashlib
from datetime import datetime, timedelta
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
from scrapy import signals
from scrapy.exceptions import IgnoreRequest

class StealthUserAgentMiddleware(UserAgentMiddleware):
    """Advanced user agent rotation with browser consistency"""
    
    def __init__(self):
        # Browser families with consistent headers
        self.browser_profiles = {
            'chrome': {
                'user_agents': [
                    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
                    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
                ],
                'sec_ch_ua': '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
                'sec_ch_ua_mobile': '?0',
                'sec_ch_ua_platform': '"Windows"',
            },
            'firefox': {
                'user_agents': [
                    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0',
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0',
                    'Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0',
                ],
            },
            'safari': {
                'user_agents': [
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15',
                    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15',
                ],
            }
        }
        self.current_profile = None
        self.session_ua = None
    
    def process_request(self, request, spider):
        # Maintain consistent user agent per session
        if not self.session_ua:
            browser = random.choice(list(self.browser_profiles.keys()))
            self.current_profile = self.browser_profiles[browser]
            self.session_ua = random.choice(self.current_profile['user_agents'])
        
        request.headers['User-Agent'] = self.session_ua
        
        # Add browser-specific headers
        if 'chrome' in self.session_ua.lower():
            profile = self.browser_profiles['chrome']
            request.headers['sec-ch-ua'] = profile['sec_ch_ua']
            request.headers['sec-ch-ua-mobile'] = profile['sec_ch_ua_mobile']
            request.headers['sec-ch-ua-platform'] = profile['sec_ch_ua_platform']

class HeaderSpoofingMiddleware:
    """Spoof headers to mimic real browser behavior"""
    
    def __init__(self):
        self.session_id = hashlib.md5(str(time.time()).encode()).hexdigest()[:16]
        self.request_count = 0
    
    def process_request(self, request, spider):
        self.request_count += 1
        
        # Add realistic timing headers
        request.headers['X-Request-ID'] = f"{self.session_id}-{self.request_count}"
        
        # Randomize header order
        headers_to_randomize = [
            'Accept-Encoding',
            'Accept-Language', 
            'Cache-Control',
            'Connection',
            'DNT',
            'Upgrade-Insecure-Requests'
        ]
        
        # Occasionally omit some headers to appear more human
        if random.random() < 0.1:  # 10% chance
            header_to_remove = random.choice(headers_to_randomize)
            if header_to_remove in request.headers:
                del request.headers[header_to_remove]
        
        # Add referer for non-start URLs
        if not request.meta.get('is_start_url', False):
            if 'Referer' not in request.headers:
                # Use the domain as referer
                domain = request.url.split('/')[2]
                request.headers['Referer'] = f"https://{domain}/"

class RequestTimingMiddleware:
    """Implement realistic request timing patterns"""
    
    def __init__(self):
        self.last_request_time = {}
        self.human_patterns = [
            # Reading pattern: quick succession then pause
            [0.5, 0.3, 0.8, 5.0],
            # Browsing pattern: varied intervals
            [1.2, 2.1, 0.7, 3.5, 1.8],
            # Search pattern: quick then slow
            [0.4, 0.6, 8.0, 2.0],
        ]
        self.current_pattern = []
        self.pattern_index = 0
    
    def process_request(self, request, spider):
        domain = request.url.split('/')[2]
        current_time = time.time()
        
        # Get or initialize timing for this domain
        if domain not in self.last_request_time:
            self.last_request_time[domain] = current_time
            return
        
        # Calculate time since last request to this domain
        time_since_last = current_time - self.last_request_time[domain]
        
        # Choose a human-like delay pattern
        if not self.current_pattern:
            self.current_pattern = random.choice(self.human_patterns).copy()
            self.pattern_index = 0
        
        target_delay = self.current_pattern[self.pattern_index]
        self.pattern_index = (self.pattern_index + 1) % len(self.current_pattern)
        
        # Apply delay if needed
        if time_since_last < target_delay:
            sleep_time = target_delay - time_since_last
            spider.logger.debug(f'Applying human-like delay: {sleep_time:.2f}s for {domain}')
            time.sleep(sleep_time)
        
        self.last_request_time[domain] = time.time()

class FingerprintResistanceMiddleware:
    """Resist browser fingerprinting attempts"""
    
    def process_request(self, request, spider):
        # Randomize Accept header slightly
        accept_variations = [
            'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
            'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        ]
        
        if random.random() < 0.3:  # 30% chance to vary
            request.headers['Accept'] = random.choice(accept_variations)
        
        # Randomize Accept-Language
        lang_variations = [
            'en-US,en;q=0.9',
            'en-US,en;q=0.8',
            'en-US,en;q=0.9,es;q=0.8',
            'en-US,en;q=0.5',
        ]
        
        if random.random() < 0.2:  # 20% chance to vary
            request.headers['Accept-Language'] = random.choice(lang_variations)

class CookiePersistenceMiddleware:
    """Advanced cookie management for session consistency"""
    
    def __init__(self):
        self.session_cookies = {}
    
    def process_response(self, request, response, spider):
        # Store cookies for this domain
        domain = request.url.split('/')[2]
        cookies = response.headers.getlist('Set-Cookie')
        
        if cookies:
            if domain not in self.session_cookies:
                self.session_cookies[domain] = {}
            
            for cookie in cookies:
                cookie_str = cookie.decode('utf-8')
                if '=' in cookie_str:
                    name, value = cookie_str.split('=', 1)
                    # Store only the name=value part
                    value = value.split(';')[0]
                    self.session_cookies[domain][name] = value
        
        return response
    
    def process_request(self, request, spider):
        domain = request.url.split('/')[2]
        
        # Add stored cookies for this domain
        if domain in self.session_cookies:
            cookie_header = '; '.join([
                f"{name}={value}" 
                for name, value in self.session_cookies[domain].items()
            ])
            if cookie_header:
                request.headers['Cookie'] = cookie_header

Proxy Rotation System

# webscraper/proxy_manager.py
import random
import time
import requests
from typing import List, Dict, Optional
import threading
from dataclasses import dataclass
from enum import Enum

class ProxyStatus(Enum):
    ACTIVE = "active"
    FAILED = "failed"
    RATE_LIMITED = "rate_limited"
    TESTING = "testing"

@dataclass
class ProxyInfo:
    host: str
    port: int
    username: Optional[str] = None
    password: Optional[str] = None
    protocol: str = "http"
    status: ProxyStatus = ProxyStatus.TESTING
    success_count: int = 0
    failure_count: int = 0
    last_used: Optional[float] = None
    response_time: Optional[float] = None
    rate_limit_until: Optional[float] = None
    
    @property
    def url(self) -> str:
        if self.username and self.password:
            return f"{self.protocol}://{self.username}:{self.password}@{self.host}:{self.port}"
        return f"{self.protocol}://{self.host}:{self.port}"
    
    @property
    def success_rate(self) -> float:
        total = self.success_count + self.failure_count
        return self.success_count / total if total > 0 else 0.0
    
    def is_available(self) -> bool:
        if self.status == ProxyStatus.FAILED:
            return False
        if self.rate_limit_until and time.time() < self.rate_limit_until:
            return False
        return True

class ProxyManager:
    """Advanced proxy rotation and health management"""
    
    def __init__(self, proxy_list: List[Dict], health_check_interval: int = 300):
        self.proxies = [ProxyInfo(**proxy) for proxy in proxy_list]
        self.health_check_interval = health_check_interval
        self.last_health_check = 0
        self.lock = threading.Lock()
        
        # Start health checking thread
        self.health_thread = threading.Thread(target=self._health_check_loop, daemon=True)
        self.health_thread.start()
    
    def get_proxy(self) -> Optional[ProxyInfo]:
        """Get the best available proxy"""
        with self.lock:
            available_proxies = [p for p in self.proxies if p.is_available()]
            
            if not available_proxies:
                return None
            
            # Sort by success rate and response time
            available_proxies.sort(
                key=lambda p: (p.success_rate, -p.response_time or 0),
                reverse=True
            )
            
            # Weighted selection favoring better proxies
            weights = [max(0.1, p.success_rate) for p in available_proxies]
            proxy = random.choices(available_proxies, weights=weights)[0]
            
            proxy.last_used = time.time()
            return proxy
    
    def report_success(self, proxy: ProxyInfo, response_time: float = None):
        """Report successful proxy usage"""
        with self.lock:
            proxy.success_count += 1
            proxy.status = ProxyStatus.ACTIVE
            if response_time:
                proxy.response_time = response_time
    
    def report_failure(self, proxy: ProxyInfo, is_rate_limit: bool = False):
        """Report proxy failure"""
        with self.lock:
            proxy.failure_count += 1
            
            if is_rate_limit:
                proxy.status = ProxyStatus.RATE_LIMITED
                proxy.rate_limit_until = time.time() + 300  # 5 minute cooldown
            elif proxy.failure_count > 5:
                proxy.status = ProxyStatus.FAILED
    
    def _health_check_loop(self):
        """Background thread for proxy health checking"""
        while True:
            time.sleep(self.health_check_interval)
            self._perform_health_checks()
    
    def _perform_health_checks(self):
        """Check health of all proxies"""
        test_url = "http://httpbin.org/ip"
        
        for proxy in self.proxies:
            if proxy.status == ProxyStatus.FAILED:
                continue
                
            try:
                start_time = time.time()
                response = requests.get(
                    test_url,
                    proxies={"http": proxy.url, "https": proxy.url},
                    timeout=10
                )
                response_time = time.time() - start_time
                
                if response.status_code == 200:
                    self.report_success(proxy, response_time)
                else:
                    self.report_failure(proxy)
                    
            except Exception:
                self.report_failure(proxy)
    
    def get_stats(self) -> Dict:
        """Get proxy pool statistics"""
        with self.lock:
            total = len(self.proxies)
            active = sum(1 for p in self.proxies if p.status == ProxyStatus.ACTIVE)
            failed = sum(1 for p in self.proxies if p.status == ProxyStatus.FAILED)
            rate_limited = sum(1 for p in self.proxies if p.status == ProxyStatus.RATE_LIMITED)
            
            return {
                'total': total,
                'active': active,
                'failed': failed,
                'rate_limited': rate_limited,
                'success_rate': sum(p.success_rate for p in self.proxies) / total if total > 0 else 0
            }

# Proxy rotation middleware
class AdvancedProxyRotationMiddleware:
    """Advanced proxy rotation with health management"""
    
    def __init__(self, proxy_list):
        self.proxy_manager = ProxyManager(proxy_list)
    
    @classmethod
    def from_crawler(cls, crawler):
        proxy_list = crawler.settings.get('PROXY_LIST', [])
        return cls(proxy_list)
    
    def process_request(self, request, spider):
        proxy = self.proxy_manager.get_proxy()
        
        if proxy:
            request.meta['proxy'] = proxy.url
            request.meta['proxy_info'] = proxy
            spider.logger.debug(f'Using proxy: {proxy.host}:{proxy.port}')
        else:
            spider.logger.warning('No available proxies')
    
    def process_response(self, request, response, spider):
        proxy_info = request.meta.get('proxy_info')
        
        if proxy_info:
            if response.status == 200:
                self.proxy_manager.report_success(proxy_info)
            elif response.status == 429:  # Rate limited
                self.proxy_manager.report_failure(proxy_info, is_rate_limit=True)
            elif response.status >= 400:
                self.proxy_manager.report_failure(proxy_info)
        
        return response
    
    def process_exception(self, request, exception, spider):
        proxy_info = request.meta.get('proxy_info')
        
        if proxy_info:
            self.proxy_manager.report_failure(proxy_info)

Distributed Scraping with Scrapy-Redis

When one machine isn't enough, Scrapy-Redis coordinates multiple workers through a shared queue:

Scrapy-Redis Setup

# Install Scrapy-Redis
pip install scrapy-redis

# Start Redis server
redis-server

# Or using Docker
docker run -d -p 6379:6379 redis:alpine

Distributed Spider Configuration

# webscraper/settings.py - Redis Configuration

# Enable Scrapy-Redis
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 300,
    'webscraper.pipelines.ValidationPipeline': 400,
}

# Redis connection
REDIS_URL = 'redis://localhost:6379'

# Or with detailed configuration
REDIS_PARAMS = {
    'host': 'localhost',
    'port': 6379,
    'db': 0,
    'password': None,
}

# Scheduler configuration
SCHEDULER_PERSIST = True  # Keep scheduler data after spider closes
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.PriorityQueue'

# Request serialization
SCHEDULER_SERIALIZER = "scrapy_redis.picklecompat"

Distributed Spider

# webscraper/spiders/distributed_spider.py
from scrapy_redis.spiders import RedisSpider
from webscraper.items import ProductItem
from scrapy.loader import ItemLoader
import json

class DistributedEcommerceSpider(RedisSpider):
    """Distributed spider using Redis for coordination"""
    
    name = 'distributed_ecommerce'
    redis_key = 'distributed_ecommerce:start_urls'
    
    custom_settings = {
        'CONCURRENT_REQUESTS': 32,
        'CONCURRENT_REQUESTS_PER_DOMAIN': 16,
        'DOWNLOAD_DELAY': 1,
        'SCHEDULER_IDLE_BEFORE_CLOSE': 10,  # Wait 10 seconds before closing
    }
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.processed_count = 0
        self.error_count = 0
    
    def parse(self, response):
        """Parse category pages and product listings"""
        self.logger.info(f'Worker {self.name} processing: {response.url}')
        
        # Extract product links
        product_links = response.css('.product-item a::attr(href)').getall()
        
        for link in product_links:
            product_url = response.urljoin(link)
            yield response.follow(
                product_url,
                callback=self.parse_product,
                meta={'category_url': response.url}
            )
        
        # Extract pagination
        next_page = response.css('.pagination .next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)
        
        # Add more category pages to Redis queue
        self.discover_more_urls(response)
    
    def parse_product(self, response):
        """Parse individual product pages"""
        try:
            loader = ItemLoader(item=ProductItem(), response=response)
            
            loader.add_css('name', 'h1.product-title::text')
            loader.add_css('price', '.price-current::text')
            loader.add_css('description', '.product-description::text')
            loader.add_css('brand', '.brand-name::text')
            loader.add_css('rating', '.rating-value::text')
            
            # Add metadata
            loader.add_value('url', response.url)
            loader.add_value('category_url', response.meta.get('category_url'))
            loader.add_value('worker_id', self.name)
            loader.add_value('scraped_at', self.get_current_time())
            
            item = loader.load_item()
            self.processed_count += 1
            
            if self.processed_count % 100 == 0:
                self.logger.info(f'Worker {self.name} processed {self.processed_count} products')
            
            yield item
            
        except Exception as e:
            self.error_count += 1
            self.logger.error(f'Error parsing product {response.url}: {e}')
    
    def discover_more_urls(self, response):
        """Discover and add more URLs to the Redis queue"""
        # Find category links
        category_links = response.css('.category-nav a::attr(href)').getall()
        
        for link in category_links:
            category_url = response.urljoin(link)
            # Add to Redis queue for other workers
            self.server.lpush(
                f'{self.redis_key}:discovered',
                json.dumps({'url': category_url, 'priority': 1})
            )
    
    def get_current_time(self):
        from datetime import datetime
        return datetime.now().isoformat()
    
    def closed(self, reason):
        """Spider closing callback"""
        self.logger.info(f'Worker {self.name} closing: {reason}')
        self.logger.info(f'Processed: {self.processed_count}, Errors: {self.error_count}')

# Master coordinator spider
class CoordinatorSpider(RedisSpider):
    """Coordinator spider that feeds URLs to workers"""
    
    name = 'coordinator'
    redis_key = 'distributed_ecommerce:start_urls'
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.seed_urls()
    
    def seed_urls(self):
        """Seed initial URLs into Redis"""
        initial_urls = [
            'https://example-store.com/categories/electronics',
            'https://example-store.com/categories/clothing',
            'https://example-store.com/categories/home',
            'https://example-store.com/categories/books',
        ]
        
        for url in initial_urls:
            self.server.lpush(self.redis_key, url)
        
        self.logger.info(f'Seeded {len(initial_urls)} URLs')
    
    def parse(self, response):
        """This spider doesn't parse, it just coordinates"""
        pass

Worker Management Script

# scripts/worker_manager.py
import subprocess
import sys
import time
import signal
import json
import redis
from typing import List, Dict

class WorkerManager:
    """Manage distributed scraping workers"""
    
    def __init__(self, redis_url: str = 'redis://localhost:6379'):
        self.redis_client = redis.from_url(redis_url)
        self.workers: List[subprocess.Popen] = []
        self.running = True
        
        # Register signal handlers
        signal.signal(signal.SIGINT, self.shutdown_handler)
        signal.signal(signal.SIGTERM, self.shutdown_handler)
    
    def start_workers(self, spider_name: str, num_workers: int = 4):
        """Start multiple worker processes"""
        print(f"Starting {num_workers} workers for spider '{spider_name}'")
        
        for i in range(num_workers):
            worker_id = f"{spider_name}_worker_{i}"
            cmd = [
                'scrapy', 'crawl', spider_name,
                '-s', f'BOT_NAME={worker_id}',
                '-L', 'INFO'
            ]
            
            worker = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
                text=True
            )
            
            self.workers.append(worker)
            print(f"Started worker {worker_id} (PID: {worker.pid})")
    
    def monitor_workers(self):
        """Monitor worker health and restart if needed"""
        while self.running:
            for i, worker in enumerate(self.workers):
                if worker.poll() is not None:  # Worker has terminated
                    print(f"Worker {i} terminated with code {worker.returncode}")
                    
                    # Restart worker if it crashed
                    if worker.returncode != 0:
                        print(f"Restarting worker {i}")
                        # Implementation for restarting worker
                        pass
            
            # Monitor Redis queue sizes
            self.print_queue_stats()
            
            time.sleep(30)  # Check every 30 seconds
    
    def print_queue_stats(self):
        """Print Redis queue statistics"""
        try:
            queue_size = self.redis_client.llen('distributed_ecommerce:start_urls')
            processing = self.redis_client.scard('distributed_ecommerce:dupefilter')
            
            print(f"Queue size: {queue_size}, Processed: {processing}")
            
        except Exception as e:
            print(f"Error getting Redis stats: {e}")
    
    def shutdown_handler(self, signum, frame):
        """Handle shutdown signals"""
        print("\nShutting down workers...")
        self.running = False
        
        for i, worker in enumerate(self.workers):
            print(f"Terminating worker {i}")
            worker.terminate()
            
            # Wait for graceful shutdown
            try:
                worker.wait(timeout=10)
            except subprocess.TimeoutExpired:
                print(f"Force killing worker {i}")
                worker.kill()
        
        print("All workers shut down")
        sys.exit(0)
    
    def run(self, spider_name: str, num_workers: int = 4):
        """Main execution method"""
        self.start_workers(spider_name, num_workers)
        self.monitor_workers()

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python worker_manager.py <spider_name> [num_workers]")
        sys.exit(1)
    
    spider_name = sys.argv[1]
    num_workers = int(sys.argv[2]) if len(sys.argv) > 2 else 4
    
    manager = WorkerManager()
    manager.run(spider_name, num_workers)

Monitoring and Alerting

Monitoring System

# webscraper/monitoring.py
import time
import json
import smtplib
from datetime import datetime, timedelta
from email.mime.text import MimeText
from email.mime.multipart import MimeMultipart
from typing import Dict, List, Optional
import psutil
import redis
from dataclasses import dataclass, asdict

@dataclass
class ScrapingMetrics:
    timestamp: str
    spider_name: str
    pages_scraped: int
    items_extracted: int
    errors: int
    response_time_avg: float
    memory_usage_mb: float
    cpu_usage_percent: float
    active_requests: int
    queue_size: int
    success_rate: float

class PerformanceMonitor:
    """Monitor scraping performance and system resources"""
    
    def __init__(self, redis_url: str = 'redis://localhost:6379'):
        self.redis_client = redis.from_url(redis_url)
        self.metrics_history: List[ScrapingMetrics] = []
        self.alert_thresholds = {
            'error_rate': 0.1,      # 10% error rate
            'memory_usage': 2048,    # 2GB memory usage
            'cpu_usage': 80,         # 80% CPU usage
            'response_time': 10,     # 10 second average response time
            'queue_stall': 300,      # 5 minutes without queue progress
        }
        self.last_queue_size = 0
        self.queue_stall_start = None
    
    def collect_metrics(self, spider_name: str) -> ScrapingMetrics:
        """Collect current performance metrics"""
        # Get system metrics
        memory_usage = psutil.virtual_memory().used / 1024 / 1024  # MB
        cpu_usage = psutil.cpu_percent(interval=1)
        
        # Get Redis metrics
        try:
            queue_size = self.redis_client.llen(f'{spider_name}:start_urls')
            processed_count = self.redis_client.scard(f'{spider_name}:dupefilter')
            
            # Detect queue stall
            if queue_size == self.last_queue_size:
                if not self.queue_stall_start:
                    self.queue_stall_start = time.time()
            else:
                self.queue_stall_start = None
            
            self.last_queue_size = queue_size
            
        except Exception:
            queue_size = 0
            processed_count = 0
        
        # Create metrics object (simplified - in real implementation, 
        # you'd collect these from your spider's stats)
        metrics = ScrapingMetrics(
            timestamp=datetime.now().isoformat(),
            spider_name=spider_name,
            pages_scraped=processed_count,
            items_extracted=processed_count * 0.8,  # Estimate
            errors=processed_count * 0.05,  # Estimate
            response_time_avg=2.5,  # Would be collected from actual stats
            memory_usage_mb=memory_usage,
            cpu_usage_percent=cpu_usage,
            active_requests=queue_size,
            queue_size=queue_size,
            success_rate=0.95  # Would be calculated from actual stats
        )
        
        self.metrics_history.append(metrics)
        
        # Keep only last 24 hours of metrics
        cutoff_time = datetime.now() - timedelta(hours=24)
        self.metrics_history = [
            m for m in self.metrics_history 
            if datetime.fromisoformat(m.timestamp) > cutoff_time
        ]
        
        return metrics
    
    def check_alerts(self, metrics: ScrapingMetrics) -> List[Dict]:
        """Check metrics against alert thresholds"""
        alerts = []
        
        # Error rate alert
        error_rate = metrics.errors / max(1, metrics.pages_scraped)
        if error_rate > self.alert_thresholds['error_rate']:
            alerts.append({
                'type': 'error_rate',
                'severity': 'high',
                'message': f'High error rate: {error_rate:.2%}',
                'value': error_rate,
                'threshold': self.alert_thresholds['error_rate']
            })
        
        # Memory usage alert
        if metrics.memory_usage_mb > self.alert_thresholds['memory_usage']:
            alerts.append({
                'type': 'memory_usage',
                'severity': 'medium',
                'message': f'High memory usage: {metrics.memory_usage_mb:.1f}MB',
                'value': metrics.memory_usage_mb,
                'threshold': self.alert_thresholds['memory_usage']
            })
        
        # CPU usage alert
        if metrics.cpu_usage_percent > self.alert_thresholds['cpu_usage']:
            alerts.append({
                'type': 'cpu_usage',
                'severity': 'medium',
                'message': f'High CPU usage: {metrics.cpu_usage_percent:.1f}%',
                'value': metrics.cpu_usage_percent,
                'threshold': self.alert_thresholds['cpu_usage']
            })
        
        # Response time alert
        if metrics.response_time_avg > self.alert_thresholds['response_time']:
            alerts.append({
                'type': 'response_time',
                'severity': 'medium',
                'message': f'Slow response time: {metrics.response_time_avg:.1f}s',
                'value': metrics.response_time_avg,
                'threshold': self.alert_thresholds['response_time']
            })
        
        # Queue stall alert
        if self.queue_stall_start:
            stall_duration = time.time() - self.queue_stall_start
            if stall_duration > self.alert_thresholds['queue_stall']:
                alerts.append({
                    'type': 'queue_stall',
                    'severity': 'high',
                    'message': f'Queue stalled for {stall_duration/60:.1f} minutes',
                    'value': stall_duration,
                    'threshold': self.alert_thresholds['queue_stall']
                })
        
        return alerts
    
    def generate_report(self, hours: int = 24) -> Dict:
        """Generate performance report"""
        cutoff_time = datetime.now() - timedelta(hours=hours)
        recent_metrics = [
            m for m in self.metrics_history 
            if datetime.fromisoformat(m.timestamp) > cutoff_time
        ]
        
        if not recent_metrics:
            return {'error': 'No metrics available'}
        
        # Calculate aggregated statistics
        total_pages = sum(m.pages_scraped for m in recent_metrics)
        total_items = sum(m.items_extracted for m in recent_metrics)
        total_errors = sum(m.errors for m in recent_metrics)
        avg_response_time = sum(m.response_time_avg for m in recent_metrics) / len(recent_metrics)
        avg_memory = sum(m.memory_usage_mb for m in recent_metrics) / len(recent_metrics)
        avg_cpu = sum(m.cpu_usage_percent for m in recent_metrics) / len(recent_metrics)
        
        return {
            'period': f'Last {hours} hours',
            'total_pages_scraped': total_pages,
            'total_items_extracted': total_items,
            'total_errors': total_errors,
            'error_rate': total_errors / max(1, total_pages),
            'average_response_time': avg_response_time,
            'average_memory_usage_mb': avg_memory,
            'average_cpu_usage_percent': avg_cpu,
            'current_queue_size': recent_metrics[-1].queue_size if recent_metrics else 0,
            'metrics_count': len(recent_metrics)
        }

class AlertManager:
    """Manage and send alerts"""
    
    def __init__(self, config: Dict):
        self.config = config
        self.sent_alerts = {}  # Track sent alerts to avoid spam
        self.cooldown_period = 300  # 5 minutes cooldown between same alerts
    
    def send_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Send alert via configured channels"""
        alert_key = f"{alert['type']}_{alert.get('severity', 'medium')}"
        current_time = time.time()
        
        # Check cooldown
        if alert_key in self.sent_alerts:
            if current_time - self.sent_alerts[alert_key] < self.cooldown_period:
                return  # Skip sending due to cooldown
        
        # Send via email
        if self.config.get('email'):
            self._send_email_alert(alert, metrics)
        
        # Send via webhook
        if self.config.get('webhook'):
            self._send_webhook_alert(alert, metrics)
        
        # Log to file
        self._log_alert(alert, metrics)
        
        self.sent_alerts[alert_key] = current_time
    
    def _send_email_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Send alert via email"""
        try:
            smtp_config = self.config['email']
            
            msg = MimeMultipart()
            msg['From'] = smtp_config['from']
            msg['To'] = smtp_config['to']
            msg['Subject'] = f"Scraping Alert: {alert['type']} - {alert['severity'].upper()}"
            
            body = f"""
            Alert: {alert['message']}
            
            Spider: {metrics.spider_name}
            Time: {metrics.timestamp}
            Current Value: {alert['value']}
            Threshold: {alert['threshold']}
            
            Current Metrics:
            - Pages Scraped: {metrics.pages_scraped}
            - Items Extracted: {metrics.items_extracted}
            - Error Rate: {metrics.errors / max(1, metrics.pages_scraped):.2%}
            - Memory Usage: {metrics.memory_usage_mb:.1f}MB
            - CPU Usage: {metrics.cpu_usage_percent:.1f}%
            - Queue Size: {metrics.queue_size}
            """
            
            msg.attach(MimeText(body, 'plain'))
            
            with smtplib.SMTP(smtp_config['smtp_server'], smtp_config['smtp_port']) as server:
                if smtp_config.get('use_tls'):
                    server.starttls()
                if smtp_config.get('username'):
                    server.login(smtp_config['username'], smtp_config['password'])
                server.send_message(msg)
                
        except Exception as e:
            print(f"Failed to send email alert: {e}")
    
    def _send_webhook_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Send alert via webhook"""
        import requests
        
        try:
            webhook_config = self.config['webhook']
            
            payload = {
                'alert': alert,
                'metrics': asdict(metrics),
                'timestamp': metrics.timestamp
            }
            
            response = requests.post(
                webhook_config['url'],
                json=payload,
                headers=webhook_config.get('headers', {}),
                timeout=10
            )
            response.raise_for_status()
            
        except Exception as e:
            print(f"Failed to send webhook alert: {e}")
    
    def _log_alert(self, alert: Dict, metrics: ScrapingMetrics):
        """Log alert to file"""
        import logging
        
        logging.basicConfig(
            filename='scraping_alerts.log',
            level=logging.INFO,
            format='%(asctime)s - %(levelname)s - %(message)s'
        )
        
        logging.info(
            f"ALERT: {alert['type']} - {alert['message']} "
            f"(Spider: {metrics.spider_name}, Value: {alert['value']})"
        )

Monitoring Dashboard Script

# scripts/monitoring_dashboard.py
import time
import json
from webscraper.monitoring import PerformanceMonitor, AlertManager

def main():
    """Main monitoring loop"""
    
    # Configuration
    config = {
        'email': {
            'smtp_server': 'smtp.gmail.com',
            'smtp_port': 587,
            'use_tls': True,
            'username': 'your-email@gmail.com',
            'password': 'your-app-password',
            'from': 'your-email@gmail.com',
            'to': 'alerts@yourcompany.com'
        },
        'webhook': {
            'url': 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK',
            'headers': {'Content-Type': 'application/json'}
        }
    }
    
    monitor = PerformanceMonitor()
    alert_manager = AlertManager(config)
    
    spider_name = 'distributed_ecommerce'
    
    print("Starting monitoring dashboard...")
    print("Press Ctrl+C to stop")
    
    try:
        while True:
            # Collect metrics
            metrics = monitor.collect_metrics(spider_name)
            
            # Check for alerts
            alerts = monitor.check_alerts(metrics)
            
            # Send alerts
            for alert in alerts:
                alert_manager.send_alert(alert, metrics)
                print(f"ALERT: {alert['message']}")
            
            # Print current status
            print(f"\n[{metrics.timestamp}] Spider: {spider_name}")
            print(f"Pages: {metrics.pages_scraped}, Items: {metrics.items_extracted}")
            print(f"Errors: {metrics.errors}, Queue: {metrics.queue_size}")
            print(f"Memory: {metrics.memory_usage_mb:.1f}MB, CPU: {metrics.cpu_usage_percent:.1f}%")
            
            if alerts:
                print(f"Active alerts: {len(alerts)}")
            
            # Generate hourly report
            if int(time.time()) % 3600 == 0:  # Every hour
                report = monitor.generate_report(hours=1)
                print("\n--- Hourly Report ---")
                print(json.dumps(report, indent=2))
            
            time.sleep(60)  # Check every minute
            
    except KeyboardInterrupt:
        print("\nMonitoring stopped")

if __name__ == "__main__":
    main()

Legal Compliance and Ethics

Compliance Framework

# webscraper/compliance.py
import time
import requests
from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin, urlparse
from typing import Dict, List, Optional
import logging

class ComplianceManager:
    """Ensure legal and ethical scraping compliance"""
    
    def __init__(self):
        self.robots_cache = {}
        self.rate_limits = {}
        self.compliance_rules = {
            'respect_robots_txt': True,
            'rate_limit_default': 1.0,  # 1 second between requests
            'rate_limit_per_domain': {},
            'user_agent_required': True,
            'contact_info': 'your-email@company.com',
            'personal_data_handling': 'exclude',
            'copyright_respect': True,
        }
        
        self.logger = logging.getLogger(__name__)
    
    def check_robots_txt(self, url: str, user_agent: str = '*') -> bool:
        """Check if URL is allowed by robots.txt"""
        domain = urlparse(url).netloc
        
        if domain not in self.robots_cache:
            robots_url = urljoin(f"https://{domain}", "/robots.txt")
            
            try:
                rp = RobotFileParser()
                rp.set_url(robots_url)
                rp.read()
                self.robots_cache[domain] = rp
                
                # Extract crawl delay if specified
                delay = rp.crawl_delay(user_agent)
                if delay:
                    self.rate_limits[domain] = float(delay)
                
            except Exception as e:
                self.logger.warning(f"Could not fetch robots.txt for {domain}: {e}")
                return True  # Allow if robots.txt can't be fetched
        
        robots_parser = self.robots_cache.get(domain)
        if robots_parser:
            return robots_parser.can_fetch(user_agent, url)
        
        return True
    
    def get_rate_limit(self, domain: str) -> float:
        """Get appropriate rate limit for domain"""
        # Check domain-specific rate limit
        if domain in self.compliance_rules['rate_limit_per_domain']:
            return self.compliance_rules['rate_limit_per_domain'][domain]
        
        # Check robots.txt specified delay
        if domain in self.rate_limits:
            return self.rate_limits[domain]
        
        # Use default
        return self.compliance_rules['rate_limit_default']
    
    def validate_request(self, request) -> bool:
        """Validate request against compliance rules"""
        url = request.url
        domain = urlparse(url).netloc
        user_agent = request.headers.get('User-Agent', '')
        
        # Check robots.txt
        if self.compliance_rules['respect_robots_txt']:
            if not self.check_robots_txt(url, user_agent):
                self.logger.warning(f"Blocked by robots.txt: {url}")
                return False
        
        # Check user agent requirement
        if self.compliance_rules['user_agent_required']:
            if not user_agent or 'bot' not in user_agent.lower():
                self.logger.warning(f"Invalid user agent for compliance: {user_agent}")
                return False
        
        return True
    
    def filter_personal_data(self, item: Dict) -> Dict:
        """Filter out personal data from scraped items"""
        if self.compliance_rules['personal_data_handling'] == 'exclude':
            sensitive_fields = [
                'email', 'phone', 'address', 'ssn', 'credit_card',
                'personal_id', 'passport', 'driver_license'
            ]
            
            filtered_item = {}
            for key, value in item.items():
                if key.lower() not in sensitive_fields:
                    # Additional check for values that look like personal data
                    if not self.looks_like_personal_data(str(value)):
                        filtered_item[key] = value
                    else:
                        self.logger.info(f"Filtered potential personal data: {key}")
            
            return filtered_item
        
        return item
    
    def looks_like_personal_data(self, text: str) -> bool:
        """Heuristic check for personal data patterns"""
        import re
        
        patterns = [
            r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',  # Email
            r'\b\d{3}-\d{2}-\d{4}\b',  # SSN
            r'\b\d{3}-\d{3}-\d{4}\b',  # Phone
            r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b',  # Credit card
        ]
        
        for pattern in patterns:
            if re.search(pattern, text):
                return True
        
        return False
    
    def generate_compliance_report(self) -> Dict:
        """Generate compliance status report"""
        return {
            'robots_txt_domains_checked': len(self.robots_cache),
            'custom_rate_limits': len(self.rate_limits),
            'compliance_rules': self.compliance_rules,
            'timestamp': time.time()
        }

# Compliance middleware
class ComplianceMiddleware:
    """Scrapy middleware for compliance enforcement"""
    
    def __init__(self):
        self.compliance_manager = ComplianceManager()
        self.last_request_time = {}
    
    def process_request(self, request, spider):
        # Validate compliance
        if not self.compliance_manager.validate_request(request):
            raise IgnoreRequest(f"Request blocked by compliance rules: {request.url}")
        
        # Enforce rate limiting
        domain = urlparse(request.url).netloc
        rate_limit = self.compliance_manager.get_rate_limit(domain)
        
        if domain in self.last_request_time:
            time_since_last = time.time() - self.last_request_time[domain]
            if time_since_last < rate_limit:
                sleep_time = rate_limit - time_since_last
                spider.logger.debug(f"Rate limiting: sleeping {sleep_time:.2f}s for {domain}")
                time.sleep(sleep_time)
        
        self.last_request_time[domain] = time.time()
    
    def process_item(self, item, spider):
        # Filter personal data
        filtered_item = self.compliance_manager.filter_personal_data(dict(item))
        return filtered_item

Summary

This part covered the operational side: stealth middleware and fingerprint consistency, proxy pools with health checks, distributed workers on Scrapy-Redis, monitoring with alerting, and a compliance layer that enforces robots.txt and rate limits.

What's Next?

Part 4: Data Processing and Storage covers:

Advanced data cleaning and validation
Multiple storage backends (MongoDB, PostgreSQL, Elasticsearch)
Real-time data pipelines
Data quality monitoring
ETL processes and data warehousing

Practice Exercise

Build a distributed scraper that:

Uses anti-detection techniques to scrape a major e-commerce site
Scales across multiple workers with Redis coordination
Implements monitoring and alerting
Ensures legal compliance and ethical scraping
Handles millions of products with proper rate limiting

Scrapy, part 2: JavaScript, forms, and dynamic pages

Sun, 22 Dec 2024 00:00:00 GMT

JavaScript, forms, and dynamic pages

Part 1 covered static HTML. This part deals with sites that don't hand you clean markup: rendering JavaScript with Splash, submitting forms and holding sessions, and pulling data straight from the AJAX endpoints that actually hold it.

What this part covers

JavaScript rendering with Scrapy-Splash integration
Handling forms, logins, and session management
Extracting data from AJAX requests and APIs
Advanced selector techniques and data extraction
Custom middleware development
Handling cookies, headers, and authentication
Working with infinite scroll and pagination

JavaScript-Heavy Websites with Scrapy-Splash

Scrapy alone cannot execute JavaScript. For pages that render their content client-side, Scrapy-Splash fills the gap.

Setting Up Splash

# Install Docker (required for Splash)
# On macOS with Homebrew:
brew install docker

# Start Docker service and run Splash
docker run -p 8050:8050 scrapinghub/splash

# Install Scrapy-Splash
pip install scrapy-splash

Configuring Scrapy for Splash

# webscraper/settings.py

# Splash settings
SPLASH_URL = 'http://localhost:8050'

# Enable Splash middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

# Enable Splash spider middleware
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

# Splash duplicate filter
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

# Splash HTTP cache storage backend
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

JavaScript-Enabled Spider

# webscraper/spiders/spa_spider.py
import scrapy
from scrapy_splash import SplashRequest
from webscraper.items import ProductItem
from scrapy.loader import ItemLoader
import json

class SPASpider(scrapy.Spider):
    name = 'spa_scraper'
    allowed_domains = ['example-spa.com']
    
    custom_settings = {
        'DOWNLOAD_DELAY': 2,
        'SPLASH_URL': 'http://localhost:8050',
    }
    
    def start_requests(self):
        urls = ['https://example-spa.com/products']
        
        # Lua script for complex interactions
        lua_script = """
        function main(splash, args)
            splash.private_mode_enabled = false
            splash:go(args.url)
            splash:wait(3)
            
            -- Wait for products to load
            splash:wait_for_resume([[
                function main(splash) {
                    var products = document.querySelectorAll('.product-item');
                    if (products.length > 0) {
                        splash.resume('Products loaded');
                    } else {
                        setTimeout(function() {
                            splash.resume('Timeout');
                        }, 10000);
                    }
                }
            ]], 15)
            
            -- Scroll to load more content
            splash:runjs([[
                window.scrollTo(0, document.body.scrollHeight);
            ]])
            splash:wait(2)
            
            -- Click "Load More" button if present
            local load_more = splash:select('.load-more-btn')
            if load_more then
                load_more:click()
                splash:wait(3)
            end
            
            return {
                html = splash:html(),
                png = splash:png(),
                har = splash:har(),
                url = splash:url()
            }
        end
        """
        
        for url in urls:
            yield SplashRequest(
                url=url,
                callback=self.parse,
                args={
                    'lua_source': lua_script,
                    'timeout': 30,
                    'resource_timeout': 10,
                    'wait': 5,
                }
            )
    
    def parse(self, response):
        """Parse SPA product listings"""
        self.logger.info(f'Parsing SPA page: {response.url}')
        
        # Extract products from JavaScript-rendered content
        products = response.css('.product-item')
        
        for product in products:
            product_url = product.css('a::attr(href)').get()
            if product_url:
                # Use SplashRequest for product pages too
                yield SplashRequest(
                    url=response.urljoin(product_url),
                    callback=self.parse_product,
                    args={'wait': 3}
                )
        
        # Handle pagination in SPA
        next_page_data = response.css('script[type="application/json"]::text').get()
        if next_page_data:
            try:
                data = json.loads(next_page_data)
                if data.get('nextPage'):
                    yield SplashRequest(
                        url=data['nextPage'],
                        callback=self.parse,
                        args={'wait': 3}
                    )
            except json.JSONDecodeError:
                pass
    
    def parse_product(self, response):
        """Parse individual product from SPA"""
        loader = ItemLoader(item=ProductItem(), response=response)
        
        # Extract data that might be loaded via JavaScript
        loader.add_css('name', 'h1.product-title::text')
        loader.add_css('price', '.price-display::text')
        loader.add_css('description', '.product-description::text')
        
        # Extract from JavaScript variables
        js_data = self.extract_js_data(response)
        if js_data:
            loader.add_value('name', js_data.get('productName'))
            loader.add_value('price', js_data.get('price'))
            loader.add_value('sku', js_data.get('sku'))
        
        loader.add_value('url', response.url)
        loader.add_value('source', 'spa')
        
        yield loader.load_item()
    
    def extract_js_data(self, response):
        """Extract data from JavaScript variables"""
        # Look for common patterns
        js_patterns = [
            r'window\.productData\s*=\s*({[^}]+})',
            r'var\s+product\s*=\s*({[^}]+})',
            r'__INITIAL_STATE__\s*=\s*({.+?});'
        ]
        
        for pattern in js_patterns:
            import re
            match = re.search(pattern, response.text)
            if match:
                try:
                    return json.loads(match.group(1))
                except json.JSONDecodeError:
                    continue
        
        return None

Form Handling and Authentication

Many sites put the useful data behind a login or a search form. Here's how to handle both:

Login Spider

# webscraper/spiders/login_spider.py
import scrapy
from scrapy import FormRequest
from webscraper.items import ProductItem

class LoginSpider(scrapy.Spider):
    name = 'login_scraper'
    allowed_domains = ['secure-store.com']
    start_urls = ['https://secure-store.com/login']
    
    def parse(self, response):
        """Handle login form"""
        # Check if already logged in
        if self.is_logged_in(response):
            return self.after_login(response)
        
        # Extract form data and CSRF tokens
        csrf_token = response.css('input[name="csrf_token"]::attr(value)').get()
        
        # Submit login form
        return FormRequest.from_response(
            response,
            formdata={
                'username': 'your_username',
                'password': 'your_password',
                'csrf_token': csrf_token,
                'remember_me': '1'
            },
            callback=self.after_login,
            dont_filter=True
        )
    
    def is_logged_in(self, response):
        """Check if successfully logged in"""
        return bool(response.css('.user-dashboard'))
    
    def after_login(self, response):
        """Handle post-login logic"""
        if not self.is_logged_in(response):
            self.logger.error('Login failed')
            return
        
        self.logger.info('Successfully logged in')
        
        # Navigate to protected areas
        protected_urls = [
            'https://secure-store.com/members/products',
            'https://secure-store.com/premium/catalog'
        ]
        
        for url in protected_urls:
            yield response.follow(url, callback=self.parse_protected_content)
    
    def parse_protected_content(self, response):
        """Parse content that requires authentication"""
        products = response.css('.premium-product')
        
        for product in products:
            loader = ItemLoader(item=ProductItem(), selector=product)
            loader.add_css('name', '.product-name::text')
            loader.add_css('price', '.member-price::text')
            loader.add_css('description', '.product-desc::text')
            loader.add_value('source', 'premium')
            
            yield loader.load_item()

Complex Form Handling

# webscraper/spiders/form_spider.py
import scrapy
from scrapy import FormRequest
import json

class FormSpider(scrapy.Spider):
    name = 'form_handler'
    
    def start_requests(self):
        # Start with a search form
        yield scrapy.Request(
            'https://example.com/search',
            callback=self.parse_search_form
        )
    
    def parse_search_form(self, response):
        """Handle complex search forms"""
        # Extract all form fields and hidden values
        form_data = {}
        
        # Get all input fields
        for input_field in response.css('form input'):
            name = input_field.css('::attr(name)').get()
            value = input_field.css('::attr(value)').get()
            input_type = input_field.css('::attr(type)').get()
            
            if name:
                if input_type == 'checkbox' and not input_field.css('::attr(checked)').get():
                    continue  # Skip unchecked checkboxes
                form_data[name] = value or ''
        
        # Get select fields
        for select in response.css('form select'):
            name = select.css('::attr(name)').get()
            selected = select.css('option[selected]::attr(value)').get()
            if name:
                form_data[name] = selected or ''
        
        # Add our search parameters
        search_params = {
            'query': 'laptops',
            'category': 'electronics',
            'price_min': '500',
            'price_max': '2000',
            'sort': 'price_desc'
        }
        form_data.update(search_params)
        
        # Submit form
        yield FormRequest.from_response(
            response,
            formdata=form_data,
            callback=self.parse_search_results
        )
    
    def parse_search_results(self, response):
        """Parse search results"""
        products = response.css('.search-result-item')
        
        for product in products:
            # Extract product details
            product_data = {
                'name': product.css('.product-title::text').get(),
                'price': product.css('.price::text').get(),
                'rating': product.css('.rating::attr(data-rating)').get(),
                'url': response.urljoin(product.css('a::attr(href)').get())
            }
            
            if product_data['url']:
                yield response.follow(
                    product_data['url'],
                    callback=self.parse_product,
                    meta={'product_data': product_data}
                )

AJAX Requests and API Integration

When a site loads its data via AJAX, hitting the API directly is usually faster and more reliable than rendering the page. Here's how to find and work with those requests:

AJAX Spider

# webscraper/spiders/ajax_spider.py
import scrapy
import json
from urllib.parse import urlencode

class AjaxSpider(scrapy.Spider):
    name = 'ajax_scraper'
    allowed_domains = ['api-example.com']
    
    def start_requests(self):
        # Start with the main page to get initial data
        yield scrapy.Request(
            'https://api-example.com/products',
            callback=self.parse_initial_page
        )
    
    def parse_initial_page(self, response):
        """Extract API endpoints and initial data"""
        # Look for API endpoints in JavaScript
        api_endpoints = self.extract_api_endpoints(response)
        
        # Extract pagination info
        total_pages = response.css('.pagination::attr(data-total-pages)').get()
        if total_pages:
            total_pages = int(total_pages)
        else:
            total_pages = 10  # Default fallback
        
        # Generate API requests for all pages
        for page in range(1, total_pages + 1):
            api_url = f'https://api-example.com/api/products'
            params = {
                'page': page,
                'limit': 20,
                'format': 'json'
            }
            
            url = f"{api_url}?{urlencode(params)}"
            yield scrapy.Request(
                url=url,
                callback=self.parse_api_response,
                headers={
                    'Accept': 'application/json',
                    'X-Requested-With': 'XMLHttpRequest',
                    'Referer': response.url
                }
            )
    
    def extract_api_endpoints(self, response):
        """Extract API endpoints from JavaScript"""
        endpoints = []
        
        # Common patterns for API endpoints
        import re
        patterns = [
            r'api[\'"]:\s*[\'"]([^\'\"]+)',
            r'endpoint[\'"]:\s*[\'"]([^\'\"]+)',
            r'fetch\([\'"]([^\'\"]+)',
            r'axios\.get\([\'"]([^\'\"]+)'
        ]
        
        for pattern in patterns:
            matches = re.findall(pattern, response.text)
            endpoints.extend(matches)
        
        return list(set(endpoints))  # Remove duplicates
    
    def parse_api_response(self, response):
        """Parse JSON API responses"""
        try:
            data = json.loads(response.text)
        except json.JSONDecodeError:
            self.logger.error(f'Invalid JSON response from {response.url}')
            return
        
        # Handle different API response structures
        products = data.get('products', data.get('data', data.get('items', [])))
        
        for product in products:
            # Create item from API data
            product_item = {
                'id': product.get('id'),
                'name': product.get('name', product.get('title')),
                'price': product.get('price'),
                'description': product.get('description'),
                'category': product.get('category'),
                'images': product.get('images', []),
                'url': product.get('url'),
                'api_source': response.url
            }
            
            yield product_item
        
        # Handle API pagination
        pagination = data.get('pagination', {})
        if pagination.get('hasNextPage'):
            next_page = pagination.get('nextPage')
            if next_page:
                yield response.follow(
                    next_page,
                    callback=self.parse_api_response
                )

Real-time Data Spider

# webscraper/spiders/realtime_spider.py
import scrapy
import json
import time
from datetime import datetime

class RealtimeSpider(scrapy.Spider):
    name = 'realtime_scraper'
    
    custom_settings = {
        'DOWNLOAD_DELAY': 5,  # Respectful delay for real-time data
        'CONCURRENT_REQUESTS': 1,  # Sequential requests for real-time
    }
    
    def start_requests(self):
        # Monitor real-time endpoints
        endpoints = [
            'https://api.example.com/live/stock-prices',
            'https://api.example.com/live/crypto-prices',
            'https://api.example.com/live/forex-rates'
        ]
        
        for endpoint in endpoints:
            yield scrapy.Request(
                endpoint,
                callback=self.parse_realtime_data,
                meta={
                    'endpoint_type': self.get_endpoint_type(endpoint),
                    'start_time': time.time()
                }
            )
    
    def get_endpoint_type(self, endpoint):
        """Determine endpoint type from URL"""
        if 'stock' in endpoint:
            return 'stocks'
        elif 'crypto' in endpoint:
            return 'cryptocurrency'
        elif 'forex' in endpoint:
            return 'forex'
        return 'unknown'
    
    def parse_realtime_data(self, response):
        """Parse real-time financial data"""
        try:
            data = json.loads(response.text)
        except json.JSONDecodeError:
            return
        
        endpoint_type = response.meta['endpoint_type']
        timestamp = datetime.now().isoformat()
        
        # Process based on endpoint type
        if endpoint_type == 'stocks':
            yield from self.process_stock_data(data, timestamp)
        elif endpoint_type == 'cryptocurrency':
            yield from self.process_crypto_data(data, timestamp)
        elif endpoint_type == 'forex':
            yield from self.process_forex_data(data, timestamp)
        
        # Schedule next request for continuous monitoring
        yield scrapy.Request(
            response.url,
            callback=self.parse_realtime_data,
            meta=response.meta,
            dont_filter=True  # Allow duplicate requests
        )
    
    def process_stock_data(self, data, timestamp):
        """Process stock price data"""
        stocks = data.get('stocks', [])
        
        for stock in stocks:
            yield {
                'type': 'stock',
                'symbol': stock.get('symbol'),
                'price': stock.get('price'),
                'change': stock.get('change'),
                'change_percent': stock.get('changePercent'),
                'volume': stock.get('volume'),
                'timestamp': timestamp,
                'market_cap': stock.get('marketCap')
            }
    
    def process_crypto_data(self, data, timestamp):
        """Process cryptocurrency data"""
        currencies = data.get('data', [])
        
        for currency in currencies:
            yield {
                'type': 'cryptocurrency',
                'symbol': currency.get('symbol'),
                'name': currency.get('name'),
                'price_usd': currency.get('price_usd'),
                'price_btc': currency.get('price_btc'),
                'volume_24h': currency.get('24h_volume_usd'),
                'market_cap': currency.get('market_cap_usd'),
                'change_24h': currency.get('percent_change_24h'),
                'timestamp': timestamp
            }
    
    def process_forex_data(self, data, timestamp):
        """Process forex rates data"""
        rates = data.get('rates', {})
        base_currency = data.get('base', 'USD')
        
        for currency, rate in rates.items():
            yield {
                'type': 'forex',
                'base_currency': base_currency,
                'target_currency': currency,
                'rate': rate,
                'timestamp': timestamp
            }

Advanced Selector Techniques

Complex XPath and CSS Selectors

# Advanced selector utilities
class AdvancedSelectors:
    
    @staticmethod
    def extract_with_fallbacks(response, selectors):
        """Try multiple selectors until one works"""
        for selector in selectors:
            if selector.startswith('//'):
                # XPath selector
                result = response.xpath(selector).get()
            else:
                # CSS selector
                result = response.css(selector).get()
            
            if result:
                return result.strip()
        return None
    
    @staticmethod
    def extract_text_near_element(response, anchor_text, search_area='following'):
        """Extract text near a specific element"""
        if search_area == 'following':
            xpath = f"//text()[contains(., '{anchor_text}')]/following::text()[1]"
        elif search_area == 'preceding':
            xpath = f"//text()[contains(., '{anchor_text}')]/preceding::text()[1]"
        elif search_area == 'parent':
            xpath = f"//text()[contains(., '{anchor_text}')]/parent::*/text()"
        
        return response.xpath(xpath).get()
    
    @staticmethod
    def extract_table_data(response, table_selector):
        """Extract structured data from tables"""
        table = response.css(table_selector)
        if not table:
            return []
        
        headers = table.css('thead tr th::text').getall()
        if not headers:
            headers = table.css('tr:first-child td::text').getall()
        
        rows = []
        for row in table.css('tbody tr, tr')[1:]:  # Skip header row
            cells = row.css('td::text').getall()
            if len(cells) == len(headers):
                row_data = dict(zip(headers, cells))
                rows.append(row_data)
        
        return rows
    
    @staticmethod
    def extract_nested_json(response, script_selector):
        """Extract JSON data from script tags"""
        scripts = response.css(script_selector)
        
        for script in scripts:
            content = script.get()
            # Try to find JSON objects
            import re
            import json
            
            json_patterns = [
                r'var\s+\w+\s*=\s*({.+?});',
                r'window\.\w+\s*=\s*({.+?});',
                r'data:\s*({.+?})',
            ]
            
            for pattern in json_patterns:
                matches = re.findall(pattern, content, re.DOTALL)
                for match in matches:
                    try:
                        return json.loads(match)
                    except json.JSONDecodeError:
                        continue
        
        return None

# Usage in spider
def parse_complex_page(self, response):
    """Example using advanced selectors"""
    selectors = AdvancedSelectors()
    
    # Try multiple price selectors
    price = selectors.extract_with_fallbacks(response, [
        '.price-current::text',
        '.price::text',
        '//span[@class="price"]//text()',
        '.product-price .value::text'
    ])
    
    # Extract text near "Price:" label
    price_alt = selectors.extract_text_near_element(response, 'Price:', 'following')
    
    # Extract table data
    specs = selectors.extract_table_data(response, '.specifications-table')
    
    # Extract JSON configuration
    config = selectors.extract_nested_json(response, 'script[type="application/json"]')
    
    yield {
        'price': price or price_alt,
        'specifications': specs,
        'config': config
    }

Custom Middleware Development

Rotation Middleware

# webscraper/middlewares.py
import random
import time
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
from scrapy.exceptions import NotConfigured

class RotatingUserAgentMiddleware(UserAgentMiddleware):
    """Rotate user agents to avoid detection"""
    
    def __init__(self, user_agent=''):
        self.user_agent = user_agent
        
        # List of realistic user agents
        self.user_agent_list = [
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/121.0',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15',
            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
        ]
    
    def process_request(self, request, spider):
        ua = random.choice(self.user_agent_list)
        request.headers['User-Agent'] = ua
        return None

class ProxyRotationMiddleware:
    """Rotate proxies to distribute requests"""
    
    def __init__(self, proxy_list=None):
        if not proxy_list:
            raise NotConfigured('No proxy list provided')
        
        self.proxy_list = proxy_list
        self.proxy_index = 0
    
    @classmethod
    def from_crawler(cls, crawler):
        proxy_list = crawler.settings.getlist('PROXY_LIST')
        return cls(proxy_list)
    
    def process_request(self, request, spider):
        proxy = self.proxy_list[self.proxy_index]
        self.proxy_index = (self.proxy_index + 1) % len(self.proxy_list)
        
        request.meta['proxy'] = proxy
        spider.logger.debug(f'Using proxy: {proxy}')

class RetryWithBackoffMiddleware:
    """Implement exponential backoff for retries"""
    
    def __init__(self, max_retry_times=3, initial_delay=1):
        self.max_retry_times = max_retry_times
        self.initial_delay = initial_delay
    
    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            max_retry_times=crawler.settings.getint('RETRY_TIMES', 3),
            initial_delay=crawler.settings.getfloat('RETRY_INITIAL_DELAY', 1)
        )
    
    def process_response(self, request, response, spider):
        if response.status in [429, 503, 502, 504]:  # Rate limited or server errors
            retry_times = request.meta.get('retry_times', 0)
            
            if retry_times < self.max_retry_times:
                # Calculate exponential backoff delay
                delay = self.initial_delay * (2 ** retry_times)
                spider.logger.info(f'Retrying {request.url} after {delay}s (attempt {retry_times + 1})')
                
                # Add delay
                time.sleep(delay)
                
                # Create retry request
                retry_request = request.copy()
                retry_request.meta['retry_times'] = retry_times + 1
                retry_request.dont_filter = True
                
                return retry_request
        
        return response

class HeaderRotationMiddleware:
    """Rotate request headers to appear more natural"""
    
    def __init__(self):
        self.header_sets = [
            {
                'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
                'Accept-Language': 'en-US,en;q=0.5',
                'Accept-Encoding': 'gzip, deflate',
                'DNT': '1',
                'Connection': 'keep-alive',
                'Upgrade-Insecure-Requests': '1',
            },
            {
                'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
                'Accept-Language': 'en-US,en;q=0.9',
                'Accept-Encoding': 'gzip, deflate, br',
                'Connection': 'keep-alive',
                'Sec-Fetch-Dest': 'document',
                'Sec-Fetch-Mode': 'navigate',
                'Sec-Fetch-Site': 'none',
            }
        ]
    
    def process_request(self, request, spider):
        headers = random.choice(self.header_sets)
        for key, value in headers.items():
            request.headers[key] = value
        
        return None

Infinite Scroll and Dynamic Loading

# webscraper/spiders/infinite_scroll_spider.py
import scrapy
from scrapy_splash import SplashRequest
import json

class InfiniteScrollSpider(scrapy.Spider):
    name = 'infinite_scroll'
    allowed_domains = ['infinite-example.com']
    
    def start_requests(self):
        lua_script = """
        function main(splash, args)
            splash:go(args.url)
            splash:wait(2)
            
            -- Function to scroll and wait for content
            local function scroll_and_wait(times)
                for i = 1, times do
                    splash:runjs([[
                        window.scrollTo(0, document.body.scrollHeight);
                    ]])
                    splash:wait(2)
                    
                    -- Check if "Load More" button exists and click it
                    local load_more = splash:select('.load-more')
                    if load_more then
                        load_more:click()
                        splash:wait(3)
                    end
                    
                    -- Check if reached end
                    local end_marker = splash:select('.end-of-content')
                    if end_marker then
                        break
                    end
                end
            end
            
            -- Scroll multiple times to load content
            scroll_and_wait(5)
            
            return {
                html = splash:html(),
                url = splash:url()
            }
        end
        """
        
        yield SplashRequest(
            url='https://infinite-example.com/products',
            callback=self.parse,
            args={
                'lua_source': lua_script,
                'timeout': 60
            }
        )
    
    def parse(self, response):
        """Parse infinite scroll content"""
        products = response.css('.product-item')
        
        for product in products:
            yield {
                'name': product.css('.product-name::text').get(),
                'price': product.css('.price::text').get(),
                'url': response.urljoin(product.css('a::attr(href)').get())
            }
        
        # Look for AJAX endpoints to continue pagination
        self.extract_ajax_pagination(response)
    
    def extract_ajax_pagination(self, response):
        """Extract AJAX pagination endpoints"""
        import re
        
        # Look for pagination API endpoints in JavaScript
        ajax_patterns = [
            r'loadMore[\'\"]\s*:\s*[\'\"](.*?)[\'\"',
            r'pagination[\'\"]\s*:\s*[\'\"](.*?)[\'\"',
            r'nextPage[\'\"]\s*:\s*[\'\"](.*?)[\'\"'
        ]
        
        for pattern in ajax_patterns:
            matches = re.findall(pattern, response.text)
            for match in matches:
                if match.startswith('http') or match.startswith('/'):
                    yield scrapy.Request(
                        url=response.urljoin(match),
                        callback=self.parse_ajax_page,
                        headers={'X-Requested-With': 'XMLHttpRequest'}
                    )
    
    def parse_ajax_page(self, response):
        """Parse AJAX loaded content"""
        try:
            data = json.loads(response.text)
            
            # Extract HTML content from AJAX response
            html_content = data.get('html', '')
            if html_content:
                from scrapy import Selector
                selector = Selector(text=html_content)
                
                products = selector.css('.product-item')
                for product in products:
                    yield {
                        'name': product.css('.product-name::text').get(),
                        'price': product.css('.price::text').get(),
                        'ajax_source': True
                    }
            
            # Continue pagination if available
            next_page = data.get('nextPage')
            if next_page:
                yield response.follow(
                    next_page,
                    callback=self.parse_ajax_page
                )
                
        except json.JSONDecodeError:
            self.logger.error(f'Invalid JSON from {response.url}')

Summary

You can now render JavaScript pages through Splash, authenticate and keep sessions, pull data straight from AJAX endpoints, and write middleware for rotation and retries. That handles most sites; the ones that actively fight back are next.

What's Next?

Part 3: Anti-Detection and Scaling covers:

Advanced anti-detection techniques
Distributed scraping with Scrapy-Redis
Monitoring and alerting systems
Performance optimization strategies
Legal compliance and ethical scraping

Practice Exercise

Build a spider that can handle a modern e-commerce site with:

JavaScript-rendered product listings
User authentication for member prices
AJAX-loaded reviews and ratings
Infinite scroll pagination
Form-based search functionality

Scrapy, part 1: fundamentals and your first spider

Sat, 21 Dec 2024 00:00:00 GMT

Scrapy fundamentals and your first spider

This is part 1 of a five-part series on Scrapy. By the end of it you'll have a working project: a spider that crawls an e-commerce catalog, item loaders that clean the extracted data, and pipelines that validate and store it.

Series Overview

The full series:

Part 1: Scrapy Fundamentals (This tutorial)
Part 2: Advanced Scraping Techniques
Part 3: Anti-Detection and Scaling
Part 4: Data Processing and Storage
Part 5: Production Deployment

What this part covers

Web scraping ethics and legality
Setting up a Scrapy development environment
Building a first spider with a sane structure
Data extraction using selectors and XPath
Handling different data types and edge cases
Patterns for maintainable scraping code

Understanding Web Scraping

What is Web Scraping?

Web scraping is the automated extraction of data from websites: a program fetches pages and pulls out the fields you tell it to.

When to Use Web Scraping

# Common use cases for web scraping:
use_cases = {
    "e_commerce": [
        "Price monitoring and comparison",
        "Product catalog aggregation", 
        "Inventory tracking",
        "Competitor analysis"
    ],
    "real_estate": [
        "Property listings collection",
        "Market price analysis",
        "Investment opportunity identification"
    ],
    "news_media": [
        "News aggregation",
        "Sentiment analysis",
        "Content monitoring"
    ],
    "research": [
        "Academic paper collection",
        "Social media data analysis",
        "Market research"
    ]
}

Legal and Ethical Considerations

Before writing any code, the legal and ethical ground rules:

# Legal scraping checklist
legal_checklist = {
    "robots_txt": "Always check and respect robots.txt",
    "terms_of_service": "Review website terms before scraping",
    "rate_limiting": "Don't overload servers with requests",
    "personal_data": "Be careful with personal/sensitive information",
    "copyright": "Respect intellectual property rights",
    "public_data": "Focus on publicly available information"
}

# Ethical scraping principles
ethical_principles = [
    "Be respectful of website resources",
    "Don't impact site performance for other users", 
    "Use scraped data responsibly",
    "Give attribution when appropriate",
    "Consider contacting site owners for large-scale scraping"
]

Setting Up Your Scrapy Environment

Step 1: Python Environment Setup

Start with an isolated Python environment:

# Create a virtual environment
python -m venv scrapy_env

# Activate the environment
# On Windows:
scrapy_env\Scripts\activate
# On macOS/Linux:
source scrapy_env/bin/activate

# Upgrade pip
pip install --upgrade pip

Step 2: Install Scrapy and Dependencies

# Install Scrapy with all recommended packages
pip install scrapy

# Install additional useful packages
pip install scrapy-splash        # For JavaScript rendering
pip install scrapy-user-agents   # For rotating user agents
pip install scrapy-rotating-proxies  # For proxy rotation
pip install itemadapter          # For item processing
pip install pymongo             # For MongoDB storage
pip install psycopg2-binary     # For PostgreSQL storage
pip install redis               # For Redis-based deduplication

# Development tools
pip install ipython             # Better REPL
pip install scrapy-shell        # Enhanced shell
pip install black               # Code formatting
pip install flake8             # Linting

# Save requirements
pip freeze > requirements.txt

Step 3: Create Project Structure

# Create a new Scrapy project
scrapy startproject webscraper

# Navigate to project directory
cd webscraper

# Create additional directories for organization
mkdir -p data/raw data/processed data/exports
mkdir -p logs
mkdir -p scripts
mkdir -p tests

Your project structure should look like this:

webscraper/
├── scrapy.cfg                 # Deploy configuration
├── requirements.txt           # Python dependencies
├── data/                      # Data storage
│   ├── raw/                  # Raw scraped data
│   ├── processed/            # Cleaned data
│   └── exports/              # Final exports
├── logs/                     # Log files
├── scripts/                  # Utility scripts
├── tests/                    # Test files
└── webscraper/               # Main package
    ├── __init__.py
    ├── items.py              # Item definitions
    ├── middlewares.py        # Custom middlewares
    ├── pipelines.py          # Data processing pipelines
    ├── settings.py           # Project settings
    └── spiders/              # Spider modules
        └── __init__.py

Building Your First Spider

Step 1: Define Data Items

First, let's define what data we want to extract. We'll build a spider for scraping e-commerce products:

# webscraper/items.py
import scrapy
from itemloaders.processors import TakeFirst, MapCompose, Join
from w3lib.html import remove_tags


def clean_price(value):
    """Clean price string and convert to float"""
    if value:
        # Remove currency symbols and whitespace
        cleaned = ''.join(char for char in value if char.isdigit() or char == '.')
        try:
            return float(cleaned)
        except ValueError:
            return None
    return None


def clean_text(value):
    """Clean text by removing extra whitespace and HTML tags"""
    if value:
        cleaned = remove_tags(value).strip()
        return ' '.join(cleaned.split())
    return None


class ProductItem(scrapy.Item):
    # Basic product information
    name = scrapy.Field(
        input_processor=MapCompose(clean_text),
        output_processor=TakeFirst()
    )
    
    price = scrapy.Field(
        input_processor=MapCompose(clean_price),
        output_processor=TakeFirst()
    )
    
    original_price = scrapy.Field(
        input_processor=MapCompose(clean_price),
        output_processor=TakeFirst()
    )
    
    currency = scrapy.Field(
        output_processor=TakeFirst()
    )
    
    description = scrapy.Field(
        input_processor=MapCompose(clean_text),
        output_processor=Join(' ')
    )
    
    # Product details
    brand = scrapy.Field(
        input_processor=MapCompose(clean_text),
        output_processor=TakeFirst()
    )
    
    category = scrapy.Field(
        input_processor=MapCompose(clean_text),
        output_processor=TakeFirst()
    )
    
    sku = scrapy.Field(
        output_processor=TakeFirst()
    )
    
    availability = scrapy.Field(
        output_processor=TakeFirst()
    )
    
    rating = scrapy.Field(
        output_processor=TakeFirst()
    )
    
    review_count = scrapy.Field(
        output_processor=TakeFirst()
    )
    
    # Images and media
    images = scrapy.Field()
    
    # Metadata
    url = scrapy.Field(
        output_processor=TakeFirst()
    )
    
    scraped_at = scrapy.Field(
        output_processor=TakeFirst()
    )
    
    # Additional fields for tracking
    source = scrapy.Field(
        output_processor=TakeFirst()
    )

Step 2: Create Your First Spider

Now the spider itself. It uses fallback selectors and structured data so a single layout change doesn't break extraction:

# webscraper/spiders/ecommerce_spider.py
import scrapy
from scrapy.loader import ItemLoader
from webscraper.items import ProductItem
from datetime import datetime
import json
import re


class EcommerceSpider(scrapy.Spider):
    name = 'ecommerce'
    allowed_domains = ['example-store.com']
    
    # Custom settings for this spider
    custom_settings = {
        'DOWNLOAD_DELAY': 1,
        'RANDOMIZE_DOWNLOAD_DELAY': True,
        'AUTOTHROTTLE_ENABLED': True,
        'AUTOTHROTTLE_START_DELAY': 0.5,
        'AUTOTHROTTLE_MAX_DELAY': 3,
        'AUTOTHROTTLE_TARGET_CONCURRENCY': 2.0,
        'FEEDS': {
            'data/raw/products_%(time)s.json': {
                'format': 'json',
                'encoding': 'utf8',
                'store_empty': False,
                'fields': None,
                'indent': 2,
            },
        }
    }
    
    def start_requests(self):
        """Generate initial requests"""
        start_urls = [
            'https://example-store.com/products',
            'https://example-store.com/categories/electronics',
            'https://example-store.com/categories/clothing',
        ]
        
        for url in start_urls:
            yield scrapy.Request(
                url=url,
                callback=self.parse,
                meta={
                    'source': 'category_page',
                    'playwright': True,  # Enable JavaScript rendering if needed
                }
            )
    
    def parse(self, response):
        """Parse category pages and extract product links"""
        self.logger.info(f'Parsing category page: {response.url}')
        
        # Extract product links using CSS selectors
        product_links = response.css('.product-item a::attr(href)').getall()
        
        if not product_links:
            # Try alternative selectors
            product_links = response.css('.product-link::attr(href)').getall()
        
        # Follow product links
        for link in product_links:
            product_url = response.urljoin(link)
            yield scrapy.Request(
                url=product_url,
                callback=self.parse_product,
                meta={
                    'source': 'product_page',
                    'category_url': response.url
                }
            )
        
        # Follow pagination
        next_page = response.css('.pagination .next::attr(href)').get()
        if next_page:
            yield scrapy.Request(
                url=response.urljoin(next_page),
                callback=self.parse,
                meta=response.meta
            )
    
    def parse_product(self, response):
        """Parse individual product pages"""
        self.logger.info(f'Parsing product: {response.url}')
        
        # Create item loader for clean data extraction
        loader = ItemLoader(item=ProductItem(), response=response)
        
        # Basic product information
        loader.add_css('name', 'h1.product-title::text')
        loader.add_css('name', '.product-name::text')  # Fallback selector
        
        # Price extraction with multiple selectors
        loader.add_css('price', '.price-current::text')
        loader.add_css('price', '.current-price::text')
        loader.add_xpath('price', '//span[@class="price"]//text()')
        
        # Original price (if on sale)
        loader.add_css('original_price', '.price-original::text')
        loader.add_css('original_price', '.old-price::text')
        
        # Product description
        loader.add_css('description', '.product-description p::text')
        loader.add_xpath('description', '//div[@class="description"]//text()')
        
        # Product details
        loader.add_css('brand', '.brand-name::text')
        loader.add_css('category', '.breadcrumb li:last-child::text')
        loader.add_css('sku', '.product-sku::text')
        
        # Availability
        availability = response.css('.stock-status::text').get()
        if availability:
            loader.add_value('availability', 'in_stock' if 'in stock' in availability.lower() else 'out_of_stock')
        
        # Rating and reviews
        rating = response.css('.rating-value::text').get()
        if rating:
            loader.add_value('rating', float(rating))
        
        review_count_text = response.css('.review-count::text').get()
        if review_count_text:
            review_count = re.search(r'(\d+)', review_count_text)
            if review_count:
                loader.add_value('review_count', int(review_count.group(1)))
        
        # Images
        image_urls = response.css('.product-images img::attr(src)').getall()
        if image_urls:
            # Convert relative URLs to absolute
            absolute_urls = [response.urljoin(url) for url in image_urls]
            loader.add_value('images', absolute_urls)
        
        # Metadata
        loader.add_value('url', response.url)
        loader.add_value('scraped_at', datetime.now().isoformat())
        loader.add_value('source', response.meta.get('source', 'unknown'))
        
        # Extract structured data if available
        structured_data = self.extract_structured_data(response)
        if structured_data:
            self.update_loader_from_structured_data(loader, structured_data)
        
        yield loader.load_item()
    
    def extract_structured_data(self, response):
        """Extract JSON-LD structured data"""
        scripts = response.xpath('//script[@type="application/ld+json"]/text()').getall()
        
        for script in scripts:
            try:
                data = json.loads(script)
                if isinstance(data, dict) and data.get('@type') == 'Product':
                    return data
                elif isinstance(data, list):
                    for item in data:
                        if isinstance(item, dict) and item.get('@type') == 'Product':
                            return item
            except json.JSONDecodeError:
                continue
        
        return None
    
    def update_loader_from_structured_data(self, loader, data):
        """Update item loader with structured data"""
        if 'name' in data:
            loader.add_value('name', data['name'])
        
        if 'offers' in data and isinstance(data['offers'], dict):
            offer = data['offers']
            if 'price' in offer:
                loader.add_value('price', float(offer['price']))
            if 'priceCurrency' in offer:
                loader.add_value('currency', offer['priceCurrency'])
            if 'availability' in offer:
                availability = offer['availability'].split('/')[-1].lower()
                loader.add_value('availability', availability)
        
        if 'brand' in data:
            brand = data['brand']
            if isinstance(brand, dict) and 'name' in brand:
                loader.add_value('brand', brand['name'])
            elif isinstance(brand, str):
                loader.add_value('brand', brand)
        
        if 'aggregateRating' in data:
            rating_data = data['aggregateRating']
            if 'ratingValue' in rating_data:
                loader.add_value('rating', float(rating_data['ratingValue']))
            if 'reviewCount' in rating_data:
                loader.add_value('review_count', int(rating_data['reviewCount']))

Step 3: Data Processing Pipeline

Create a pipeline to process and validate extracted data:

# webscraper/pipelines.py
from itemadapter import ItemAdapter
import logging
import json
from datetime import datetime


class ValidationPipeline:
    """Validate and clean scraped items"""
    
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        
        # Validate required fields
        required_fields = ['name', 'url']
        for field in required_fields:
            if not adapter.get(field):
                raise DropItem(f"Missing required field: {field}")
        
        # Clean and validate price
        price = adapter.get('price')
        if price is not None:
            if not isinstance(price, (int, float)) or price < 0:
                spider.logger.warning(f"Invalid price for {adapter['name']}: {price}")
                adapter['price'] = None
        
        # Validate rating
        rating = adapter.get('rating')
        if rating is not None:
            if not isinstance(rating, (int, float)) or not (0 <= rating <= 5):
                spider.logger.warning(f"Invalid rating for {adapter['name']}: {rating}")
                adapter['rating'] = None
        
        return item


class DuplicationFilterPipeline:
    """Filter out duplicate items"""
    
    def __init__(self):
        self.seen_items = set()
    
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        
        # Create a unique identifier for the item
        identifier = f"{adapter['name']}_{adapter['url']}"
        
        if identifier in self.seen_items:
            raise DropItem(f"Duplicate item found: {adapter['name']}")
        else:
            self.seen_items.add(identifier)
            return item


class JsonWriterPipeline:
    """Write items to JSON file"""
    
    def __init__(self):
        self.file = None
        self.items = []
    
    def open_spider(self, spider):
        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
        filename = f"data/processed/{spider.name}_{timestamp}.json"
        self.file = open(filename, 'w', encoding='utf-8')
        spider.logger.info(f"Opened file: {filename}")
    
    def close_spider(self, spider):
        if self.file:
            json.dump(self.items, self.file, indent=2, ensure_ascii=False)
            self.file.close()
            spider.logger.info(f"Saved {len(self.items)} items")
    
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        self.items.append(dict(adapter))
        return item


class StatisticsPipeline:
    """Collect scraping statistics"""
    
    def __init__(self):
        self.stats = {
            'items_scraped': 0,
            'items_dropped': 0,
            'start_time': None,
            'end_time': None
        }
    
    def open_spider(self, spider):
        self.stats['start_time'] = datetime.now()
        spider.logger.info("Statistics collection started")
    
    def close_spider(self, spider):
        self.stats['end_time'] = datetime.now()
        duration = self.stats['end_time'] - self.stats['start_time']
        
        spider.logger.info("=== SCRAPING STATISTICS ===")
        spider.logger.info(f"Items scraped: {self.stats['items_scraped']}")
        spider.logger.info(f"Items dropped: {self.stats['items_dropped']}")
        spider.logger.info(f"Duration: {duration}")
        spider.logger.info(f"Items per minute: {self.stats['items_scraped'] / (duration.total_seconds() / 60):.2f}")
    
    def process_item(self, item, spider):
        self.stats['items_scraped'] += 1
        return item

Step 4: Configure Settings

Update the project settings:

# webscraper/settings.py

# Scrapy settings for webscraper project
BOT_NAME = 'webscraper'

SPIDER_MODULES = ['webscraper.spiders']
NEWSPIDER_MODULE = 'webscraper.spiders'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure pipelines
ITEM_PIPELINES = {
    'webscraper.pipelines.ValidationPipeline': 300,
    'webscraper.pipelines.DuplicationFilterPipeline': 400,
    'webscraper.pipelines.JsonWriterPipeline': 500,
    'webscraper.pipelines.StatisticsPipeline': 600,
}

# Configure delays and throttling
DOWNLOAD_DELAY = 1
RANDOMIZE_DOWNLOAD_DELAY = True

# AutoThrottle settings
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 0.5
AUTOTHROTTLE_MAX_DELAY = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
AUTOTHROTTLE_DEBUG = False

# User agent settings
USER_AGENT = 'webscraper (+http://www.yourdomain.com)'

# Configure caching
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 3600
HTTPCACHE_DIR = 'httpcache'

# Logging settings
LOG_LEVEL = 'INFO'
LOG_FILE = 'logs/scrapy.log'

# Retry settings
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]

# Concurrent requests
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8

# Memory usage optimization
MEMUSAGE_ENABLED = True
MEMUSAGE_LIMIT_MB = 2048
MEMUSAGE_WARNING_MB = 1024

# Request and response size limits
DOWNLOAD_MAXSIZE = 1073741824  # 1GB
DOWNLOAD_WARNSIZE = 33554432   # 32MB

Running Your Spider

Basic Execution

# Run the spider
scrapy crawl ecommerce

# Run with custom settings
scrapy crawl ecommerce -s DOWNLOAD_DELAY=2

# Save output to specific file
scrapy crawl ecommerce -o products.json

# Run with custom log level
scrapy crawl ecommerce -L DEBUG

Advanced Execution with Parameters

# Create a script to run spiders with parameters
# scripts/run_spider.py

import subprocess
import sys
from datetime import datetime

def run_spider(spider_name, **kwargs):
    """Run spider with custom parameters"""
    
    cmd = ['scrapy', 'crawl', spider_name]
    
    # Add custom settings
    for key, value in kwargs.items():
        cmd.extend(['-s', f'{key}={value}'])
    
    # Add timestamp to output file
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    output_file = f'data/raw/{spider_name}_{timestamp}.json'
    cmd.extend(['-o', output_file])
    
    print(f"Running command: {' '.join(cmd)}")
    
    try:
        result = subprocess.run(cmd, check=True, capture_output=True, text=True)
        print("Spider completed successfully!")
        print(f"Output saved to: {output_file}")
        return True
    except subprocess.CalledProcessError as e:
        print(f"Spider failed with error: {e}")
        print(f"Error output: {e.stderr}")
        return False

if __name__ == "__main__":
    # Example usage
    run_spider(
        'ecommerce',
        DOWNLOAD_DELAY=1.5,
        CONCURRENT_REQUESTS=8,
        LOG_LEVEL='INFO'
    )

Testing Your Spider

Unit Tests

# tests/test_ecommerce_spider.py
import unittest
from scrapy.http import HtmlResponse, Request
from webscraper.spiders.ecommerce_spider import EcommerceSpider

class TestEcommerceSpider(unittest.TestCase):
    
    def setUp(self):
        self.spider = EcommerceSpider()
    
    def test_parse_product(self):
        """Test product parsing"""
        # Sample HTML response
        html = """
        <html>
        <body>
            <h1 class="product-title">Test Product</h1>
            <span class="price-current">$99.99</span>
            <p class="product-description">This is a test product</p>
        </body>
        </html>
        """
        
        request = Request(url='http://example.com/product/1')
        response = HtmlResponse(
            url='http://example.com/product/1',
            request=request,
            body=html.encode('utf-8')
        )
        
        # Process the response
        items = list(self.spider.parse_product(response))
        
        # Assertions
        self.assertEqual(len(items), 1)
        item = items[0]
        self.assertEqual(item['name'], 'Test Product')
        self.assertEqual(item['price'], 99.99)

if __name__ == '__main__':
    unittest.main()

Tips

1. Selector fallbacks

def extract_with_fallbacks(response, selectors):
    """Extract data with multiple fallback selectors"""
    for selector in selectors:
        result = response.css(selector).get()
        if result:
            return result.strip()
    return None

# Usage example
price = extract_with_fallbacks(response, [
    '.price-current::text',
    '.current-price::text', 
    '.price::text',
    '[data-price]::attr(data-price)'
])

2. Error Handling

def safe_extract_float(value, default=None):
    """Safely extract float from string"""
    if not value:
        return default
    
    try:
        # Clean the string
        cleaned = ''.join(char for char in str(value) if char.isdigit() or char in '.-')
        return float(cleaned)
    except (ValueError, TypeError):
        return default

3. Logging and Monitoring

# Add custom logging to your spider
import logging

class EcommerceSpider(scrapy.Spider):
    
    def __init__(self):
        self.stats = {
            'products_found': 0,
            'products_processed': 0,
            'errors': 0
        }
    
    def parse_product(self, response):
        try:
            self.stats['products_found'] += 1
            # ... processing logic ...
            self.stats['products_processed'] += 1
            
        except Exception as e:
            self.stats['errors'] += 1
            self.logger.error(f"Error processing {response.url}: {e}")
    
    def closed(self, reason):
        self.logger.info(f"Spider closed: {reason}")
        self.logger.info(f"Statistics: {self.stats}")

Summary

You now have a working Scrapy project: an isolated environment, a spider that extracts data through fallback selectors and JSON-LD, pipelines for validation and deduplication, and a unit test to keep it honest.

What's Next?

Part 2: Advanced Scraping Techniques covers:

Handling JavaScript-heavy websites with Splash
Form submission and login handling
Advanced selector techniques and data extraction
Handling AJAX requests and dynamic content
Custom middleware development

Practice Exercise

Before moving to Part 2, try building a spider for your favorite e-commerce site:

Create a new spider targeting a simple e-commerce site
Extract product names, prices, and descriptions
Implement proper error handling and logging
Add data validation pipelines
Test your spider with different product categories

Resources

Installing tmux on Synology, the cleaner way

Mon, 09 Dec 2024 00:00:00 GMT

I was working on a project where my client wanted me to put all the crawlers and applications on his own Synology NAS. It was a new thing for me. He only knew that he was able to run containers on his server. The rest was my task to figure out. Among a lot of problems at each step of this project, I ran into this one too, so I am sharing it here.

Installing tmux on NAS can be frustrating.

To install tmux on a Synology NAS, you typically need to use package managers because Synology's DiskStation Manager (DSM) doesn't include tmux by default. While many guides suggest using ipkg, the modern approach is to use Entware with opkg.

Here's a step-by-step guide on how to install tmux on your Synology NAS:

Step 1: Prepare Your System

Enable SSH Access:
- Open DiskStation Manager (DSM)
- Navigate to Control Panel → Terminal & SNMP
- Check "Enable SSH service"
- Click Apply
Connect via SSH:
- Use your preferred SSH client to connect to your NAS
- Login with your admin credentials

Step 2: Install Entware

Create Required Directories:

sudo mkdir -p /volume1/@entware/opt
sudo mount -o bind /volume1/@entware/opt /opt

Download and Install Entware:

wget -O - http://bin.entware.net/x64-k3.2/installer/generic.sh | /bin/sh

Add Entware to PATH:
```
export PATH=/opt/bin:/opt/sbin:$PATH
```

Step 3: Install tmux

Update Package List:
```
opkg update
```
Install tmux:
```
opkg install tmux
```

Step 4: Make Installation Persistent

To ensure Entware remains available after system reboots:

Create Startup Script:

sudo vi /usr/local/etc/rc.d/S99entware.sh

Add Following Content:

#!/bin/sh
case $1 in
  start)
    mkdir -p /volume1/@entware/opt
    mount -o bind /volume1/@entware/opt /opt
    ;;
  stop)
    umount /opt
    ;;
  *)
    echo "Usage: $0 [start|stop]"
    exit 1
    ;;
esac

Make Script Executable:

sudo chmod +x /usr/local/etc/rc.d/S99entware.sh

Basic tmux Usage

Once installed, you can start using tmux with these basic commands:

Start New Session:
```
tmux
```
Create Named Session:
```
tmux new -s mysession
```
List Sessions:
```
tmux ls
```
Attach to Session:
```
tmux attach -t mysession
```

Common tmux Shortcuts

All commands are prefixed with Ctrl+b:

d: Detach from current session
c: Create new window
n: Next window
p: Previous window
%: Split pane horizontally
": Split pane vertically
Arrow keys: Navigate between panes

Troubleshooting Common Issues

Command Not Found
- Verify Entware installation
- Check if PATH includes /opt/bin and /opt/sbin
- Try running hash -r to clear command cache
Permission Denied
- Ensure you're using sudo for administrative tasks
- Check file permissions on startup scripts
- Verify your user has sufficient privileges
Mount Point Errors
- Confirm /volume1/@entware exists
- Check if /opt is already mounted
- Try unmounting and remounting if necessary

Why This Method?

While some guides suggest using older methods like ipkg or the SynoCommunity package source, the Entware approach offers several advantages:

Modern Package Management: Entware is actively maintained and updated
Better Compatibility: Works across different Synology models
Larger Package Repository: Access to more software packages
Easier Maintenance: Simpler updates and package management

Conclusion

With the startup script in place, Entware and tmux survive DSM reboots, which was the part missing from my first write-up. From here, tmux configs and plugins work the same as on any other Linux box.

How to install tmux on a Synology NAS

Sun, 09 Jun 2024 00:00:00 GMT

Installing tmux on NAS can be frustrating.

To install tmux on a Synology NAS, you typically need to use ipkg or opkg because Synology's DiskStation Manager (DSM) doesn’t include tmux by default, and it uses a busybox-based system that lacks many common Linux utilities.

Here's a step-by-step guide on how to install tmux on a Synology NAS using opkg:

Step 1: Install Entware

Download and Install Entware:
- Entware is a software repository for embedded devices like Synology NAS. It provides a package management system similar to apt on Debian or yum on CentOS.
- You can download and install Entware by following these steps:
```
wget -O - http://bin.entware.net/x64-k3.2/installer/generic.sh | /bin/sh
```
Initialize Entware:
- After the installation script runs, it will set up Entware. You might need to add the Entware binary path to your PATH environment variable:
```
echo 'export PATH=/opt/bin:/opt/sbin:$PATH' >> ~/.profile
source ~/.profile
```

Step 2: Install `tmux` using `opkg`

Update the Entware Package List:
- Ensure you have the latest package list:
```
opkg update
```
Install tmux:
- Now, you can install tmux:
```
opkg install tmux
```
Verify Installation:
- Check if tmux is installed correctly:
```
tmux -V
```
- You should see the version number of tmux if it was installed successfully.

Alternative Method: Using Docker

If you prefer not to use Entware or have trouble with it, you can use Docker to run tmux. Here's how:

Ensure Docker is Installed:
- Install the Docker package from the Synology Package Center if it's not already installed.
Run a tmux Docker Container:
- You can use an existing Docker image with tmux. Pull a lightweight image like Alpine Linux:
```
docker run -it --rm alpine:latest /bin/sh
```
- Inside the Docker container, install tmux:
```
apk add tmux
tmux
```

Troubleshooting

Missing Commands: If you encounter errors like command not found, make sure that /opt/bin and /opt/sbin are included in your PATH.
Permission Issues: Ensure you have root access or use sudo where necessary.
Compatibility: Verify that your Synology NAS model supports Entware or Docker.

Summary

Installing tmux on a Synology NAS comes down to installing Entware and then using opkg. Entware also gives you a lot of other Linux utilities that DSM does not ship, so it is worth having anyway.