Crawler best practices and production lessons

Home / Web crawling / Crawler best practices and production lessons

Introduction

This final post in our web crawling series distills lessons from operating production crawling systems at scale. These aren’t theoretical best practices—they’re hard-won insights from real deployments, failures, and recoveries.

We’ll cover:

Reliability patterns and fault tolerance
Monitoring and alerting strategies
Resource optimization and scalability
Error handling and recovery
Scheduling and orchestration
Lessons learned from failures
Gradual deployment and testing

By the end, you’ll be prepared to deploy and maintain crawling systems in production environments.

Reliability and Fault Tolerance

Idempotency: The Foundation

Idempotent operations produce the same result whether they run once or multiple times. This is essential for resilient systems.

// ❌ Non-idempotent: Running twice creates duplicates
const processArticle = async (article: Article) => {
  await db.query(
    'INSERT INTO articles (slug, title, content) VALUES ($1, $2, $3)',
    [article.slug, article.title, article.content]
  );
};

// ✅ Idempotent: Upsert ensures same result
const processArticleIdempotent = async (article: Article) => {
  await db.query(`
    INSERT INTO articles (slug, title, content, updated_at)
    VALUES ($1, $2, $3, NOW())
    ON CONFLICT (slug) DO UPDATE SET
      title = EXCLUDED.title,
      content = EXCLUDED.content,
      updated_at = NOW()
  `, [article.slug, article.title, article.content]);
};

// ✅ Idempotent with deduplication token
const processWithToken = async (
  article: Article,
  idempotencyKey: string
) => {
  const result = await db.query(
    'SELECT id FROM processed_articles WHERE idempotency_key = $1',
    [idempotencyKey]
  );
  
  if (result.rows.length > 0) {
    console.log('Already processed, skipping');
    return result.rows[0].id;
  }
  
  const articleId = await insertArticle(article);
  
  await db.query(
    'INSERT INTO processed_articles (id, idempotency_key) VALUES ($1, $2)',
    [articleId, idempotencyKey]
  );
  
  return articleId;
};

Circuit Breaker Pattern

Detect and handle repeated failures gracefully, avoiding cascading failures.

enum CircuitState {
  CLOSED = 'CLOSED',      // Normal, requests flowing
  OPEN = 'OPEN',          // Failing, reject requests
  HALF_OPEN = 'HALF_OPEN' // Testing recovery
}

class CircuitBreaker {
  private state: CircuitState = CircuitState.CLOSED;
  private failureCount = 0;
  private failureThreshold = 5;
  private resetTimeout = 60000; // 1 minute
  private lastFailureTime = 0;

  async execute<T>(
    operation: () => Promise<T>,
    fallback?: () => Promise<T>
  ): Promise<T> {
    if (this.state === CircuitState.OPEN) {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        console.log('Circuit breaker entering HALF_OPEN state');
        this.state = CircuitState.HALF_OPEN;
      } else if (fallback) {
        return fallback();
      } else {
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await operation();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess() {
    this.failureCount = 0;
    if (this.state === CircuitState.HALF_OPEN) {
      console.log('Circuit breaker recovered, entering CLOSED state');
      this.state = CircuitState.CLOSED;
    }
  }

  private onFailure() {
    this.failureCount++;
    this.lastFailureTime = Date.now();

    if (this.failureCount >= this.failureThreshold) {
      console.error('Failure threshold exceeded, opening circuit');
      this.state = CircuitState.OPEN;
    }
  }
}

// Usage
const circuitBreaker = new CircuitBreaker();

const crawlWithFallback = async (url: string) => {
  return circuitBreaker.execute(
    () => fetchAndParse(url),
    () => getCachedData(url) // Fallback to cache
  );
};

Exponential Backoff and Retry

interface RetryConfig {
  maxAttempts: number;
  initialDelayMs: number;
  maxDelayMs: number;
  backoffMultiplier: number;
  retryableStatusCodes: number[];
}

const defaultRetryConfig: RetryConfig = {
  maxAttempts: 5,
  initialDelayMs: 1000,
  maxDelayMs: 60000,
  backoffMultiplier: 2,
  retryableStatusCodes: [408, 429, 500, 502, 503, 504]
};

const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));

const retryWithBackoff = async <T>(
  operation: () => Promise<T>,
  config: RetryConfig = defaultRetryConfig,
  attempt: number = 1
): Promise<T> => {
  try {
    return await operation();
  } catch (error: any) {
    const isRetryable =
      config.retryableStatusCodes.includes(error.statusCode) ||
      ['ECONNREFUSED', 'ETIMEDOUT', 'ENOTFOUND'].includes(error.code);

    if (attempt < config.maxAttempts && isRetryable) {
      const delayMs = Math.min(
        config.initialDelayMs * Math.pow(config.backoffMultiplier, attempt - 1),
        config.maxDelayMs
      );

      // Add jitter to prevent thundering herd
      const jitter = Math.random() * delayMs * 0.1;
      const totalDelay = delayMs + jitter;

      console.log(
        `Attempt ${attempt} failed, retrying in ${totalDelay.toFixed(0)}ms: ${error.message}`
      );

      await sleep(totalDelay);
      return retryWithBackoff(operation, config, attempt + 1);
    }

    throw error;
  }
};

// Usage
const crawlWithRetry = async (url: string) => {
  return retryWithBackoff(() => fetchPage(url));
};

Monitoring and Alerting

Metrics Collection

interface MetricPoint {
  timestamp: Date;
  name: string;
  value: number;
  tags: Record<string, string>;
}

class MetricsCollector {
  private metrics: MetricPoint[] = [];
  private flushInterval = 60000; // 1 minute

  constructor(private metricsBackend: MetricsBackend) {
    setInterval(() => this.flush(), this.flushInterval);
  }

  record(
    name: string,
    value: number,
    tags: Record<string, string> = {}
  ) {
    this.metrics.push({
      timestamp: new Date(),
      name,
      value,
      tags
    });
  }

  gauge(name: string, value: number, tags?: Record<string, string>) {
    this.record(name, value, tags);
  }

  counter(name: string, increment: number = 1, tags?: Record<string, string>) {
    this.record(name, increment, tags);
  }

  histogram(name: string, value: number, tags?: Record<string, string>) {
    this.record(name, value, tags);
  }

  private async flush() {
    if (this.metrics.length === 0) return;

    try {
      await this.metricsBackend.send(this.metrics);
      this.metrics = [];
    } catch (error) {
      console.error('Failed to flush metrics:', error);
      // Keep metrics for next flush
    }
  }
}

interface MetricsBackend {
  send(metrics: MetricPoint[]): Promise<void>;
}

// Usage
const metrics = new MetricsCollector(new PrometheusBackend());

const crawlPage = async (url: string) => {
  const startTime = Date.now();
  let success = false;

  try {
    const result = await fetchPage(url);
    success = true;
    return result;
  } finally {
    const duration = Date.now() - startTime;
    metrics.histogram('crawl_duration_ms', duration, { url });
    metrics.counter('crawl_' + (success ? 'success' : 'failure'), 1, { url });
  }
};

Health Checks

interface HealthStatus {
  status: 'healthy' | 'degraded' | 'unhealthy';
  checks: {
    [name: string]: { status: string; message?: string };
  };
  timestamp: Date;
}

class HealthChecker {
  private checks: Array<() => Promise<[string, boolean, string?]>> = [];

  register(
    name: string,
    check: () => Promise<boolean>,
    description?: string
  ) {
    this.checks.push(async () => [
      name,
      await check(),
      description
    ]);
  }

  async check(): Promise<HealthStatus> {
    const results = await Promise.allSettled(
      this.checks.map(check => check())
    );

    const checks: Record<string, { status: string; message?: string }> = {};
    let overallStatus: 'healthy' | 'degraded' | 'unhealthy' = 'healthy';

    for (const result of results) {
      if (result.status === 'fulfilled') {
        const [name, passed, message] = result.value;
        checks[name] = {
          status: passed ? 'pass' : 'fail',
          message
        };
        if (!passed) overallStatus = 'degraded';
      } else {
        overallStatus = 'unhealthy';
      }
    }

    return {
      status: overallStatus,
      checks,
      timestamp: new Date()
    };
  }
}

// Register health checks
const health = new HealthChecker();

health.register('database', async () => {
  try {
    const result = await db.query('SELECT 1');
    return result.rows.length > 0;
  } catch {
    return false;
  }
});

health.register('memory', async () => {
  const used = process.memoryUsage().heapUsed / 1024 / 1024;
  return used < 500; // Alert if > 500MB
}, `Memory: ${used}MB`);

health.register('disk_space', async () => {
  const available = await checkDiskSpace();
  return available > 1000 * 1024 * 1024; // > 1GB free
});

// Express endpoint
app.get('/health', async (req, res) => {
  const status = await health.check();
  const statusCode = status.status === 'healthy' ? 200 : 503;
  res.status(statusCode).json(status);
});

Resource Optimization

Connection Pooling

// ❌ Creates new connection per request (wasteful)
const inefficientCrawl = async (urls: string[]) => {
  for (const url of urls) {
    const response = await fetch(url);
    // ...
  }
};

// ✅ Reuses connection pool
const efficientCrawl = async (urls: string[]) => {
  const agent = new https.Agent({
    keepAlive: true,
    maxSockets: 50,
    maxFreeSockets: 10,
    timeout: 30000,
    freeSocketTimeout: 30000
  });

  for (const url of urls) {
    const response = await fetch(url, { agent });
    // ...
  }
};

Memory Management

// ❌ Accumulates all data in memory
const inefficientParse = async (htmlArray: string[]) => {
  const results = [];
  for (const html of htmlArray) {
    results.push(parseAndExtract(html));
  }
  return results;
};

// ✅ Processes in batches, releasing memory
async function* processBatches(
  htmlArray: string[],
  batchSize: number = 100
) {
  for (let i = 0; i < htmlArray.length; i += batchSize) {
    const batch = htmlArray.slice(i, i + batchSize);
    const results = batch.map(parseAndExtract);
    yield results;

    // Force garbage collection after batch
    if (global.gc) {
      global.gc();
    }
  }
}

// Stream processing for very large datasets
const processStream = async (inputFile: string) => {
  const stream = fs.createReadStream(inputFile);
  const rl = readline.createInterface({ input: stream });

  for await (const line of rl) {
    const article = JSON.parse(line);
    await processArticle(article);
  }
};

Rate Limiting

class RateLimiter {
  private requestTimestamps: number[] = [];

  constructor(
    private maxRequests: number,
    private windowMs: number = 1000
  ) {}

  async acquire(weight: number = 1): Promise<void> {
    const now = Date.now();

    // Remove old timestamps outside window
    this.requestTimestamps = this.requestTimestamps.filter(
      ts => now - ts < this.windowMs
    );

    // Check if we can make request
    const currentLoad = this.requestTimestamps.reduce((sum, _) => sum + 1, 0);

    if (currentLoad + weight > this.maxRequests) {
      // Calculate wait time
      const oldestRequest = this.requestTimestamps[0];
      const waitTime = oldestRequest ? oldestRequest + this.windowMs - now : 0;

      console.log(`Rate limit exceeded, waiting ${waitTime}ms`);
      await new Promise(resolve => setTimeout(resolve, waitTime));

      return this.acquire(weight);
    }

    // Record request
    this.requestTimestamps.push(now);
  }
}

// Usage: 100 requests per second
const limiter = new RateLimiter(100, 1000);

const limitedFetch = async (url: string) => {
  await limiter.acquire();
  return fetch(url);
};

Error Handling and Recovery

Structured Logging

interface LogContext {
  requestId?: string;
  url?: string;
  attempt?: number;
  duration?: number;
  error?: Error | string;
  statusCode?: number;
}

class Logger {
  log(level: string, message: string, context: LogContext = {}) {
    const logEntry = {
      timestamp: new Date().toISOString(),
      level,
      message,
      ...context,
      // Include stack trace for errors
      ...(context.error instanceof Error && {
        stack: context.error.stack,
        errorMessage: context.error.message
      })
    };

    console.log(JSON.stringify(logEntry));
  }

  info(message: string, context?: LogContext) {
    this.log('INFO', message, context);
  }

  warn(message: string, context?: LogContext) {
    this.log('WARN', message, context);
  }

  error(message: string, context?: LogContext) {
    this.log('ERROR', message, context);
  }
}

// Usage
const logger = new Logger();

const crawlWithLogging = async (url: string, requestId: string) => {
  const startTime = Date.now();

  try {
    logger.info('Starting crawl', { requestId, url });

    const result = await fetch(url);

    logger.info('Crawl successful', {
      requestId,
      url,
      duration: Date.now() - startTime,
      statusCode: result.status
    });

    return result;
  } catch (error) {
    logger.error('Crawl failed', {
      requestId,
      url,
      duration: Date.now() - startTime,
      error
    });

    throw error;
  }
};

Dead Letter Queues

interface FailedJob {
  id: string;
  url: string;
  attempts: number;
  lastError: string;
  createdAt: Date;
  failedAt: Date;
}

class DeadLetterQueue {
  constructor(private db: Database) {}

  async addFailedJob(job: any, error: Error, attempts: number) {
    await this.db.query(`
      INSERT INTO dead_letter_queue (job_data, error_message, attempts, created_at)
      VALUES ($1, $2, $3, NOW())
    `, [JSON.stringify(job), error.message, attempts]);

    console.error(`Job moved to DLQ: ${job.url} (attempt ${attempts})`);
  }

  async getFailedJobs(limit: number = 100): Promise<FailedJob[]> {
    const result = await this.db.query(`
      SELECT * FROM dead_letter_queue
      ORDER BY failed_at DESC
      LIMIT $1
    `, [limit]);

    return result.rows;
  }

  async retryJob(jobId: string) {
    const result = await this.db.query(
      'SELECT job_data FROM dead_letter_queue WHERE id = $1',
      [jobId]
    );

    if (result.rows.length === 0) return;

    const job = JSON.parse(result.rows[0].job_data);
    await this.db.query('DELETE FROM dead_letter_queue WHERE id = $1', [jobId]);

    return job;
  }
}

Scheduling and Orchestration

Cron-Based Scheduling

import cron from 'node-cron';

class CrawlScheduler {
  constructor(private crawler: WebCrawler) {}

  schedule() {
    // Daily crawl at 2 AM
    cron.schedule('0 2 * * *', async () => {
      console.log('Running daily crawl');
      try {
        await this.crawler.crawlMainSites();
      } catch (error) {
        console.error('Daily crawl failed:', error);
        // Send alert
      }
    });

    // Hourly incremental update
    cron.schedule('0 * * * *', async () => {
      console.log('Running incremental crawl');
      try {
        await this.crawler.updateRecent();
      } catch (error) {
        console.error('Incremental crawl failed:', error);
      }
    });

    // Cleanup old records (monthly)
    cron.schedule('0 3 1 * *', async () => {
      console.log('Cleaning up old records');
      try {
        const deleted = await this.crawler.cleanupOldData(90);
        console.log(`Deleted ${deleted} old records`);
      } catch (error) {
        console.error('Cleanup failed:', error);
      }
    });
  }
}

Queue-Based Processing

import Queue from 'bull';

// Create queue with persistence
const crawlQueue = new Queue('crawl', {
  redis: {
    host: 'localhost',
    port: 6379
  }
});

// Process jobs with concurrency control
crawlQueue.process(10, async (job) => {
  const { url } = job.data;

  try {
    // Update progress
    job.progress(10);

    const html = await fetchPage(url);
    job.progress(50);

    const data = extractData(html);
    job.progress(80);

    await saveData(data);
    job.progress(100);

    return { success: true, url, recordsInserted: data.length };
  } catch (error) {
    // Automatic retry on failure
    throw error;
  }
});

// Add jobs to queue
const addCrawlJobs = async (urls: string[]) => {
  for (const url of urls) {
    await crawlQueue.add(
      { url },
      {
        attempts: 3,
        backoff: {
          type: 'exponential',
          delay: 2000
        },
        removeOnComplete: true,
        removeOnFail: false
      }
    );
  }
};

// Monitor queue
crawlQueue.on('completed', (job, result) => {
  console.log(`Job ${job.id} completed:`, result);
});

crawlQueue.on('failed', (job, error) => {
  console.error(`Job ${job.id} failed:`, error.message);
});

Testing and Deployment

Testing Crawlers

import { test, expect } from '@jest/globals';
import nock from 'nock';

describe('Web Crawler', () => {
  // Mock HTTP requests
  beforeEach(() => {
    nock('https://example.com')
      .get('/articles')
      .reply(200, `
        <html>
          <article class="post">
            <h2>Test Article</h2>
            <p class="content">Article content</p>
          </article>
        </html>
      `);
  });

  test('correctly extracts article data', async () => {
    const result = await crawler.crawl('https://example.com/articles');

    expect(result).toHaveLength(1);
    expect(result[0]).toEqual({
      title: 'Test Article',
      content: 'Article content'
    });
  });

  test('handles network errors gracefully', async () => {
    nock('https://fail.com').get('/').replyWithError('Network error');

    const result = await crawler.crawl('https://fail.com', { maxRetries: 1 });

    expect(result).toEqual([]);
  });

  test('respects robots.txt', async () => {
    nock('https://example.com').get('/robots.txt').reply(200, 'User-agent: *\nDisallow: /admin/');

    const allowed = await crawler.isAllowed('https://example.com/admin/');

    expect(allowed).toBe(false);
  });
});

Gradual Rollout

class DeploymentController {
  async deploy(newVersion: string) {
    // Start with 10% traffic
    await this.setTrafficPercentage(newVersion, 10);
    await this.monitor(5 * 60 * 1000); // 5 minutes

    // Increase to 50%
    await this.setTrafficPercentage(newVersion, 50);
    await this.monitor(10 * 60 * 1000); // 10 minutes

    // Full rollout
    await this.setTrafficPercentage(newVersion, 100);

    console.log(`Successfully deployed ${newVersion}`);
  }

  async monitor(durationMs: number) {
    const startMetrics = await this.getMetrics();

    await new Promise(resolve => setTimeout(resolve, durationMs));

    const endMetrics = await this.getMetrics();
    const errorRate = (endMetrics.errors / endMetrics.total) * 100;

    if (errorRate > 2) { // Rollback on > 2% error rate
      throw new Error(`High error rate: ${errorRate}%`);
    }
  }

  private async setTrafficPercentage(version: string, percentage: number) {
    // Updated load balancer or service mesh configuration
    console.log(`Setting ${version} to ${percentage}% traffic`);
  }

  private async getMetrics() {
    return {
      total: 10000,
      errors: 50,
      latency: 200
    };
  }
}

Production Lessons Learned

Failure is inevitable; prepare for it
- Design for graceful degradation
- Implement comprehensive error handling
- Make systems observable from day one
Test your assumptions
- Website redesigns break scrapers
- Test against real-world data, not mocked
- Run integration tests in staging environment
Monitor Everything
- Implement health checks and alerting
- Track not just success, but data quality
- Alert on anomalies (drops in data variety, unexpected spikes)
Respect the hosts
- Honor robots.txt and rate limits
- Work with website owners when possible
- Avoid overwhelming servers with requests
Data governance matters
- Know what personal data you’re collecting
- Implement retention policies
- Maintain audit trails
Expect change
- Website structures evolve
- Selectors break
- Have systems to detect and alert on breaking changes
Communicate deployment
- Notify stakeholders before major changes
- Have rollback plans
- Monitor metrics closely during deployments

Conclusion

Building reliable, production-grade web crawling systems requires attention to detail across many dimensions:

Reliability through idempotence and fault tolerance
Observability through comprehensive metrics and logging
Efficiency through resource optimization
Maintainability through testing and established patterns

The lessons here represent experience from operating systems at scale. Start here, learn from your own failures, and adapt to your specific context.

Discussion Questions

What’s the biggest failure you’ve experienced in a crawling system?
How do you monitor crawler health in production?
What unexpected challenges have you encountered?

What’s Next?

You’ve now completed the web crawling series. You understand:

✅ Ethical foundations and legal frameworks
✅ Available tools and how to choose them
✅ Data extraction techniques
✅ Storage strategies for scale
✅ Analysis methodologies
✅ Production best practices

Recommended next steps:

Start a small crawling project to apply these concepts
Build your monitoring and alerting infrastructure
Read the case studies referenced in this series
Join communities of practice (IEEE, ACM) studying web crawling

The web is vast and constantly changing. These foundational concepts will serve you well as technologies evolve.

Part of the Web Crawling and Data Analysis Series

Previous: Part 5: Data Analysis and Insights
Series Overview: Web Crawling and Data Analysis Series

Further Resources:

You also might like

About this post

This post was written by a human with AI assistance, which was used for research, outlining, editing and code suggestions. Claims, code samples and references were reviewed before publication, and the sources are cited so you can judge them yourself.