Data Extraction Techniques: From DOM to Structured Data

Home / Web crawling / Data Extraction Techniques: From DOM to Structured Data

Introduction

Extracting data from HTML is an art balanced between precision and maintainability. You may have the right tool from Part 2, but without effective extraction techniques, you’ll end up with messy, unreliable data pipelines.

This post covers practical strategies for:

Identifying reliable selectors (CSS, XPath, Regex)
Handling dynamic and variable content
Cleaning and normalizing data
Building maintainable extraction pipelines
Testing and validating extraction logic

By the end, you’ll be able to extract clean, structured data even from complex, inconsistently-formatted web pages.

Foundation Concepts

The DOM as Data Source

Web browsers convert HTML into a Document Object Model (DOM)—a tree structure representing the page:

<article class="post">
  <header>
    <h1 class="title">Article Title</h1>
    <span class="date">2026-02-23</span>
  </header>
  <div class="content">
    <p>First paragraph</p>
    <p>Second paragraph</p>
  </div>
  <footer class="meta">
    <span class="author">John Doe</span>
    <span class="tags">
      <a href="/tags/web">web</a>
      <a href="/tags/scraping">scraping</a>
    </span>
  </footer>
</article>

DOM traversal targets specific nodes and extracts their content.

Selector Types

CSS Selectors: Browser-native, performant, intuitive

.post h1.title        /* Article title */
article .date         /* Date within article */
.tags a               /* All tag links */

XPath: Powerful but verbose, better for complex logic

//article[@class='post']//h1[@class='title']
//footer//a[contains(@href, '/tags/')]

Regex Patterns: Last resort, brittle but sometimes necessary

Published: (\d{4}-\d{2}-\d{2})

CSS Selector Strategies

Basic Selectors

import * as cheerio from 'cheerio';

const html = `
  <article class="post">
    <h1 class="title">Article Title</h1>
    <span class="date">2026-02-23</span>
    <p class="content">Article body</p>
  </article>
`;

const $ = cheerio.load(html);

// Class selector
const title = $('.title').text();        // "Article Title"

// Element selector
const firstPara = $('p').text();         // "Article body"

// Attribute selector
const link = $('a[href="/about"]').text();

// Combination
const postTitle = $('article.post > h1.title').text();

Pseudo-selectors

// First/Last child
const firstPost = $('.post:first').text();
const lastPost = $('.post:last').text();

// nth-child
const secondPost = $('.post:nth-child(2)').text();

// Type selectors
const inputsOnly = $('input[type="text"]');

// Attribute patterns
const externalLinks = $('a[href^="http"]');  // Start with http
const downloads = $('a[href$=".pdf"]');      // End with .pdf
const webLinks = $('a[href*="example.com"]'); // Contains example.com

Advanced Selection Patterns

// Multiple selector groups
const headings = $('h1, h2, h3').map((i, el) => $(el).text()).get();

// Has filter (elements containing specific content)
const articlesWithAuthor = $('article:has(.author)');

// Parent/sibling selection (backwards from element)
const titleFromContent = $('p.content').prev('h1');
const nextArticle = $('article').next('article');

// find vs children
const allSpans = $('article').find('span');           // Descendant
const directChildSpans = $('article').children('span'); // Direct children

Handling Text Extraction

// Direct text
const title = $('h1').text().trim();

// Text with HTML preserved
const contentHtml = $('.content').html();

// Text from input values
const email = $('input[name="email"]').val();

// Text from data attributes
const id = $('[data-article-id]').data('article-id');
const id2 = $('[data-article-id]').attr('data-article-id');

Complex Extraction Patterns

Scraping Lists of Items

const extractArticles = (html: string) => {
  const $ = cheerio.load(html);
  
  return $('.article').map((index, element) => {
    const $el = $(element);
    
    return {
      title: $el.find('h2.title').text().trim(),
      author: $el.find('.meta .author').text().trim(),
      date: $el.find('.meta .date').attr('data-published'),
      summary: $el.find('.summary').text().trim(),
      tags: $el.find('.tags a')
        .map((_, tag) => $(tag).text().trim())
        .get(),
      url: $el.find('a.article-link').attr('href')
    };
  }).get();
};

const articles = extractArticles(html);
// Result: [{ title, author, date, summary, tags, url }, ...]

Nested Data Structures

const extractPostWithComments = (html: string) => {
  const $ = cheerio.load(html);
  
  const post = {
    title: $('h1.title').text().trim(),
    content: $('.post-content').text().trim(),
    comments: $('.comment').map((_, comment) => {
      const $comment = $(comment);
      return {
        author: $comment.find('.comment-author').text().trim(),
        date: $comment.find('.comment-date').attr('data-timestamp'),
        text: $comment.find('.comment-text').text().trim(),
        likes: parseInt(
          $comment.find('.likes span').text() || '0'
        ),
        replies: $comment.find('.reply').map((_, reply) => {
          const $reply = $(reply);
          return {
            author: $reply.find('.reply-author').text().trim(),
            text: $reply.find('.reply-text').text().trim()
          };
        }).get()
      };
    }).get()
  };
  
  return post;
};

Data Types and Parsing

const extractProductData = (html: string) => {
  const $ = cheerio.load(html);
  
  const product = {
    // String
    name: $('h1.product-name').text().trim(),
    
    // Number
    price: parseFloat(
      $('.price').text()
        .replace(/[$,]/g, '')  // Remove formatting
    ),
    
    // Integer
    rating: parseInt($('.stars').attr('data-rating') || '0'),
    
    // Boolean
    inStock: $('.stock-status').text().includes('In Stock'),
    
    // Date
    availableFrom: new Date(
      $('[data-available-date]').attr('data-available-date') || ''
    ),
    
    // Array
    images: $('.product-image img')
      .map((_, el) => $(el).attr('src'))
      .get(),
    
    // Enum-like value
    condition: $('[data-condition]')
      .attr('data-condition')
      ?.toLowerCase() ?? 'unknown',
    
    // Custom object
    details: {
      sku: $('[data-sku]').attr('data-sku'),
      manufacturer: $('.manufacturer-info span').text().trim()
    }
  };
  
  return product;
};

Handling Variable and Problematic Content

Optional Fields

// Safe extraction with defaults
const getTitle = ($: any, element: any) => {
  const title = $(element).find('h2.title').text().trim();
  return title || 'Untitled';
};

// Using optional chaining equivalent
const getPrice = ($: any, element: any) => {
  const priceText = $(element)
    .find('.price')?.text()?.trim();
  
  if (!priceText) return null;
  
  const parsed = parseFloat(priceText.replace(/[$,]/g, ''));
  return isNaN(parsed) ? null : parsed;
};

// Flexible selector fallback
const getDate = ($: any, element: any) => {
  // Try primary selector
  let date = $(element).find('.published-date').attr('data-iso');
  
  // Fall back to alternative selector
  if (!date) {
    date = $(element).find('.date').text().trim();
  }
  
  // Fall back to title attribute
  if (!date) {
    date = $(element).find('.metadata').attr('title');
  }
  
  return date || null;
};

Whitespace and Formatting Issues

// Clean excessive whitespace
const cleanText = (text: string) => {
  return text
    .replace(/\s+/g, ' ')           // Multiple spaces → single space
    .replace(/\n\t/g, '')           // Remove newlines/tabs
    .trim();
};

// Extract numbers from formatted strings
const extractNumber = (text: string): number | null => {
  const match = text.match(/[\d.]+/);
  return match ? parseFloat(match[0]) : null;
};

// Handle HTML entities
import { decode } from 'html-entities';
const decodeHtml = (html: string) => {
  return decode(html);
};

// Example usage
const title = cleanText($('h1').text());
const price = extractNumber($('.price').text());
const summary = decodeHtml($('.summary').html());

Dynamic and Hidden Content

// Elements loaded via JavaScript
const extractDynamicContent = async (page: any) => {
  // Wait for content to load
  await page.waitForSelector('.dynamic-content');
  
  const content = await page.evaluate(() => {
    return {
      items: Array.from(
        document.querySelectorAll('.item')
      ).map(el => ({
        title: el.querySelector('.title')?.textContent,
        data: el.dataset.value
      }))
    };
  });
  
  return content;
};

// Hidden elements (display: none)
const extractHidden = ($: any) => {
  // Cheerio includes hidden elements
  const hiddenValues = $('input[type="hidden"]')
    .map((_, el) => $(el).attr('value'))
    .get();
  
  return hiddenValues;
};

// Remove script/style tags before parsing
const cleanHtml = (html: string) => {
  const $ = cheerio.load(html);
  $('script, style').remove();
  return $.html();
};

Extraction Validation and Testing

Validation Schema

import { z } from 'zod';

// Define extraction contract
const ArticleSchema = z.object({
  title: z.string().min(1),
  author: z.string().min(1),
  date: z.string().datetime().or(z.null()),
  content: z.string().min(10),
  tags: z.array(z.string()).default([]),
  views: z.number().int().nonnegative().default(0)
});

type Article = z.infer<typeof ArticleSchema>;

// Extraction with validation
const extractAndValidate = (html: string): Article => {
  const $ = cheerio.load(html);
  
  const data = {
    title: $('h1').text().trim(),
    author: $('.author').text().trim(),
    date: $('[data-created]').attr('data-created') || null,
    content: $('.content').text().trim(),
    tags: $('.tags a').map((_, el) => $(el).text()).get(),
    views: parseInt($('.views').text().match(/\d+/)?.[0] || '0')
  };
  
  return ArticleSchema.parse(data);
};

Testing Extraction Logic

import { test, expect } from '@jest/globals';

describe('Article Extraction', () => {
  const sampleHtml = `
    <article>
      <h1>Test Article</h1>
      <span class="author">Jane Doe</span>
      <div class="content">Sample content here</div>
      <div class="tags"><a>testing</a><a>extraction</a></div>
    </article>
  `;

  test('extracts title correctly', () => {
    const $ = cheerio.load(sampleHtml);
    const title = $('h1').text();
    expect(title).toBe('Test Article');
  });

  test('handles missing optional fields', () => {
    const html = '<article><h1>Title Only</h1></article>';
    const $ = cheerio.load(html);
    
    const article = {
      title: $('h1').text(),
      author: $('.author').text() || null
    };
    
    expect(article.author).toBeNull();
  });

  test('extracts multiple items', () => {
    const html = `
      <div class="item">First</div>
      <div class="item">Second</div>
      <div class="item">Third</div>
    `;
    const $ = cheerio.load(html);
    
    const items = $('.item').map((_, el) => $(el).text()).get();
    expect(items).toHaveLength(3);
  });
});

Performance Monitoring

// Track extraction performance
const measureExtraction = async (
  url: string,
  extractor: (html: string) => any
) => {
  const response = await fetch(url);
  const html = await response.text();
  
  const start = performance.now();
  const data = extractor(html);
  const duration = performance.now() - start;
  
  console.log(`Extracted ${data.length} items in ${duration.toFixed(2)}ms`);
  return { data, duration };
};

// Benchmark different selector strategies
const benchmarkSelectors = () => {
  const html = /* large HTML document */;
  const $ = cheerio.load(html);
  
  console.time('CSS Selector');
  const css = $('article.featured h2.title');
  console.timeEnd('CSS Selector');
  
  console.time('XPath alternate');
  const alt = $('article').filter('.featured').find('h2.title');
  console.timeEnd('XPath alternate');
};

Best Practices

Be Specific: Use most specific selectors to avoid false matches
Plan for Changes: Web pages evolve; make selectors resilient
Validate Data: Always validate extracted data against expectations
Handle Errors: Gracefully handle missing, malformed, or unexpected content
Document Selectors: Comment why specific selectors are used
Test Extraction: Unit test your extraction logic
Monitor Performance: Track extraction speed and success rates

Discussion Questions

What’s the most complex data structure you’ve had to extract?
How do you handle selector brittleness when websites change?
What validation strategies have worked best for you?

Next Steps

With clean data extraction, you’re ready for Part 4: Storage Strategies and Data Persistence, where we’ll explore how to efficiently store and manage millions of records.

Part of the Web Crawling and Data Analysis Series

Previous: Part 2: Scraping Tools Comparison
Next: Part 4: Storage Strategies
Series Overview: Web Crawling and Data Analysis Series

Further Resources:

You also might like

About this post

This post was written by a human with AI assistance, which was used for research, outlining, editing and code suggestions. Claims, code samples and references were reviewed before publication, and the sources are cited so you can judge them yourself.