JavaScript Web Scraping Guide

Reviewed September 2026. Confirm runtime and library versions against their official documentation, and collect only data you are authorized to use.

Choose the simplest fetch path

Use ordinary HTTP requests for server-rendered HTML or JSON. Add browser automation only when the required, permitted content depends on client-side execution. This keeps resource use, failure handling, and tests much simpler.

Before collecting, document the source, purpose, allowed fields, rate and concurrency limits, retention, and deletion path. Review terms, access controls, privacy, copyright, and the Robots Exclusion Protocol. robots.txt is not authorization by itself.

HTTP collection with bounded concurrency

const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), 15_000)

try {
  const response = await fetch(url, {
    signal: controller.signal,
    headers: { 'user-agent': 'ExampleResearchBot/1.0 [email protected]' }
  })
  if (!response.ok) throw new Error('HTTP ' + response.status)
  const html = await response.text()
} finally {
  clearTimeout(timeout)
}

Use a queue with a per-origin concurrency cap. Retry only transient network failures, 429 responses with an appropriate delay, and selected 5xx responses. Bound attempts and add jitter. Do not retry authentication failures or explicit access denials through new identities.

Give jobs a contract

A queue message should include a stable job ID, source URL, approved request policy, parser version, attempt count, and deadline. The worker should return a typed outcome such as success, retryable_network_error, invalid_response, or policy_stop. A job must be safe to execute again: use an idempotency key or uniqueness constraint where results are stored.

const job = {
  id: crypto.randomUUID(),
  url: 'https://source.example/catalog',
  parserVersion: 'catalog-v3',
  maxAttempts: 3,
  deadline: new Date(Date.now() + 60_000).toISOString()
}

function retryDelay(attempt) {
  const ceiling = 10_000
  const delay = Math.min(ceiling, 250 * 2 ** attempt)
  return delay / 2 + Math.random() * delay / 2
}

Apply a shared rate limiter before dispatch, rather than leaving every worker to make a local decision. Separate origins so one slow source cannot occupy all workers. Honor a documented retry-after delay where your authorized integration permits retrying.

Parse and validate separately

Keep selectors and transformations outside the fetcher. Parse saved fixtures in unit tests and validate the result against a schema before storage.

import * as cheerio from 'cheerio'

function parseProduct(html, sourceUrl) {
  const $ = cheerio.load(html)
  return {
    sourceUrl,
    name: $('h1').first().text().trim() || null,
    priceText: $('[data-price]').first().text().trim() || null
  }
}

Preserve the raw observation or a content hash, source URL, collection time, parser version, and validation result. Quarantine a record with missing identifiers or ambiguous values instead of turning an empty selector into valid business data.

Use a schema at the publish boundary

Selectors extract strings, not trustworthy domain values. Normalize Unicode, whitespace, dates, currency, units, and URLs explicitly. Validate required fields, ranges, enumerations, and relationships before a record is published. Keep a reason code for each quarantine outcome so an operator can distinguish a page redesign from a one-off missing value.

function validateProduct(record) {
  if (!record.sourceUrl?.startsWith('https://')) return { ok: false, reason: 'source_url' }
  if (!record.name || record.name.length > 500) return { ok: false, reason: 'name' }
  return { ok: true, value: record }
}

Save representative pages as fixtures, including missing fields, alternate layouts, empty states, and error documents. A parser test should compare the complete normalized record and assert that an unexpected layout is quarantined rather than silently published.

Handle response failures at the fetch boundary

Inspect a response before parsing it. An HTML error page can have a 200 status, and a redirect can lead to a sign-in page or a different locale. Set a response-size limit when the client supports one, check the content type where the source contract requires it, and keep a short diagnostic reference for quarantined responses. Do not retain complete response bodies in general logs.

async function fetchHtml(url, signal) {
  const response = await fetch(url, { signal, redirect: 'follow' })
  const contentType = response.headers.get('content-type') || ''
  if (!response.ok) return { kind: 'http_error', status: response.status }
  if (!contentType.includes('text/html')) return { kind: 'unexpected_type', contentType }
  const html = await response.text()
  if (html.length > 2_000_000) return { kind: 'response_too_large' }
  return { kind: 'ok', html }
}

Classify an abort, DNS error, connection failure, HTTP status, parse failure, and validation failure separately. The queue can retry a transient network outcome within its budget. A source page that no longer satisfies the extraction contract should enter review with its parser version and collection time.

When a browser is justified

Playwright or Puppeteer can render approved JavaScript applications, interact with controls, and capture traces. Block unnecessary resources only when doing so cannot change the data being measured. Use fresh contexts for isolation and close pages and browsers in a finally block.

import { chromium } from 'playwright'

const browser = await chromium.launch()
try {
  const context = await browser.newContext()
  const page = await context.newPage()
  await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30_000 })
  const heading = await page.getByRole('heading').first().textContent()
} finally {
  await browser.close()
}

Pin the browser package and review its official release notes. Do not add stealth patches, CAPTCHA solving, or fingerprint spoofing to overcome a site's controls.

Keep browser work isolated

Use a browser only for a documented rendering need. Limit page count, navigation timeout, response size where supported, and artifact retention. Capture a trace, screenshot, or HTML snapshot only under an approved policy because those artifacts can contain personal data or credentials. Browser jobs belong in a separate pool from ordinary HTTP jobs because their CPU and memory profile is different.

Test and operate the collector

1. Unit-test parsing and normalization against versioned fixtures without the network.
2. Run an integration test against a service you control to cover redirects, timeouts, retry classification, and log redaction.
3. Run a low-rate authorized canary for each source after a parser or runtime change.
4. Alert on queue age, status changes, null rates, schema failures, and duplicate spikes by source and parser version.
5. Pause a source when policy signals, error rates, or structural drift cross its reviewed threshold.

Observability

• Queue age and jobs completed.
• Status and application error classes per origin.
• Latency percentiles, timeouts, retries, and bytes.
• Parse coverage, schema failures, duplicate rate, and freshness.
• Parser version and structural-drift alerts.

Keep secrets, authorization headers, cookies, and personal payloads out of logs. Use correlation identifiers and short retention.

Release changes safely

Run fixture tests before a deployment, then send a small authorized canary through the same queue and storage path. Compare field coverage and validation results with the prior parser version. Roll back the parser or pause the source when the canary exposes a structural change. Keep the worker build and parser version on every result so a later correction has a defined replay range.

Production checklist

• Pin Node.js and dependency versions; consult the official Node.js API documentation.
• Use explicit timeouts, bounded queues, and graceful shutdown.
• Test fetch and parser behavior independently with deterministic fixtures.
• Make jobs idempotent and storage writes safe to replay.
• Stop or seek permission when a destination rejects the workload.