Web Scraping Data Quality Guide

Reviewed September 2026. Quality targets should be defined and measured for each dataset rather than borrowed from generic benchmarks.

Write a data contract

Define required fields, types, identifiers, units, allowed values, freshness, uniqueness, and acceptable missingness before collection. Include the source URL, collection time, parser version, and validation result so every record has provenance.

Make the contract executable

Write the contract where workers and downstream users can test it. It should say which fields are required, whether a null is allowed, how dates and money are normalized, what makes two records the same entity, and when a record becomes too old to publish. Assign an owner and a version. A changed parser should either continue meeting the same contract or publish a reviewed contract change.

{
  "schemaVersion": "listing-v2",
  "required": ["sourceUrl", "observedAt", "externalId", "title"],
  "freshnessHours": 24,
  "price": { "currency": "ISO-4217", "minorUnits": "integer-or-null" }
}

Store the original source reference, observation time, collector version, parser version, normalization version, and validation outcome with the normalized record. A provenance record lets an analyst distinguish “not available on the source” from “parser failed to find it.”

Separate the stages

Keep raw capture, parsing, normalization, validation, and publication distinct. Preserve raw observations or integrity hashes under an approved retention policy. A layout change should fail validation or enter quarantine, not silently become valid business data.

Quality dimensions

• Completeness: required records and fields are present.
• Accuracy: values match a verified source sample.
• Consistency: equivalent values use the same representation.
• Conformity: types, formats, units, and ranges follow the contract.
• Uniqueness: duplicate entities and observations are controlled.
• Timeliness: data arrives within the documented freshness target.
• Integrity: relationships and identifiers remain valid.

Validation controls

• Reject or quarantine missing identifiers and impossible values.
• Normalize currency, time zones, units, and Unicode explicitly.
• Use stable entity keys and document fuzzy-matching thresholds.
• Compare counts and distributions with a reviewed baseline.
• Sample records for human verification after every material parser change.

Test for structural drift

Save representative source fixtures and expected parsed records. Test selectors, transformations, edge cases, and empty states. Alert on coverage loss, unexpected nulls, new categories, and abrupt distribution changes. Version the parser so downstream users can explain a change.

Validate at more than one layer

Use explicit transformations. Parse a monetary amount into integer minor units with a named currency, parse a date with an input time zone, and preserve the original string where allowed. Avoid silently coercing an invalid value to zero or a current timestamp; that makes a source change look like valid data.

Write validation that returns evidence

A validator should return a result that an operator can act on, rather than a bare true or false. Include the schema version, field path, rule name, observed value only when retention permits it, and a reason code. A record may have more than one failure; preserve all useful reasons so a layout change does not appear as a series of unrelated defects.

function validateListing(record) {
  const errors = []
  if (!record.externalId) errors.push({ field: 'externalId', rule: 'required' })
  if (!Number.isInteger(record.priceMinor) || record.priceMinor < 0) {
    errors.push({ field: 'priceMinor', rule: 'non_negative_integer' })
  }
  if (!/^https:\/\//.test(record.sourceUrl || '')) {
    errors.push({ field: 'sourceUrl', rule: 'https_url' })
  }
  return { ok: errors.length === 0, schemaVersion: 'listing-v2', errors }
}

Keep validation pure where possible. It should not call a remote service, mutate a record, or hide a failure by inserting a default. Referential checks and duplicate matching may require a separate stage with its own timeout and audit record.

Measure quality over time

Set a baseline from reviewed historical data, then track completeness, invalid-record rate, duplicate rate, freshness delay, and manual-sample accuracy by source and parser version. A global mean hides a broken source. Use count and percentage together: a 20% null rate from five records needs a different response from the same rate across millions.

1. Draw a random or stratified sample from each published batch.
2. Compare it with the authorized source observation using a written review rubric.
3. Record reviewer agreement, error category, severity, and parser version.
4. Use the results to set alert thresholds and regression tests.
5. Recheck thresholds after a source, contract, or volume change.

Deduplication is a business rule, not simply a hash. Exact source IDs can support deterministic merges. Names and addresses need a documented normalization and confidence rule, with ambiguous candidates retained for review. Never merge unrelated records merely to improve a duplicate-rate metric.

Monitor production quality

Track source availability, collection delay, parse coverage, schema failures, missingness, duplicate rate, quarantine volume, manual-review results, and freshness. Break these down by source and parser version; a global average can hide one broken feed.

Handle corrections

Keep a clear path to correct or replay affected observations. Determine which parser version and time range produced bad data, stop publication, patch against a fixture, revalidate, and replay idempotently. Tell downstream users when a correction changes material results.

Correct without losing evidence

When a quality alert fires, first preserve the batch identifier and the evidence allowed by the retention policy. Identify affected source URLs, collection window, worker build, and parser version. Pause downstream publication if the contract requires it, add a fixture that reproduces the problem, and test the correction before replaying idempotent work. Mark superseded output with lineage rather than silently overwriting an unexplained value.

Incident runbook

1. Open an incident with source, batch, parser version, affected fields, and customer impact.
2. Pause publication or the source when its quality threshold requires it.
3. Compare a reviewed raw observation with the normalized record and add a regression fixture.
4. Patch the parser, transformation, or contract; run unit and integration checks.
5. Replay only the identified idempotent jobs, then sample the corrected output.
6. Notify downstream users when a published field or historical aggregate changed.

Governance

Collect only approved data, apply privacy and retention rules, restrict access, and delete data when its purpose expires. Review terms, copyright, robots guidance, and access controls for each source. Quality does not make an unauthorized dataset acceptable.

For the surrounding collection architecture, see web scraping at scale.