Cloud Scraping Architecture

Reviewed September 2026. Load-test architecture changes with approved fixtures and targets. Never use a scale test to multiply traffic against a third party.

A cloud scraping system should make traffic limits, failure states, data lineage, and cost visible. Adding workers is easy. Keeping those workers from repeating jobs, flooding one origin, or writing inconsistent records requires explicit boundaries.

Reference flow

schedule or request
        |
        v
 job planner --> durable queue --> fetch workers
        |               |                |
        |               |                v
        |               |          raw observation store
        |               |                |
        |               v                v
        |          retry / dead      parser + validator
        |                                |
        v                                v
 control state <---------------- normalized records
                                         |
                                         v
                                approved consumers

The planner owns when a job may run. The queue owns delivery. Workers fetch. Parsers interpret. Validators decide whether output is usable. Storage preserves lineage. Combining all of these in one process makes a page-layout change look like a network failure and makes safe replay difficult.

Define the job contract

A job needs a stable ID, source owner, target or source identifier, scheduled time, allowed representation, parser version, priority, maximum attempts, and deadline. Derive the ID from the logical work so a duplicate delivery reaches the same write boundary.

{
  "job_id": "sha256(source|target|window|parser)",
  "source": "approved-catalog-feed",
  "target": "product:1234",
  "parser_version": 12,
  "not_before": "2026-09-21T20:00:00Z",
  "deadline": "2026-09-21T20:10:00Z",
  "max_attempts": 3,
  "priority": "standard"
}

Queue delivery is usually at least once. Design the fetch and write path to tolerate duplicates instead of assuming that a message appears once.

Put policy before worker capacity

Maintain a central per-origin or per-source traffic budget. It should cover concurrency, requests per interval, minimum delay, permitted hours, and stop state. A global ceiling protects your infrastructure, while the per-origin ceiling protects each destination.

Workers must acquire capacity before a request. Autoscaling should increase queue consumers without increasing the approved origin budget. If fifty workers wake up, only the allowed number should reach the same origin.

Apply backpressure

When storage, parsing, or downstream services slow down, reduce fetch admission. A growing raw backlog should not trigger more collection. Use queue age, downstream saturation, and deadline importance to decide whether to scale consumers, pause planning, or shed low-priority work.

Separate HTTP and browser pools

HTTP workers are light and suitable for APIs, feeds, and static pages. Browser workers need browser binaries, more memory, longer startup, and stricter isolation. Route work to a browser only when the approved field requires client execution.

Recycle browser processes after a bounded amount of work or when health checks fail. Isolate credentials and cookies per source. Do not let one page navigate to arbitrary internal destinations.

Classify outcomes before retrying

• succeeded: an observation passed validation and the idempotent write completed.
• empty_valid: the source explicitly returned no record and that state is valid.
• transient_failure: network interruption, selected server errors, or temporary dependency failure.
• permanent_failure: invalid target, unsupported content, or a deterministic parser error.
• policy_stop: denial, authentication change, CAPTCHA, robots or terms review, or revoked approval.
• expired: the job can no longer meet its business deadline.

Retry transient failures with exponential backoff, jitter, and a deadline. Do not retry an access denial through a different identity or route. Send exhausted work to a dead-letter queue with the last sanitized error, observation reference, attempts, and owner.

Keep data layers distinct

Raw observation

Store the authorized response or a content-addressed evidence object under restricted access and retention. Record URL or source ID, collection time, response metadata, checksum, fetcher version, and job ID. Remove secrets and private headers.

Parsed source record

The parser should retain source-shaped values and a parser version. It should not silently invent business defaults. Unexpected structures enter quarantine with a bounded sample for diagnosis.

Normalized business record

Version unit conversion, categories, identity matching, and validation. Record input observation IDs and transformation versions. This allows a parser or normalization fix to replay saved evidence without fetching again.

Choose storage from access patterns

Object storage fits immutable raw observations and large artifacts. A relational or analytical store fits normalized records and joins. A fast control store can track leases, rate budgets, and job state. Avoid using one database table as queue, lock manager, blob store, and analytics warehouse.

Set lifecycle rules for raw content and browser artifacts. Deleting an object must not leave a normalized record claiming that evidence remains available. Keep lineage metadata after evidence expiry when the retention policy permits it.

Observe the whole path

Use one correlation identifier from plan through write. Emit structured state transitions rather than an unbounded text log.

• jobs planned, admitted, delayed, attempted, completed, expired, and dead-lettered;
• queue age and age by priority;
• requests, latency, status classes, bytes, retries, and policy stops by origin;
• parser coverage, null rate, schema errors, quarantine, and duplicate writes;
• freshness of validated records at the consumer boundary;
• worker CPU and memory, browser crashes, and deployment version;
• cost per validated usable record.

Exclude credentials, cookies, authorization headers, private page content, and raw personal data from logs. Operators can follow the observation reference under tighter access when investigation requires content.

Plan capacity with a simple model

If a source allows 30 concurrent requests and median service time is two seconds, worker count above that concurrency cannot increase safe throughput to that source. It can serve other origins or absorb variance. Browser jobs need a memory constraint as well:

max_browser_contexts = floor(
  (node_memory - operating_reserve) / tested_memory_per_context
)

Use high-percentile memory and duration from a realistic fixture set. Leave room for browser processes, sidecars, and failure artifacts. Load-test the queue, parser, and storage with saved fixtures rather than increasing third-party traffic.

Control cloud cost

Tag cost by source, job class, and stage. Track compute seconds, browser minutes, transfer, proxy use, raw storage, logs, retries, and analyst review. A cheap request that produces an invalid record is wasted spend, so optimize cost per validated record.

Use schedule tiers, cache stable responses, avoid repeat browser rendering, compress approved evidence, and expire artifacts according to policy. Budget alerts should pause low-priority planning before an unexpected bill threatens critical jobs.

Secure the system

Give each worker type a workload identity and the minimum queue, storage, and secret access. Restrict egress to approved destinations and block cloud metadata, loopback, and control-plane networks. Encrypt transport and storage. Rotate source credentials and test revocation.

Review source permission, terms, robots guidance, privacy, copyright, retention, and deletion. Treat a destination denial as policy state, not an engineering obstacle.

Deploy and recover by source

Roll out a new fetcher or parser to fixtures, then a small source cohort. Compare coverage, null distribution, quarantine, latency, and cost with the old version. Keep the version on each record so rollback does not mix meanings.

Recovery tests should cover queue outage, duplicate delivery, worker death after fetch, object-store failure, database timeout after write, browser crash, and parser deployment with bad output. Pause planning when necessary, preserve idempotent jobs, restore dependencies, and replay only work whose deadline and authorization remain valid.

Architecture review checklist

• Can one origin receive more traffic when worker count rises?
• Can every write be safely repeated?
• Does a denial enter a stop state?
• Can a parser be replayed without another request?
• Can an operator trace a normalized value to its observation and versions?
• Do dashboards show freshness and valid output, not only request success?
• Can the team attribute cost and delete every data layer on schedule?