Retailer Rendering Change Breaks a Daily Price Crawl

The 11% of HTTP 200 responses with no product block point first to a rendering failure, not a blocking failure: the HTML shell is being fetched, but the product data is injected by JavaScript. The 8% HTTP 429 responses indicate rate limiting, likely worsened by 40 concurrent workers hitting the same host. The remaining gap is probably a mix of variant-URL duplication, extraction breakage, and retries that consume the rate budget.

First fix: inspect whether product data is available in embedded JSON-LD, a __NEXT_DATA__-style state object, or a public JSON endpoint before adding a headless browser. This is the cheapest and fastest change, and it avoids the cost and fragility of full rendering. Second fix: use a headless browser only for retailer A's product template, with a fallback to static parsing and a per-page timeout. This is appropriate because the site now requires client-side execution, but it must be scoped to avoid turning the whole crawl into a browser farm. Third fix: reduce concurrency per host and honor the 2-second crawl-delay with a token bucket. Rotating residential proxies to evade 429s is not appropriate: it converts a rate-limit problem into an adversarial one and may breach the retailer's terms or legal approval.

I would not treat all 61% failures as one bug. Separate metrics for fetch status, render success, extraction success, and schema validation will show whether the remaining loss is technical or content-related.

Crawl-delay: 2 means at most one request every two seconds to that host, or 1,800 requests per hour per host, regardless of how many workers or proxies exist. If retailer A has roughly 180,000 product pages, a single compliant stream needs 180,000 / 1,800 = 100 hours. Even a four-hour window allows only 4 x 1,800 = 7,200 requests to that host. The deadline is therefore not feasible for a full refresh of A at that crawl-delay. Adding proxies or workers to multiply the request rate would violate the host's stated crawl-delay and should be rejected.

The schedule must be redesigned around incremental collection: use sitemaps with lastmod, category pages, and cached product IDs to fetch only new or changed SKUs; prioritize high-revenue and high-volatility SKUs; and split retailers into separate queues with per-host concurrency limits and shared retry budgets. The 02:00 start can remain for hosts without a strict delay, but A needs a rolling crawl that finishes over 24 hours and feeds a last-known-good price field with a timestamp. The report should show coverage and freshness by retailer rather than implying all prices are current. If merchandising needs same-day coverage for A, the business must negotiate access or use an approved feed; it cannot be solved by concurrency alone.

The key distinction is between a product and a sellable variant. Two URLs for the same shirt in blue and red are not duplicates at the variant level, but they are duplicates at the product level if the report only needs one product record. The pipeline should normalize URLs by removing tracking parameters, sorting query parameters, and resolving canonical tags, then extract a stable product identifier where available, such as a retailer SKU, GTIN, or embedded product ID. If no stable ID exists, build a composite key from normalized brand, product name, and category, and keep variant attributes separately.

Deduplication should happen in stages: exact URL match, exact external ID match, then fuzzy product-level match with review for borderline cases. Variant attributes such as size, color, and width must be preserved in a child table so legitimate options are not dropped. The output should have one product record with many variant records, each with its own price and availability. I would also add a reconciliation check: for a sample of product pages, confirm that the number of captured variants matches the page's option selectors. This catches both over-merging, which hides real options, and under-merging, which inflates SKU counts and distorts averages.