Crawl Budget and Indexation in a 180,000-Listing Marketplace
TradeLot is a B2B equipment marketplace with 180,000 active listings and 2.3 million URLs in Google's index. Sellers can filter by brand, condition, year, location, and price, generating parameter URLs like /listings?brand=cat&condition=used&page=3. GSC crawl stats show Googlebot made 1.1 million requests in the last 90 days; 68% went to filtered URLs and 22% to expired listings that return 200 with 'no longer available.' The XML sitemap includes all active listings plus 40,000 expired listings and 12,000 filter combinations. New listings take an average of 9 days to be indexed; the team wants to get to 48 hours. The site uses client-side JavaScript to render listing details, including price and description; Google renders it but the initial HTML is mostly empty. Two engineers can work one sprint (two weeks).
1. Prioritize the fixes for one two-week sprint. What do you change first, what waits, and what impact do you expect?
Read the worked answer
Prioritize by crawl waste and indexation speed. First, expired listings: 22% of 1.1 million requests go to 200-status dead pages. Return 410 for permanently gone listings, 404 if they may return, and 301 to the closest category if the listing has backlinks. Remove them from the sitemap immediately; this is a template change, not a content project. Second, sitemap hygiene: 12,000 filter combinations and 40,000 expired listings inflate the sitemap and dilute crawl priority. Keep active listings and canonical category pages only. Third, faceted URLs: do not robots.txt all filters if they receive internal links-Google can still index the URL without crawling it. Use rel=canonical from filter combinations to the base category, and noindex only low-demand combinations that are not linked. Fourth, JavaScript rendering: price and description are critical; server-render or pre-render those fields first, not the entire page. Expected impact: crawl requests shift toward active listings; time-to-index should fall from 9 days toward 48-72 hours. Measure at 14 and 30 days.
2. How should expired listings and faceted URLs be handled technically? Compare 410, 404, 301, canonical, noindex, and robots.txt.
Read the worked answer
For expired listings, 410 is the cleanest signal for permanently removed inventory; 404 is acceptable, but 410 is more explicit. A 200 'no longer available' page is a soft 404: it wastes crawl budget and can create thin-content quality issues. If the listing has external links or seasonal demand, 301 to the category or a similar active listing; otherwise 410. For facets, separate valuable facets (brand, condition) that match search demand from infinite combinations (price sliders, page 3 of sorted results). Allow crawling of the valuable ones with self-canonical; canonicalize the rest to the unfiltered category. Do not use robots.txt for URLs that are linked internally-if Google cannot crawl, it cannot see the canonical, and the URL may still be indexed. Use noindex only when the page has no search value and no link equity. For sort parameters, disallow in robots.txt only if they are never linked as canonical. Document the rule set in a matrix so engineers apply it consistently.
3. What should the team measure after the sprint to prove the fixes worked, and what would count as failure?
Read the worked answer
Measure crawl distribution, not just rankings. In GSC Crawl Stats, compare requests by directory and parameter before and after. Target: filtered URLs down from 68% to under 30% of requests; expired-listing requests down from 22% to under 5%. Index coverage: active listings indexed versus excluded; expired listings removed. Time-to-index: sample 200 new listings weekly and check URL Inspection or log files; target median under 72 hours, then 48. Organic performance: impressions and clicks per active listing and category pages. Server logs: 200 versus 410/404 ratios; Googlebot hits per active listing. Business metric: listing views and inquiries per new listing in the first 14 days. Set a 30-day checkpoint; if time-to-index does not improve, the sitemap or internal links are still the bottleneck. Do not use total indexed count as the goal-2.3 million indexed URLs is not success if most are filters.