Search Results Degraded After Reindex on a Large Catalog
An Acquia Cloud site with roughly 400,000 indexed nodes uses Acquia Search (Solr). After a content model change, the team ran a full reindex during business hours. Immediately afterward, search result pages are slow (multi-second) and some queries return incomplete results. The Solr index reports a high number of deleted documents relative to live documents. The web tier is healthy; the slowdown is isolated to search-backed pages. The team also recently lowered the Solr commit interval and increased the batch size in the indexer to speed up the reindex. The catalog is read-heavy; peak traffic is 10am–2pm.
1. Diagnose the cause of slow and incomplete search results and explain the role of the recent indexer tuning.
Read the worked answer
The symptoms - slow queries, incomplete results, and a high deleted-to-live document ratio - point to index bloat and commit behavior rather than web-tier problems. A full reindex on 400k nodes creates a large volume of deleted (superseded) documents; Solr must merge segments and eventually expunge them, and until it does, queries scan more data and can behave inconsistently. Running the reindex during peak hours compounds the load.
The recent tuning is likely a contributing cause, not a fix. Increasing batch size speeds indexing but produces larger segments and more merge pressure; lowering the commit interval increases commit frequency, which is expensive and can cause more frequent segment churn and visibility inconsistencies (documents appear/disappear between commits). 'Incomplete results' is consistent with queries hitting the index mid-commit or during merges.
So the diagnosis is: aggressive indexing parameters plus a peak-hours full reindex produced index bloat and merge pressure that the index hasn't yet recovered from.
2. Recommend a remediation sequence and the tuning changes you would make.
Read the worked answer
Remediation, in order:
- Stop indexing immediately so merges can catch up; let the index settle and confirm the deleted-document ratio drops.
- Run an optimize/expunge to purge deleted documents and consolidate segments - but schedule it off-peak, since it is I/O heavy.
- Re-run any remaining indexing during a low-traffic window (overnight), in controlled batches, verifying query latency and result completeness as you go.
Tuning:
- Restore a sane commit interval; do not lower it purely for reindex speed. Commit frequency should balance visibility needs against segment churn.
- Reduce batch size to a level that keeps merges manageable; large batches trade indexing speed for query-time cost.
- Consider indexing in phases (by content type or ID range) so the index never holds a huge backlog of deletions at once.
- After the reindex, run an explicit expunge and verify the deleted-doc ratio returns to normal.
Finally, confirm the content model change itself didn't alter field types or analyzer configuration in a way that invalidates existing documents.
3. How would you change the operational process to avoid peak-hour search degradation in the future?
Read the worked answer
Treat reindexing as a scheduled maintenance operation, not an ad-hoc task: run full reindexes in a defined low-traffic window, with a runbook that includes pre-checks (index health, deleted-doc ratio), in-progress monitoring (query latency, error rate, index size), and post-checks (result completeness, expunge). Add alerting on Solr query latency and deleted-document ratio so degradation is caught early. For content model changes, test the reindex on Stage first and measure query latency before promoting. Keep peak-hour changes limited to incremental indexing.