Technical SEO and Crawlability
Technical questions test whether you can diagnose why a correct page is not indexed or not ranking: crawl directives, canonicals, redirects, rendering, and Core Web Vitals.
Crawl and index controls
- robots.txt disallows crawling. It does not remove a URL from the index, and a disallowed URL keeps signals it already earned.
- A meta robots noindex or X-Robots-Tag allows crawling but drops the page from results - this is the correct de-indexing tool.
- Canonicals consolidate duplicates. They are hints, and a canonical pointing at a noindexed or redirected URL is a conflict.
- XML sitemaps should list canonical, indexable, 200-status URLs only.
Redirects and status codes
A 301 passes signals and is the default for permanent moves; 302 is for temporary ones. Long chains slow crawling and dilute signals - collapse them to a single hop. Use 404 for gone content and 410 when you want to be explicit, and never redirect everything to the homepage.
Rendering and performance
Google renders JavaScript, but content injected only after a click may never be seen. Core Web Vitals thresholds for "good": LCP at or under 2.5 seconds, INP at or under 200 milliseconds, CLS at or under 0.1. Blocking CSS or JS in robots.txt breaks rendering - a classic self-inflicted wound.
Worked example
A client's category pages vanish from the index. The Search Console page indexing report shows "Excluded by noindex." A theme update shipped a staging noindex tag to production. Removing the tag, requesting validation, and confirming with URL Inspection restores indexing. The fix took minutes, but only after checking directives before questioning content quality.
Traps
Using robots.txt to deindex. Stacking noindex and canonical tags on the same page. Omitting hreflang return links. Letting faceted and paginated URLs multiply crawl paths. Assuming a sitemap guarantees indexing.