Start here
Crawling and Indexing Explained for Site Owners
A plain-language look at how search engines discover, process and store web pages, and why that sequence is the base for every technical check you run.
Technical Foundations
Crawling and Indexing
How search engines find pages and decide what to keep.
Crawl Budget for Small and Large Sites
Find out whether crawl limits actually matter for a site of your size, what tends to waste crawling, and which fixes are worth your time.
Finding Pages That Are Not Indexed and Why
A step-by-step diagnostic for pages missing from search results: how to confirm the gap, check access and tags, and weigh quality and duplication.
Fixing Broken Links and Error Pages During an Audit
How to locate dead links and error pages on your site, and how to decide whether each one should be repaired, redirected or simply removed.
How to Read a Robots File Without Breaking Your Site
Learn what the crawl rules file says, how to edit it safely, and the common mistakes that accidentally block whole sections or even an entire site.
How to Spot Duplicate Content on Your Own Site
Find near-identical pages caused by templates, URL parameters and copied text, and learn which kinds of duplication are harmless and which deserve a fix.
Noindex and Canonical Tags in Plain Language
How the noindex and canonical tags work, how they differ, and how to choose between hiding a page and consolidating it with a preferred version.
Redirects Explained: Permanent, Temporary and Chains
Learn when to use permanent and temporary redirects, how to find chains and loops that slow visitors and crawlers, and how to clean them up safely.
Soft Errors and Thin Pages That Waste Crawling
Learn how pages that look fine to visitors can signal an error or empty result to crawlers, how to find them, and how to fix or remove them properly.
XML Sitemaps: What They Do and What to Include
What an XML sitemap lists, which URLs belong in it and which to leave out, and how to check the file for errors so it actually helps search engines.
Related topics