First, confirm the page is really missing
Before troubleshooting, make sure there is a problem. A page that is not showing for a search you tried is not necessarily absent from the index. It may be indexed but ranking too low to be seen.
Reasonable ways to check:
- Search for a distinctive phrase from the page in quotation marks, and check whether the page appears.
- Search for the full address of the page, which often shows whether it is known.
- Use the inspection or coverage reports in your search tools if you have them set up. They usually tell you whether a specific URL is indexed and, if not, give a reason.
Results can vary by location, device and time, so treat a single search as a clue, not proof. The reports from your search tools are generally more reliable than hand searches.
Step one: can the page be found?
Discovery problems are common for new or weakly linked pages.
- Is the page linked from at least one other crawlable page? A page with no links pointing to it is an orphan, and it may only be found through a sitemap, if at all.
- Is it listed in your XML sitemap?
- Is the link a normal link that a crawler can follow, rather than something that only works through a script or a form?
If the page is very deep in your structure, it may be visited late. Adding a link from a prominent page is often the simplest fix.
Step two: can the page be fetched?
Open the page and check what the server returns.
- The page should return a normal success code. An error code, a redirect or a timeout will prevent it from being kept.
- Check that the page does not require a login, a cookie choice or a location choice before showing content.
- Check the robots file. If the path is disallowed, crawlers are asked not to fetch it. Keep in mind that a URL blocked this way can occasionally still be known to a search engine without its content.
- Check for server problems. Repeated errors or very slow responses can lead to fewer visits.
Step three: is the page telling search engines to leave it out?
Look at the page source or the response headers for instructions.
- A noindex directive, either in a meta tag or in a header, asks search engines not to store the page. These are sometimes left over from a development stage or switched on by a site setting.
- A canonical tag pointing to a different URL says another address is the preferred version. If it points somewhere you did not intend, your page may be treated as a duplicate.
- Check for accidental changes after a redesign or migration, since staging settings are a common culprit.
If you see a noindex directive on a page that is also blocked in the robots file, remember the crawler cannot read the tag when it cannot fetch the page. Sort out which instruction you actually want.
Step four: can the content be read?
Open the page without being logged in and check that the main text is present. If the content only appears after scripts run, a crawler may take longer to see it or may miss parts of it. Check that important text is not inside images and that nothing essential is behind a click or tab that requires interaction.
Step five: is the page too similar to another?
Search engines tend to keep one version of near-identical content. Pages that differ only by a filter, a tracking tag, a printer-friendly layout or a small template change may be folded into one. Review how to spot duplicate content, and check that each page has its own purpose and its own text. Pages where this often happens include product variants, location pages with swapped place names and tag archives.
Step six: is the page worth keeping?
If every technical check passes, the remaining explanation is often quality. Search engines choose not to store everything they crawl. Pages tend to be left out when they:
- Have very little content, or content that is mostly boilerplate.
- Look like soft errors, such as an empty search result page with a normal code.
- Offer nothing different from many other pages on the web.
- Sit on a site with a large amount of low-value material.
Ask honestly whether the page answers a real question better or differently than what already exists. Improving it, merging it with a related page or removing it may be better than pushing for indexing. Thin content and pruning decisions are covered elsewhere on this site under content pruning.
Reading common status messages
Search tools use wording such as “discovered but not indexed” or “crawled but not indexed”. Roughly speaking, the first suggests the page is known but has not yet been fetched, which can point to limited priority or crawl capacity, while the second suggests it was fetched and then judged not worth storing at the time. Exact meanings and labels change, so read the explanation your tool provides.
What to do after a fix
- Fix the cause you found, one change at a time where possible.
- Make sure the page is linked internally and in the sitemap.
- Use the request-indexing feature of your search tools if available, but do not repeat it constantly.
- Wait. Recrawling and reprocessing can take days or weeks.
- Recheck and note the result in your change log.
Not every page deserves to be indexed. Login pages, thank-you pages, internal search results and duplicates are better left out. The aim is to make sure the pages you want in results are the ones that are there, not to maximize the count.