The short version
A search engine cannot show a page it has never seen. Before a page can appear in results, it generally goes through three stages: it is discovered, it is fetched and processed, and it is stored in a very large database often called the index. Only after that is it eligible to be shown for a search.
These stages are separate, and a page can fail at any of them. Most technical checks on a site are really checks on one of these stages, so it helps to keep the sequence in mind.
Stage one: discovery
Search engines run programs, usually called crawlers or bots, that follow links from page to page. Discovery typically happens in a few ways:
- A crawler follows a link from a page it already knows about.
- The site provides a list of URLs, most often in an XML sitemap file.
- Another site links to yours.
- The URL was known from an earlier visit.
This is why links inside your own site matter so much. A page that no other page points to is hard to find, which is the problem behind orphan pages. Good internal linking gives crawlers a path to everything you care about and signals which pages you consider important.
Discovery does not guarantee a visit. A search engine decides which known URLs to fetch and how often, and it may take days or longer for a new page to be visited. Timing varies a lot between sites.
Stage two: crawling and processing
When a crawler fetches a URL, your server answers with a status code and the content. The status code tells the crawler what happened:
- A 200 code means the page was returned normally.
- A 301 or 302 code means the page redirects elsewhere.
- A 404 or 410 code means the page is missing or gone.
- A 5xx code means the server had a problem.
After fetching, the search engine reads the page. For pages that depend heavily on scripts to show their content, it may need an extra step of running that code, which can happen later than the first fetch. If your main text only appears after complicated scripts run, it is worth checking that the content is actually visible to a crawler, not only to a person in a browser.
Crawling is also limited by politeness. Crawlers try not to overload a server, and a slow or unreliable server may be visited less. Your server response time and general reliability can therefore affect how much of your site gets fetched. A site served over a properly configured https setup is also the normal expectation today.
Stage three: indexing
Indexing means the search engine analyzes the page and decides whether to store it. It looks at the text, the title, headings, images, links and structure. It also tries to work out which URL is the “main” version when several pages look alike.
Not every crawled page is indexed. Common reasons include:
- The page has a noindex instruction, which asks search engines to leave it out.
- The page is a near-copy of another page, so a different URL is chosen as the representative.
- The page has very little useful content, or looks like an error page even though it returns a 200 code.
- The search engine simply judged the page not valuable enough to keep, which does happen and is not always explainable.
Being indexed is also not the same as ranking. An indexed page is merely eligible. Whether and where it appears depends on many other things, including how well it matches search intent, and nobody can promise a particular outcome.
The controls you have
Site owners have a handful of levers, each acting at a different stage:
| Control | Stage it affects | What it does |
|---|---|---|
| Links and sitemap | Discovery | Help crawlers find URLs |
| Robots file | Crawling | Asks crawlers not to fetch certain paths |
| Status codes and redirects | Crawling | Tell crawlers what happened to a URL |
| Noindex tag | Indexing | Asks search engines not to store a page |
| Canonical tag | Indexing | Suggests which of several similar URLs to keep |
A frequent source of confusion is mixing these up. Blocking a path in the robots file stops fetching, but it does not reliably keep a URL out of results, and it also stops the crawler from ever seeing a noindex tag on that page. If you want a page removed from the index, it generally needs to be crawlable so the instruction can be read.
Why site structure feeds all of this
Crawlers move through your site the way a visitor with a very good memory and no patience would. A clear site architecture, sensible URLs and a shallow click depth make pages easy to reach. Messy structures, endless parameter combinations and long redirect trails make crawling slower and less complete.
For small sites this seldom causes trouble. Larger sites, or sites that generate many near-identical URLs, are more likely to see important pages visited late or not at all.
A simple way to check each stage
When you audit a site, ask the questions in order:
- Can a crawler find the page? Is it linked, and listed in the sitemap if you use one?
- Can it fetch the page? Does it return the right status code, and is it allowed by the robots file?
- Can it understand the page? Is the main content present, and not hidden behind a script or login?
- Is the page allowed and worth indexing? Is there a noindex tag, a canonical pointing elsewhere, or very thin content?
Working in this order saves time, because a problem at an early stage makes the later checks irrelevant. It is also a useful frame for a wider website evaluation, since many symptoms that look like content problems turn out to be access problems.
What to expect
Changes do not take effect instantly. After you fix a problem, search engines have to come back, fetch the page again and reprocess it. That can take days or weeks, and the timing is outside your control. Make your fixes, keep a note of what you changed and when, and recheck after a reasonable wait before trying something else.