Crawl Budget for Small and Large Sites

What crawl budget means

Search engines have finite resources. They cannot fetch every page on the web every day, so they make choices about which sites to visit, which URLs to fetch and how often. The rough amount of fetching a search engine is willing to do on your site over a period is commonly called the crawl budget.

It is a useful idea, but it is also one of the most over-discussed topics in technical SEO. For many sites it is simply not the thing holding them back.

The two things behind it

Crawl budget is commonly described as the combination of two forces:

• Crawl capacity. How much fetching your server can handle without slowing down or failing. A fast, reliable server generally allows more. A server that returns errors or responds slowly tends to be visited more cautiously.
• Crawl demand. How much the search engine wants to fetch. Pages that are popular, important or frequently updated tend to be revisited more. Pages that look stale, duplicated or low value tend to be visited less.

You do not set the budget directly. You influence it by keeping the server healthy and making your URLs worth fetching.

Does your site need to worry?

Exact thresholds are not published and they change, so treat the following as a rough guide.

Small sites. If your site has up to a few thousand pages, is reasonably fast, and new pages tend to appear in results within a normal period, crawl budget is very unlikely to be your problem. If a page is missing from results, the cause is much more likely to be a noindex tag, a block in the robots file, thin content or a lack of links. Chasing crawl budget here usually wastes effort.

Large sites. Sites with many thousands or millions of URLs, or sites that publish or change a lot of pages daily, are where crawl efficiency can matter. Examples include big online stores, listing sites and publishers with long archives. Here the concern is that crawlers spend time on low-value URLs and reach important pages late.

Sites that generate many URLs. Size in “real” pages is not the whole story. A small catalog can produce an enormous number of URLs through filters and sorting options. A site with 500 products might expose far more crawlable addresses than it has real pages. That behavior can make a modest site behave like a large one.

What commonly wastes crawling

If you do have a concern, these are the usual sources of waste:

• Parameter and filter combinations. Sort orders, filters, session identifiers and tracking tags can create many addresses that show nearly the same content. This is closely related to url parameters and faceted navigation.
• Duplicate pages. The same content reachable at several addresses, such as with and without a trailing slash, or on both http and https versions.
• Long redirect chains. Each hop is an extra fetch, and very long chains may be abandoned.
• Soft errors and empty pages. Pages that return a normal code but contain little or nothing.
• Broken pages. Large numbers of internal links pointing to missing pages.
• Endless spaces. Calendars, infinite scroll paths or search result pages that generate new URLs without end.
• Slow responses. If each fetch takes a long time, fewer fetches fit into the time a crawler spends.

How to look for waste

You do not need special software to get a rough picture:

1. Check your server logs, if you can access them, to see which URLs crawlers actually request. If most requests go to parameter URLs or old redirects instead of your key pages, that is a sign of waste.
2. Look at any crawl statistics your search tools provide. Look for patterns such as many requests returning errors or redirects.
3. Run a crawl of your own site with a desktop crawler and count how many distinct URLs it finds compared with how many real pages you intended to have. A large gap points to URL bloat.
4. Compare your sitemap with the pages that are actually indexed. Big differences deserve investigation.

What to do about it

Fixes are usually about tidying, not tricks:

• Keep the sitemap limited to pages you want indexed, in their preferred form.
• Use consistent internal links so each page has one main address.
• Use canonical tags where near-identical pages must exist, understanding they are a hint rather than a command.
• Use the robots file to keep crawlers out of clearly useless areas, such as internal search results or endless filter combinations. Be careful, because a mistake there can block valuable sections.
• Fix redirect chains and replace internal links that point at redirecting addresses.
• Remove or consolidate thin pages that serve no one.
• Improve server response time and reliability.

Notice that none of these require anything unusual. They are mostly the same housekeeping that helps visitors as well.

A reasonable mindset

Treat crawl budget as a diagnosis you reach after ruling out simpler causes, not a starting assumption. If your site is small and your pages are being indexed, move on. If your site is large or generates many URLs, a periodic check on where crawlers spend their time is a sensible habit, and it fits naturally into a monthly routine. No fix can promise that more pages will be indexed or that rankings will change, but removing waste at least gives your important pages a better chance of being seen promptly.