Soft Errors and Thin Pages That Waste Crawling

The mismatch behind a soft error

When a crawler requests a page, your server replies with a status code before sending any content. A 200 code means “this page exists and is fine”. A 404 means “not found”.

A soft error happens when the page content says one thing and the code says another. The classic case: a visitor requests an address that does not exist, the site shows a friendly “Sorry, we could not find that” page, but the server still answers 200. To a person it reads as an error. To a crawler, the code says the page is fine, so it may fetch it, process it and then decide that it looks like an error anyway.

Search engines have learned to spot these on their own. Search tools often report them as soft 404 errors. The result is wasted crawling on pages that will never be useful and, often, reports full of noise.

What counts as a soft error

Common examples:

• A custom “page not found” screen that returns a success code.
• Redirecting every missing address to the home page, which can look to a search engine like a missing page that has been disguised.
• A category or search page that displays “No results found” with a success code.
• A product page for an item that no longer exists, showing only a short message.
• A page that loads a blank shell because a script failed or data did not arrive.
• A page whose main content is hidden behind a login but returns a success code.

Thin pages are a related problem

Thin pages return the correct code, but offer so little that they have nothing to index. They are not errors in the strict sense, though search engines may treat some of them similarly. Typical examples:

• Tag and archive pages listing one or two items.
• Placeholder pages created from a template but never filled in.
• Doorway-style pages that repeat the same short text with only a place name or product name changed.
• Auto-generated pages with only a heading and a few links.
• Pages whose entire content is a sentence or two.

Thin does not just mean short. A short page that fully answers a narrow question, such as a contact page or a simple definition, can be perfectly valid. The question is whether the page gives a visitor something useful that they could not get more easily elsewhere on your own site. The broader topic, including keyword stuffing and other page problems, has its own treatment, and merging or removing weak pages is discussed under content pruning.

Why it matters

• Wasted crawling. Crawlers spend time on pages that give nothing back. On a small site this barely matters. On a site with many generated pages it can be a drag, as discussed under crawl budget.
• Weaker overall impression. A site with a large share of near-empty pages is harder for search engines to judge favorably.
• Poor visitor experience. Landing on an empty listing is frustrating.
• Misleading reports. Genuine problems get buried among the noise.

How to find them

1. Check search tools reports for soft 404 or similar messages. They list the exact addresses affected.
2. Test missing addresses. Type a made-up address on your site and look at the status code that comes back. It should be 404. Many platforms get this wrong for certain sections.
3. Run a crawl and sort pages by word count or content size. Review the smallest ones. Also look for pages sharing the same title or very similar text.
4. Look at generated sections. Open a few filter results, tag pages, empty categories and searches with no matches.
5. Check how the page behaves with scripts disabled or failing, to see if some pages are blank when data is missing.
6. Review analytics for pages with a lot of exits and almost no engagement. Interpret with care, since a quick answer is not always a bad page.

What to do with each page

• A missing page showing a friendly message. Make the server return a real 404 code (or 410 if removed on purpose) while keeping the helpful design.
• A removed product or page with a close replacement. Redirect permanently to the relevant replacement. If no good replacement exists, return 404 or 410 rather than redirecting to the home page. The out-of-stock pages article covers the store version of this decision.
• An empty search or category page. Return a suitable code or add noindex, and consider not generating the page until it has results.
• Thin pages that could be useful. Improve them with original, helpful content.
• Thin pages on closely related topics. Merge them into one stronger page and redirect the old addresses.
• Thin pages with a purpose for visitors but not for search. Keep them and apply noindex.
• Pages with no purpose. Delete them and let them return a 404 or 410.

Prevention

Set up the platform so that missing content returns the correct code from the start. Avoid creating pages automatically for every possible combination of tag, filter and place name. Add a quick check to your regular review: after each major content or template change, test a missing address, an empty search and an empty category.

As always, search engines may take time to reprocess the pages you fix, and no change can guarantee a result. A cleaner set of pages simply gives your better ones a fairer chance.