How to Read a Robots File Without Breaking Your Site

What the file is

A robots file is a plain text file placed at the root of your site, at the address /robots.txt. It gives instructions to automated crawlers about which parts of the site they are asked not to fetch. Well-behaved crawlers read it before visiting other pages.

Two points are worth stating early:

• It is a request, not a lock. Reputable search engine crawlers follow it, but it does not enforce anything and it does not protect private content. Never rely on it to hide sensitive material.
• It controls crawling, not indexing. A page blocked from crawling can sometimes still appear in results as a bare address, without a description, if other pages link to it.

The basic structure

The file is a list of groups. Each group begins with the crawler it applies to, followed by rules.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

Sitemap: https://www.example.com/sitemap.xml

Reading it line by line:

• User-agent names the crawler the group applies to. An asterisk means all crawlers that do not have a more specific group.
• Disallow lists a path prefix the crawler is asked not to fetch.
• Allow makes an exception inside a disallowed area.
• Sitemap points to your XML sitemap. It can appear anywhere in the file.

Lines beginning with # are comments and are ignored.

How paths are matched

Rules match from the beginning of the path. Disallow: /search blocks /search, /search/, /search?q=shoes and also /search-tips, because all of them start with those characters. That last example surprises people. If you want to block only a folder, include the trailing slash: /search/.

Many major crawlers also understand two pattern characters:

• * matches any run of characters.
• $ marks the end of the address.

So Disallow: /*.pdf$ asks crawlers to skip addresses ending in .pdf. Not every crawler supports these patterns, so check before relying on them for something important.

When an Allow and a Disallow both match, the more specific (longer) rule generally wins, though the details can vary between crawlers.

The dangerous empty-looking line

The most damaging mistake is a single slash:

User-agent: *
Disallow: /

This asks every crawler to stay away from the entire site. It is often left over from a staging or development version and then copied to the live site during a launch or a redesign. Compare it with:

User-agent: *
Disallow:

An empty Disallow blocks nothing. The two look similar at a glance and mean opposite things, so read them slowly.

Other mistakes that block more than intended

• Blocking script and style folders. Crawlers may need these files to understand how a page looks and works. Blocking them can lead to pages being understood poorly.
• Prefix surprises. A short rule like /p blocks everything starting with those characters, including /products/ and /privacy/.
• Blocking a page you want removed. If you want a page kept out of the index, blocking it can prevent the crawler from seeing a noindex tag. In that case allow crawling and use noindex instead. The difference is explained in the article on canonical tags.
• Blocking the sitemap. A sitemap file or the pages in it should not be disallowed.
• Case and spelling. Paths are case-sensitive in many setups, and the file name must be exactly lowercase robots.txt.
• Wrong location. The file works only at the root of the host. A copy in a subfolder is ignored, and each subdomain needs its own.
• Server errors. If the file returns a server error, crawlers may treat the whole site as temporarily off limits. A missing file (404) is generally treated as “no restrictions”.

A sensible approach to editing

1. Save a copy of the current file before any change.
2. Write down the intent of each rule. If you cannot say what a rule protects against, question whether it is needed.
3. Keep it simple. Many small sites need only a sitemap line and perhaps a rule or two for internal search results or cart pages.
4. Test the change. Use the robots testing feature in your search tools, if available, or another checker, to confirm that your important pages are allowed and the intended ones are blocked. Test specific real addresses, including your home page, a key category and a key article.
5. Publish and recheck by opening /robots.txt in a browser to confirm what is live.
6. Monitor afterwards. Watch for a sudden drop in crawled or indexed pages in the weeks after any change. Search engines typically cache the file and may take a day or so to notice an update.

What belongs in the file

Reasonable candidates for blocking include internal search result pages, cart and checkout steps, and endless parameter combinations that produce no useful new content, which relates to url parameters. Things that generally do not belong are your content, your images and anything you want to appear in results.

It also helps to review the file whenever you move a site, since an accidental carry-over of old rules is one of the more common causes of sudden visibility problems during a site migration.