How to Spot Duplicate Content on Your Own Site

What duplicate content really is

Duplicate content means the same or very similar text available at more than one address. It can exist inside one site or across different sites. This article is about your own site.

It is worth correcting a common fear. In most cases there is no special penalty for duplication. The practical issue is that search engines tend to choose one version to keep and may fold the others into it. If they choose a version you did not prefer, or if signals like links are split across several addresses, your preferred page may do worse than it could. Duplicates also waste crawling and make reports harder to read.

Deliberate attempts to mass-copy content to manipulate results are a separate matter and are viewed unfavorably. Most site owners are dealing with accidental duplication.

Where duplication usually comes from

Address variations

The same page can be reachable at several addresses without anyone intending it:

With and without www.
On both http and https.
With and without a trailing slash.
With uppercase and lowercase letters in the path.
With index.html or index.php added.

Parameters

Tracking tags, sort orders, filters, session identifiers and print views can all create new addresses that show the same content. Online stores are particularly prone to this, and it ties into url parameters and faceted navigation.

Templates and boilerplate

Pages built from a template can share so much text that the unique part is tiny. Typical examples are location pages with only the town name swapped, product pages that differ by one attribute, and tag or category archives that repeat the same excerpts. Product variants and multi-location sites are two situations where this regularly appears.

Archives and listing pages

Blogs often show the same post on its own page, in a category, under a tag, in a date archive and on an author page. If each archive shows the full text, the duplication multiplies.

Copied or syndicated text

You may have reused the same description across pages, copied manufacturer text for many products, or published an article on another site that also appears on yours. Printed or mobile versions at separate addresses are another source.

Staging and test copies

A development copy of the site left open to crawlers can duplicate everything.

How to find it

Crawl your site. A desktop crawler can report pages with identical or near-identical titles, headings, meta descriptions and body content. Identical titles are a quick, rough signal, though not proof of duplication.
Look at the URL list. Sort addresses and scan for repeating patterns, such as the same path with different parameters, or both slash versions.
Search for a sentence. Take a distinctive sentence from a page, put it in quotation marks and search. Check whether several of your own pages come up. Add the name of your site to narrow results.
Check your search tools. Index reports often include messages about pages treated as duplicates, and which address was chosen instead. This is valuable, because it tells you which version the search engine prefers.
Compare your sitemap with your crawl. Addresses in the crawl that you never listed are suspects.
Test the variations by hand. Type the same page with and without www, with http, and with a trailing slash, and see whether each redirects to one version or loads separately.

Deciding what to do

The right fix depends on why the duplicates exist.

Address variations that should never exist separately. Choose one preferred version and permanently redirect the others to it. Keep internal links, the sitemap and canonical tags all pointing at the preferred form.
Parameter versions that must keep working for visitors. Use a canonical tag pointing to the main version, and consider keeping crawlers out of pointless combinations. Remember that canonical tags are hints, not commands.
Pages that are genuinely similar but serve different needs. Rewrite each so it has its own useful content, or merge them into one stronger page.
Archives. Show short excerpts instead of full text, or exclude low-value archives from indexing with a noindex tag.
Copied text. Write original descriptions, at least for your most important pages.
Test copies. Protect them with a login, which is more reliable than relying on a request to crawlers.

Do not overreact

Some repetition is normal and harmless. Navigation, footers, legal notes and short standard blocks appear on many pages, and search engines are used to that. Similarly, quoting a short passage with attribution is ordinary. Focus on cases where whole pages or the main content repeat.

A good rule: if a visitor would be confused or bored finding two of your pages next to each other in results, they are probably too similar. Consolidate or differentiate them, then recheck after search engines have had time to revisit.