Do different URLs repeat the same content?
It groups identical page bodies, titles and descriptions after checking canonical agreement.
What Check Engine measures
Body comparison ignores pages with fewer than 50 words and hashes lowercased text with normalised whitespace. Groups sharing a single nonempty canonical are excluded. A separate substitution case needs at least 5 pages across at least 4 sections with no page above 200 words. Duplicate titles and descriptions are grouped independently. Redirect scheme splits and regional alternatives change the explanation rather than turning all identical text into the same diagnosis. For example, a page with exactly 50 words is long enough for the body-text comparison.
The check reads body text and metadata from pages with usable content. Body matching ignores capitalisation and extra spaces, then compares the resulting text exactly. A separate test compares the subjects suggested by titles.
How to read the result
Duplicate bodies are high severity, titles medium and descriptions low. Repeated short responses across unrelated sections are medium. The report distinguishes a shared error or substitute response from normal pages that genuinely contain the same text.
Why it matters
Identical bodies can appear because of duplicate publishing, multiple address forms or a server serving a substitute response. Understanding which case applies is necessary before deciding whether to merge content or repair delivery.
How to fix it
Compare the saved bodies with the expected pages. Fix substitute responses at the server or template level. For real duplicate addresses, select the intended canonical or redirect where appropriate. Write distinct titles for distinct pages. Preserve valid regional alternatives and review their language annotations before treating them as accidental copies.
Identical content is not always duplicate content
What we found in field audits
Problems in Crawlability & indexing were found on 12 of the 14 sites we audited. These are area totals. Read the field-audit method.
Method note
backend/seo_engine/checks/indexability.py, duplicates, revision 0bb8e8d3.
Common questions
How many words are needed for the duplicate-body comparison?
A page needs at least 50 words. Shorter responses are left out of the ordinary body-hash comparison.
Can identical pages with the same canonical be excluded from body duplicates?
Yes. A group can be excluded when its members share a single nonempty canonical. That declaration should still describe the right relationship.
Are duplicate descriptions as serious as duplicate page bodies?
No. Body duplication gets high severity. Titles have medium severity and descriptions low.