Have you noticed that your Sitecore content tree may be indexed by search engines?
Try searching for “/sitecore/content/” in your favourite search engine. When I entered this into Bing the third link that I was shown was:
Clicking on the link takes you to a Page Error message on the RFU site as there’s no layout associated with the content item. The url shown in the browser address bar is:
Presumably having content indexed by search engines under two separate url’s on a site isn’t going to be too good for your SEO rankings unless appropriate action is taken.
One way that I found of successfully dealing with this problem was to add the following to the sites robots.txt file:
Can anyone offer a good explanation of what is allowing the structure of the Sitecore CMS’s content tree to be indexed by the search engines?