Have you noticed that your Sitecore content tree may be indexed by search engines?
Try searching for “/sitecore/content/” in your favourite search engine. When I entered this into Bing the third link that I was shown was:
www.rfu.com
http://www.rfu.com/sitecore/content.aspx
Clicking on the link takes you to a Page Error message on the RFU site as there’s no layout associated with the content item. The url shown in the browser address bar is:
http://www.rfu.com/error?item=%2fsitecore%2fcontent&layout=%7b00000000-0000-0000-0000-000000000000%7d&device=Default
Presumably having content indexed by search engines under two separate url’s on a site isn’t going to be too good for your SEO rankings unless appropriate action is taken.
One way that I found of successfully dealing with this problem was to add the following to the sites robots.txt file:
User-agent: *
Disallow: /Sitecore/
Can anyone offer a good explanation of what is allowing the structure of the Sitecore CMS’s content tree to be indexed by the search engines?
Stone Soup Developer 