Wednesday, November 24, 2010
Do you know how Google's crawler, Googlebot, handles conflicting rules in your robots.txt
file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot's favorite
song? The answers to these questions (except for the last one :)), along with lots of other
information about controlling the crawling and indexing of your site, are now available on
code.google.com:
Controlling crawling and indexing
Now site owners have a comprehensive resource where they can learn about robots.txt files,
robots meta tags, and X-Robots-Tag HTTP header rules. Please share your
comments, and if you have questions you can post them in our
Webmaster Help Forum.