Controlling crawling and indexing now documented on code.google.com
Stay organized with collections
Save and categorize content based on your preferences.
Wednesday, November 24, 2010
Do you know how Google's crawler, Googlebot, handles conflicting rules in your robots.txt
file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot's favorite
song? The answers to these questions (except for the last one :)), along with lots of other
information about controlling the crawling and indexing of your site, are now available on
code.google.com:
Now site owners have a comprehensive resource where they can learn about robots.txt files,
robotsmeta tags, and X-Robots-Tag HTTP header rules. Please share your
comments, and if you have questions you can post them in our
Webmaster Help Forum.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["Google has launched a comprehensive resource on `code.google.com` for controlling how Google crawls and indexes websites."],["This resource provides information on robots.txt, robots meta tags, and X-Robots-Tag for managing website visibility in search results."],["Site owners can learn how Googlebot handles conflicting rules, prevent specific file types from being indexed, and more."],["For support, website owners can visit the Webmaster Help Forum to ask questions and share feedback."]]],[]]
