Tuesday, July 31, 2007
When Google originally introduced Supplemental Results in 2003, our main web index had billions of web pages. The supplemental index made it possible to index even more web pages and, just like our main web index, make this content available when generating relevant search results for user queries. This was especially useful for queries that did not return many results from the main web index, and for these the supplemental index allowed us to query even more web pages. The fewer constraints we're able to place on sites we crawl for the supplemental index means that web pages that are not in the main web index could be included in the supplemental. These are often pages with lower PageRank or those with more complex URLs. Thus the supplemental index (Matt blogged about it and here's his talk about it on video) serves a very important purpose: to index as much of the relevant content that we crawl as possible.
The changes we make must focus on improving the search experience for our users. Since 2006, we've completely overhauled the system that crawls and indexes supplemental results. The current system provides deeper and more continuous indexing. Additionally, we are indexing URLs with more parameters and are continuing to place fewer restrictions on the sites we crawl. As a result, Supplemental Results are fresher and more comprehensive than ever. We're also working towards showing more Supplemental Results by ensuring that every query is able to search the supplemental index, and expect to roll this out over the course of the summer.
The distinction between the main and the supplemental index is therefore continuing to narrow. Given all the progress that we've been able to make so far, and thinking ahead to future improvements, we've decided to stop labeling these URLs as "Supplemental Results." Of course, you will continue to benefit from Google's supplemental index being deeper and fresher.