Search Engine Optimization-How sites get into search engines
Tuesday, July 15th, 2008The base case is that spiders crawl the entire Web, starting from known pages and following all links, and also crawling pages that are hand-submitted. Google is pretty much like that still. If a site has high PageRank, it is spidered more often and more deeply.
However, search engines are trying to encourage site owners to pay for the privelege of having their pages spidered. Teoma’s index is very hard to get into without paying money, and Inktomi’s isn’t that easy either. And even if you do get into Inktomi for free, they’ll take a long time to respider, while if you pay they respider constantly. One advantage of being respidered often is that you can tweak your page to come up higher in their relevancy rankings, then see if your changes worked.
Finally, you can also pay to appear on a search page. That is, your link will appear when someone searches on a specific keyword or keyphrase. Google does a good job of making it pretty clear which results (at the top or on the right of the page) are paid; others maybe do a not-so-good job.
Paid search results are typically all pay-per-click, based on keyword. The advertiser pays the search engine vendor a specific amount of money each time an ad is clicked on, this fee having been determined by an auction of each keyword or keyphrase.