This is the most common form of text search on the Web. Most search engines do their text query and retrieval using keywords.
What is a keyword, exactly? It can simply be any word on a webpage. For example, I used the word “simply” in the previous sentence, making it one of the keywords for this particular webpage in some search engine’s index. However, since the word “simply” has nothing to do with the subject of this webpage (i.e., how search engines work), it is not a very useful keyword. Useful keywords and key phrases for this page would be “search,” “search engines,” “search engine methods,” “how search engines work,” “ranking” “relevancy,” “search engine tutorials,” etc. Those keywords would actually tell a user something about the subject and content of this page.
Unless the author of the Web document specifies the keywords for her document (this is possible by using meta tags), it’s up to the search engine to determine them. Essentially, this means that search engines pull out and index words that appear to be significant. Since since engines are software programs, not rational human beings, they work according to rules established by their creators for what words are usually important in a broad range of documents. The title of a page, for example, usually gives useful information about the subject of the page (if it doesn’t, it should!). Words that are mentioned towards the beginning of a document (think of the “topic sentence” in a high school essay, where you lay out the subject you intend to discuss) are given more weight by most search engines. The same goes for words that are repeated several times throughout the document.
Some search engines index every word on every page. Others index only part of the document.
Full-text indexing systems generally pick up every word in the text except commonly occurring stop words such as “a,” “an,” “the,” “is,” “and,” “or,” and “www.” Some of the search engines discriminate upper case from lower case; others store all words without reference to capitalization.