Web Search Tools

When searching for web pages on a topic, it’s important to pick the right search tool for the job. Sometimes a Search Engine such as Google will be the best choice, while for some topics a Subject Directory such as Librarian’s Internet Index or a Meta Search Engine such as Dogpile will be best. The NOVA Libraries Websites page provides a list of various search tools. In this next section, we’ll look at characteristics of these three main types of search tools, and also what each is best at.

Subject Directories, such as About.com, Librarian’s Internet Index, and Yahoo! Directory, tend to be smaller and more selective than larger search engines such as Google. With Subject Directories, actual human beings have searched the Internet looking for web pages on a variety of topics, and they decide whether or not to include the web page in their directory based on a set of criteria they have. The list of results provided by a Subject Directory will include a link to the website, a description of the website, and the subject heading they use to describe the content of the website. Subject directories do NOT search the complete text of websites for your search terms – they pretty much just look for your words in the descriptions and subject headings of each website they have in their directory. Take a look at one subject directory – Librarian’s Internet Index.

Subject directories tend to be good for broad topics and for academic research.

Search Engines, such as Google, tend to have a very large collection of websites that they search through. Search engines look for your terms in the complete text of their collection of websites, using “spiders” (basically, programmed computer robots). Search engines typically offer advanced search features that can help you focus in on the sorts of web pages you’re looking for. They let you do such things as:

- search for phrases, such as “global warming,”

- search for websites only from a particular domain, such as .gov or .edu,

- search for web pages from a specific website, and

- look for your search terms only in the web page title.

We’ll look at some of these advanced search features in depth later in the unit.

While a search in a search engine often results in thousands (if not millions) of results, some of the search engines have created programs that cause what they perceive to be the most relevant results to be listed first, so you certainly wouldn’t (and shouldn’t, unless you have lots of time to waste!) be looking at more than a few pages of search results.

Search engines are good if you need to be comprehensive, if you are searching for a really specific or uncommon topic, or if you just want to find the web page of a company or organization.

Meta-Search Engines, such a Dogpile and Mamma, search several search engines at once. This may sound like an extremely comprehensive search, but in reality the meta-search engines typically return just a limited number of websites from each separate search engine. There are also not many options for using advanced searching techniques.

Meta-search engines can be useful when you are searching for a unique term – something that wouldn’t require advanced searching techniques or that wouldn’t retrieve more than a few results per search engine.

Search engines and subject directories do NOT search the live web directly – they search for your terms in collections of websites that are saved on their own server(s). The search engines and directories do update their collection of websites periodically, but the web pages being searched by a search engine are typically a bit outdated. The resulting list of links to websites ARE to live sites, though.