Exercise 23: Search mechanisms

1. What is a spider? what does it do?


An Internet spider is a program used by searched engines which is designed to "crawl" over the World Wide Web, the portion of the Internet most familiar to general users, and retrieve locations of Web pages. The usual starting points are lists of heavily used servers and very popular pages. The spider will begin with a popular site, indexing the words on its pages and following every link found within the site. In this way, the spidering system quickly begins to travel, spreading out across the most widely used portions of the Web. It is sometimes referred to as a webcrawler. Many search engines use webcrawlers to obtain links, which are filed away in an index. When a user asks for information on a particular subject, the search engine pulls up pages retrieved by the Internet spider. Without spiders, the vast richness of the Web would be all but inaccessible to most users, rather as the Library of Congress would be if the books were not organized.

Some search engines are human-based, meaning that they rely on humans to submit links and other information, which the search engine categorizes, catalogues, and indexes. Most search engines today use a combination of human and crawler input. Crawler-based engines send out spiders, which are actually computer programs that have sometimes been likened to viruses because of their ability to move between, and insert themselves into, other areas in cyberspace.

Spiders visit Web sites, record the information there, read the meta tags that identify a site according to subjects, and follow the site's links to other pages. Because of the many links between pages, a spider can start at almost any point on the Web and keep moving. Eventually it returns the data gathered on its journey to the search engine's central depository of information, where it is organized and stored. Periodically the crawler will revisit the sites to check for changed information, but until it does so, the material in the search engine's index remains the same. It is for this reason that a search at any time may yield "dead" Web pages, or ones that can no longer be found.

No two search engines are exactly the same, the reason being (among other things) a difference in the choice of algorithm by which the indices are searched. Algorithms can be adjusted to scan for the frequency of certain keywords, and even to circumvent attempts at keyword stuffing or "spamdexing," the insertion of irrelevant search terms intended simply to draw traffic to a site.


http://www.answers.com/topic/internet-spider

http://computer.howstuffworks.com/search-engine1.htm


2. What is a meta-search engine? Provide some examples.


Meta search engines are search engine tools that pass queries on to many other search engines and/or directories and then summarize all the results in one handy interface. They appear like an aggregator for receiving your input (search requests) and then forward the request(s) to several other search engines and/or databases and aggregates the results into a single list or displays them according to their source. Metasearch engines enable users to enter search criteria once and access several search engines simultaneously. Metasearch engines operate on the premise that the Web is too large for any one search engine to index it all and that more comprehensive search results can be obtained by combining the results from several search engines. This also may save the user from having to use multiple search engines separately.

Examples of Meta-search engines:
Clusty: www.clusty.com
Dogpille: www.dogpile.com
SurWax: www.surfwax.com
Copernic Agent: www.copernic.com


3. How can you get your site listed at major search sites; and how could you improve your site ranking?

Here are some of the ways you could improve your website ranking.

- Subscribe your Web site to major search sites for them easily to reach your Web site.
- Declare the key terms that your users will search on to find content like yours carefully. Use those key terms to guide the text and construct your web pages.
- As users are likely to click the link if the title matches their search, choose terms for the title that match the concept of your document.
- Use meta-tags as a description and write the description accurately and carefully. Make sure the document title and description attract the interest of the user.
- Use 'keyword' in meta-tag to list key words for the document.
- Keep relevant text and links in HTML. An HTML site map, with a link from the welcome page, can help to make sure all your pages are crawled by the search engines.
- Use ALT text for graphics for good page design to accommodate text browsers or visually impaired visitors.