IntelligentWeb Agent for Search Engines

Abstract: In this paper we review studies of the growth of the Internet and technologies that are useful for information search and retrieval on the Web. Search engines are retrieve the efficient information. We collected data on the Internet from several different sources, e.g., current as well as projected number of users, hosts, and Web sites. The trends cited by the sources are consistent and point to exponential growth in the past and in the coming decade. Hence it is not surprising that about 85% of Internet users surveyed claim using search engines and search services to find specific information and users are not satisfied with the performance of the current generation of search engines; the slow retrieval speed, communication delays, and poor quality of retrieved results. Web agents, programs acting autonomously on some task, are already present in the form of spiders, crawler, and robots. Agents offer substantial benefits and hazards, and because of this, their development must involve attention to technical details. This paper illustrates the different types of agents,crawlers, robots,etc for mining the contents of web in a methodical, automated manner, also discusses the use of crawler to gather specific types of information from Web pages, such as harvesting e-mail addresses