Loading...

Freshness of web search engines: Improving performance of web search engines using data mining techniques

Kharazmi, S ; Sharif University of Technology

733 Viewed
  1. Type of Document: Article
  2. Abstract:
  3. Progressive use of Web based information retrieval systems such as general purpose search engines and dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. Freshness (recency) is one of the important maintaining factors of Web search engine crawlers that takes weeks to months. Many large Web crawlers start from seed pages, fetch every links from them, and continually repeat this process without any policies that help them to better crawling and improving performance of those. We believe that data mining techniques can help us to improve the freshness parameter by extracting knowledge from crawling data. In this paper we propose a Web crawler that uses extracted knowledge by data mining techniques as policies for crawling. For this purpose we include a component to collect additional crawling information. This crawler starts by non-preferential crawling. After a few crawling, it trained by using mining techniques on crawling data and then uses policies for preferential crawling to improve freshness time. Our research represented that crawling with determined polices has better freshness than generic general purpose Web crawlers. Copyright © 2009 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved
  4. Keywords:
  5. Data mining techniques ; Dynamic nature ; General purpose ; General-purpose search engines ; Hyperlinks ; Improving performance ; Mining techniques ; Retrieval systems ; Web crawlers ; Web page ; Web search engines ; Web-based information ; Data mining ; Hypertext systems ; Information retrieval ; Information retrieval systems ; Information services ; Internet ; Search engines ; World Wide Web
  6. Source: International Conference for Internet Technology and Secured Transactions, ICITST 2009, 9 November 2009 through 12 November 2009, London ; 2009 ; 9781424456482 (ISBN)
  7. URL: http://ieeexplore.ieee.org/document/5402607