Web Crawlers and Web Crawling Algorithms - A Review

Authors(2) :-Vishakha Shukla, Dharmendra Roy

As there is profound web development, there has been expanded enthusiasm for methods that help productively find profound web interfaces. Because of accessibility of inexhaustible information on web, seeking has a noteworthy effect. On-going examines place accentuation on the pertinence and strength of the information found, as the found examples closeness is a long way from the investigated. Notwithstanding their importance pages for any inquiry subject, the outcomes are colossal to be investigated. One issue of pursuit on the Web is that internet searchers return huge hit records with low accuracy. Clients need to filter applicable reports from insignificant ones by physically bringing and skimming pages. Another debilitating viewpoint is that URLs or entire pages are returned as list items. It is likely that the response to a client question is just part of the page. Recovering the entire page really leaves the errand of inquiry inside a page to Web clients. With these two viewpoints staying unaltered, Web clients won't be liberated from the substantial weight of perusing pages and finding required data, and data got from one pursuit will be characteristically constrained.

Authors and Affiliations

Vishakha Shukla
Department of Computer Science and Engineering, Rungta College of Engineering and Technology, Bhilai, Chhattisgarh, India
Dharmendra Roy
Computer Science and Engineering Department, Rungta College of Engineering and Technology, Bhilai, Chhattisgarh, India

Web Crawlers, Web Crawling, Depth First Search (DFS), Breadth First Search (BFS), Query Processing

  1. SmartCrawler: A Two-stage Crawler for Efficiently Harvesting Deep-Web, Feng Zhao, Jingyu Zhou, Chang Nie, Heqing Huang & Hai Jin, Interfaces .IEEE Transactions on Services Computing Volume: PP Year: 2015.
  2. Survey of Web Crawling Algorithms - Rahul kumar, Anurag Jain and Chetan Agrawal. Advances in Vision Computing: An International Journal (AVC) Vol.1, No.2/3, September 2014.
  3. Algorithms and Programming Problems and Solutions, Shen and Alexander, Springer Undergraduate Texts in Mathematics and Technology 2010
  4. Ben Coppin “Artificial Intelligence illuminated” Jones and Barlett Publishers, 2004, Pg 77.
  5. Dempster-Shafer theory for a query-biased combination of evidence on the Web- Vassilis Plachouras, Iadh Ounis. Springer-Verlag Berlin Heidelberg 2014.
  6. Effective Pre-retrieval Query Performance Prediction Using Similarity and Variability Evidence, Ying Zhao, Falk Scholer, and Yohannes Tsegay, C.Macdonald et al. (Eds.): ECIR 2008, LNCS 4956, pp. 52–64, 2008._c Springer-Verlag Berlin Heidelberg 2008.
  7. Inferring Query Performance Using Pre-retrieval Predictors, Ben He and Iadh Ounis, Department of Computing Science University of Glasgow fben,ounisg@dcs.gla.ac.uk.
  8. A Unified Framework for Post-Retrieval Query-Performance Prediction, Oren Kurland, Anna Shtok, David Carmel, and Shay Hummel, ICTIR 2011, LNCS 6931, pp.  15–26, 2011. c_Springer-Verlag Berlin Heidelberg 2011.
  9. Varying Approaches to Topical Web Query Classification , Steven M. Beitzel, Eric C. Jensen, Abdur Chowdhury,& Ophir Frieder, SIGIR’07, July 23–27, 2007, Amsterdam, The Netherlands, ACM ..
  10. Survey on – Self Adaptive Focused Crawler,  Ms. Pallavi Wadibhasme, & Prof. Nitin Shivale, Pallavi Wadibhasme et al, / (IJCSIT) International Journal of Computer Science and Information Technologies, Vol. 6 (1) , 2015, 218-220 .
  11. Efficient Query Evaluation using a Two-Level Retrieval Process- Andrei Z. Broder, David Carmel, Michael Herscovici, Aya Soffer & Jason Zien.
  12. Evaluating Topic DrivenWeb Crawlers, Filippo Menczer, Gautam Pant,& Padmini Srinivasan

Publication Details

Published in : Volume 2 | Issue 2 | March-April 2016
Date of Publication : 2017-12-31
License:  This work is licensed under a Creative Commons Attribution 4.0 International License.
Page(s) : 258-260
Manuscript Number : IJSRSET1621137
Publisher : Technoscience Academy

Print ISSN : 2395-1990, Online ISSN : 2394-4099

Cite This Article :

Vishakha Shukla, Dharmendra Roy, " Web Crawlers and Web Crawling Algorithms - A Review, International Journal of Scientific Research in Science, Engineering and Technology(IJSRSET), Print ISSN : 2395-1990, Online ISSN : 2394-4099, Volume 2, Issue 2, pp.258-260, March-April-2016.
Journal URL : http://ijsrset.com/IJSRSET1621137

Article Preview

Follow Us

Contact Us