Smart crawler using deep-web interfaces

This is a survey of web crawling. While at first glance web crawling may appear to be merely an application of breadth-first-search, the truth is that there are many challenges ranging from systems concerns such as managing very large data structures to theoretical questions such ashow often to revisit evolving content sources. However, due to the large volume of web resources and the dynamic nature of deep web, achieving large coverage and high efficiency is a challenging issue. We propose a two stage framework, namely Smart-Crawler, for efficient harvesting wide web interfaces...

Download PDF