MySpiders: Evolve Your Own Intelligent Web Crawlers

The dynamic nature of the World Wide Web makes it a challenge to find information that is bothrelevant and recent. Intelligent agents can complement the power of searchengines to meet this challenge. We present a Web tool called MySpiders, which implements an evolutionary algorithm managing a population of adaptive crawlers who browse the Web autonomously. Each agent acts as an intelligent client on behalf of the user, driven by a user query and by textual and linkage clues in the crawled pages. Agents autonomously decide which links to follow, which clues to internalize, when to spawn offspring to focus the search near a relevant source, and when to starve. The tool is available to the public as a threaded Java applet. We discuss the development and deployment of such a system.

Download PDF