Research Papers Library

  2. Home
  3. Research Papers
  4. Meta Crawlers
  5. Crawler-Friendly Web Servers
Logan Hochstetler
Research Papers
Meta Crawlers

Crawler-Friendly Web Servers

In this paper we study how to make web servers (e.g., Apache) more craw ler friend ly. Current web servers o er the same interface to craw lers and regular web surfers, even though craw lers and surfers have very different performance requirements. We evaluate simple and easy-to-incorporate modi cations to web servers so that there are signi cant bandwidth savings. Speci cal ly, we propose that web servers export meta-data archives decribing their content.

Download PDF

Hits: 885

Research Paper Archives

Live Classes Schedule

AOFIRS

World's leading professional association of Internet Research Specialists - We deliver Knowledge, Education, Training, and Certification in the field of Professional Online Research. The AOFIRS is considered a major contributor in improving Web Search Skills and recognizes Online Research work as a full-time occupation for those that use the Internet as their primary source of information.

Get Exclusive Research Tips in Your Inbox

Receive Great tips via email, enter your email to Subscribe.

KNOWLEDGE NETWORK