Skip to main content
Article
Observed Web Robot Behavior on Decaying Web Subsites
D-Lib Magazine
  • Joan A. Smith, Old Dominion University
  • Frank McCown, Ph.D., Harding University
  • Michael L. Nelson, Old Dominion University
Document Type
Article
Publication Date
2-1-2006
Abstract

We describe the observed crawling patterns of various search engines (including Google, Yahoo and MSN) as they traverse a series of web subsites whose contents decay at predetermined rates. We plot the progress of the crawlers through the subsites, and their behaviors regarding the various file types included in the web subsites. We chose decaying subsites because we were originally interested in tracking the implication of using search engine caches for digital preservation. However, some of the crawling behaviors themselves proved to be interesting and have implications on using a search engine as an interface to a digital library.

Creative Commons Usage License
Creative Commons Attribution-Noncommercial-No Derivative Works 4.0
Copyright held by
Author
Disciplines
Citation Information
Joan A. Smith, Frank McCown and Michael L. Nelson. "Observed Web Robot Behavior on Decaying Web Subsites" D-Lib Magazine Vol. 12 Iss. 2 (2006)
Available at: http://works.bepress.com/fmccown/7/