Article
A Classification Framework for Web Robots
Journal of the American Society for Information Science and Technology
Document Type
Article
Publication Date
12-1-2012
Disciplines
Abstract
The behavior of modern web robots varies widely when they crawl for different purposes. In this article, we present a framework to classify these web robots from two orthogonal perspectives, namely, their functionality and the types of resources they consume. Applying the classification framework to a year-long access log from the UConn SoE web server, we present trends that point to significant differences in their crawling behavior.
DOI
10.1002/asi.22741
Citation Information
Derek Doran and Swapna S. Gokhale. "A Classification Framework for Web Robots" Journal of the American Society for Information Science and Technology Vol. 63 Iss. 12 (2012) p. 2549 - 2554 ISSN: 15322882 Available at: http://works.bepress.com/derek_doran/30/