Skip to main content
Presentation
A Comparison of Web Robot and Human Requests
Proceedings of the IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
  • Derek Doran, Wright State University - Main Campus
  • Kevin Morillo
  • Swapna S. Gokhale
Document Type
Conference Proceeding
Publication Date
8-1-2013
Find this in a Library
Catalog Record
Disciplines
Abstract

Sophisticated Web robots sport a wide variety of functionality and visiting characteristics, constituting a significant percentage of the requests serviced by a Web server. Unlike human clients that retrieve information off a site by navigating links and ignoring irrelevant information, Web robots may collect many different types of resources, and employ varying navigation strategies to find the knowledge on the site they desire. Thus, the resource request patterns of their visits are unpredictable and cannot be inferred based on our knowledge of human request patterns. In this paper, we perform an analysis on the types of resources requested by Web robots using recent Web logs from an academic Web server. We study the distribution of response sizes and response codes, the types of resources requested, and popularity of resources for requests from Web robots. Throughout, we contrast our findings against human resource request patterns. We find reasons to suggest that robots severely handicaps the ability of Web server caches to operate with high performance.

Comments

Presented at the IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, Niagara Falls, ON, August 25-28, 2013.

Citation Information
Derek Doran, Kevin Morillo and Swapna S. Gokhale. "A Comparison of Web Robot and Human Requests" Proceedings of the IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (2013) p. 1374 - 1380
Available at: http://works.bepress.com/derek_doran/29/