Skip to main content
Presentation
Detecting Web Robots Using Resource Request Patterns
Proceedings of the 11th International Conference on Machine Learning and Applications
  • Derek Doran, Wright State University - Main Campus
  • Swapna S. Gokhale
Document Type
Conference Proceeding
Publication Date
12-1-2012
Find this in a Library
Catalog Record
Disciplines
Abstract

A significant proportion of Web traffic is now attributed to Web robots, and this proportion is likely to grow over time. These robots may threaten the security, privacy, functionality, and performance of a Web server due to their unregulated crawling behavior. Therefore, to assess their impact, it must be possible to accurately detect Web robot requests. Contemporary detection approaches, however, may cease to be effective as the behavior of both robots and humans evolves. In this paper, we present a novel detection approach that is based on the contrasts in the resource request patterns of robots and humans. The proposed scheme, which relies on an invariant behavioral difference between humans and robots, builds on the lessons from contemporary approaches. We demonstrate that the proposed approach can accurately detect Web robots and argue that it is expected to remain effective even as they continue their rapid evolution.

Comments

Presented at the 11th International Conference on Machine Learning and Applications, Boca Raton, FL, December 12-15, 2012.

DOI
10.1109/ICMLA.2012.11
Citation Information
Derek Doran and Swapna S. Gokhale. "Detecting Web Robots Using Resource Request Patterns" Proceedings of the 11th International Conference on Machine Learning and Applications Vol. 1 (2012) p. 7 - 12 ISSN: 9781467346511
Available at: http://works.bepress.com/derek_doran/9/