Skip to main content
A Soft Computing Approach for Benign and Malicious Web Robot Detection
Expert Systems with Applications (2017)
  • Mahdieh Zabihimayvan, Wright State University
  • Reza Sadeghi, Wright State University
  • H. Nathan Rude, Wright State University
  • Derek Doran, Wright State University - Main Campus
The accurate detection of Web robot sessions from a web server log is essential to take accurate traffic-level measurements and to protect the performance and privacy of information on a Web server. Moreover, the irrecoverable risks of visits from malicious robots that intentionally try to evade web server intrusion detection systems, covering-up their visits with fabricated fields in their http request packets, cannot be ignored. To separate both types of robots from humans in practice, analysts turn to heuristic methods or state-of-the-art soft computing approaches that have only been tuned to the specification of a kind of web server. Noting that the landscape of web robot agents is ever changing, and that behavioral patterns and characteristics vary across different web servers, both options are lacking. To overcome this challenge, this paper presents SMART, a soft computing system that simultaneously detects benign and malicious types of robot agents from web server logs and can automatically adapt to the session characteristics of a web server. The results of experiments over some access log file servers, each servicing different domains of the web, demonstrate outperformance of the proposed method on state-of-the-art ones for benign and malicious robot detection.
  • Markov Clustering Algorithm; Web Robot Detection; Web Crawler; Malicious Web Agents; Fuzzy Rough Set Theory
Publication Date
Summer June 7, 2017
Citation Information
Mahdieh Zabihimayvan, Reza Sadeghi, H. Nathan Rude and Derek Doran. "A Soft Computing Approach for Benign and Malicious Web Robot Detection" Expert Systems with Applications (2017) ISSN: 0957-4174
Available at: