Skip to main content
Article
Detecting Anomaly Collections using Extreme Feature Ranks
Data Mining and Knowledge Discovery
  • Hanbo DAI, Hubei University
  • Feida ZHU, Singapore Management University
  • Ee Peng LIM, Singapore Management University
  • Hwee Hwa PANG, Singapore Management University
Publication Type
Journal Article
Version
acceptedVersion
Publication Date
5-2014
Abstract

Detecting anomaly collections is an important task with many applications, including spam and fraud detection. In an anomaly collection, entities often operate in collusion and hold different agendas to normal entities. As a result, they usually manifest collective extreme traits, i.e., members of an anomaly collection are consistently clustered toward the top or bottom ranks on certain features. We therefore propose to detect these anomaly collections by extreme feature ranks. We introduce a novel anomaly definition called Extreme Rank Anomalous Collection or ERAC. We propose a new measure of anomalousness capturing collective extreme traits based on a statistical model. As there can be a large number of ERACs of various sizes, for simplicity, we first investigate the ERAC detection problem of finding top-KERACs of a predefined size limit. We then tackle the follow-up ERAC expansion problem of uncovering the supersets of the detected ERACs that are more anomalous without any size constraint. Algorithms are proposed for both ERAC detection and expansion problems, followed by studies of their performance in four datasets. Specifically, in synthetic datasets, both ERAC detection and expansion algorithms demonstrate high precisions and recalls. In a web spam dataset, both ERAC detection and expansion algorithms discover web spammers with higher precisions than existing approaches. In an IMDB dataset, both ERAC detection and expansion algorithms identify unusual actor collections that are not easily identified by clustering-based methods. In a Chinese online forum dataset, our ERAC detection algorithm identifies suspicious “water army” spammer collections agreed by human evaluators. ERAC expansion algorithm successfully reveals two larger spammer collections with different spamming behaviors.

Keywords
  • Anomaly collection,
  • Extreme feature rank,
  • Anomaly cluster,
  • Outlier group,
  • Spam detection,
  • Spam cluster
Identifier
10.1007/s10618-014-0360-3
Publisher
Springer Verlag
Creative Commons License
Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International
Additional URL
https://doi.org/10.1007/s10618-014-0360-3
Citation Information
Hanbo DAI, Feida ZHU, Ee Peng LIM and Hwee Hwa PANG. "Detecting Anomaly Collections using Extreme Feature Ranks" Data Mining and Knowledge Discovery Vol. 29 Iss. 3 (2014) p. 689 - 731 ISSN: 1384-5810
Available at: http://works.bepress.com/hweehwa-pang/34/