Skip to main content
Article
Automatically Discovering the Number of Clusters in Web Page Datasets
Proceedings of the 2005 International Conference on Data Mining, DMIN 2005
  • Zhongmei Yao, University of Dayton
Document Type
Article
Publication Date
6-1-2005
Abstract

Clustering is well-suited for Web mining by automatically organizing Web pages into categories, each of which contains Web pages having similar contents. However, one problem in clustering is the lack of general methods to automatically determine the number of categories or clusters. For the Web domain in particular, currently there is no such method suitable for Web page clustering. In an attempt to address this problem, we discover a constant factor that characterizes the Web domain, based on which we propose a new method for automatically determining the number of clusters in Web page data sets. We discover that the measure of average inter-cluster similarity reaches a constant of 1.7 when all our experiments produced the best results for clustering Web pages. We determine the number of clusters by using the constant as the stopping factor in our clustering process by arranging individual Web pages into clusters and then arranging the clusters into larger clusters and so on until the average inter-cluster similarity approaches the constant. Having the new method described in this paper together with our new Bidirectional Hierarchical Clustering algorithm reported elsewhere, we have developed a clustering system suitable for mining the Web.

Document Version
Published Version
Peer Reviewed
Yes
Keywords
  • Web Mining,
  • Clustering,
  • Classification,
  • Information Retrieval,
  • Knowledge Discovery
Citation Information
Zhongmei Yao. "Automatically Discovering the Number of Clusters in Web Page Datasets" Proceedings of the 2005 International Conference on Data Mining, DMIN 2005 (2005)
Available at: http://works.bepress.com/zhongmei_yao/3/