Skip to main content
Article
LogUAD: Log unsupervised anomaly detection based on word2Vec
Computer Systems Science and Engineering
  • Jin Wang, Changsha University of Science and Technology
  • Changqing Zhao, Changsha University of Science and Technology
  • Shiming He, Changsha University of Science and Technology
  • Yu Gu, Goethe-Universität Frankfurt am Main
  • Osama Alfarraj, College of Sciences
  • Ahed Abugabah, Zayed University
Document Type
Article
Publication Date
1-1-2022
Abstract

System logs record detailed information about system operation and are important for analyzing the system's operational status and performance. Rapid and accurate detection of system anomalies is of great significance to ensure system stability. However, large-scale distributed systems are becoming more and more complex, and the number of system logs gradually increases, which brings challenges to analyze system logs. Some recent studies show that logs can be unstable due to the evolution of log statements and noise introduced by log collection and parsing. Moreover, deep learning-based detection methods take a long time to train models. Therefore, to reduce the computational cost and avoid log instability we propose a new Word2Vec-based log unsupervised anomaly detection method (LogUAD). LogUAD does not require a log parsing step and takes original log messages as input to avoid the noise. LogUAD uses Word2Vec to generate word vectors and generates weighted log sequence feature vectors with TF-IDF to handle the evolution of log statements. At last, a computationally efficient unsupervised clustering is exploited to detect the anomaly. We conducted extensive experiments on the public dataset from Blue Gene/L (BGL). Experimental results show that the F1-score of LogUAD can be improved by 67.25% compared to LogCluster.

Publisher
Computers, Materials and Continua (Tech Science Press)
Disciplines
Keywords
  • Feature extraction,
  • Log anomaly detection,
  • Log instability,
  • Word2Vec
Scopus ID

85119442088

Creative Commons License
Creative Commons Attribution 4.0 International
Indexed in Scopus
Yes
Open Access
Yes
Open Access Type
Hybrid: This publication is openly available in a subscription-based journal/series
Citation Information
Jin Wang, Changqing Zhao, Shiming He, Yu Gu, et al.. "LogUAD: Log unsupervised anomaly detection based on word2Vec" Computer Systems Science and Engineering Vol. 41 Iss. 3 (2022) p. 1207 - 1222 ISSN: <p><a href="https://v2.sherpa.ac.uk/id/publication/issn/0267-6192" target="_blank">0267-6192</a></p>
Available at: http://works.bepress.com/ahed-abugabah/28/