Skip to main content
Article
Automating intention mining
IEEE Transactions on Software Engineering
  • Qiao HUANG, Zhejiang University
  • Xin XIA, Monash University
  • David LO, Singapore Management University
  • Gail C. MURPHY, University of British Columbia
Publication Type
Journal Article
Version
acceptedVersion
Publication Date
10-2018
Abstract

Developers frequently discuss aspects of the systems they are developing online. The comments they post to discussions form a rich information source about the system. Intention mining, a process introduced by Di Sorbo et al., classifies sentences in developer discussions to enable further analysis. As one example of use, intention mining has been used to help build various recommenders for software developers. The technique introduced by Di Sorbo et al. to categorize sentences is based on linguistic patterns derived from two projects. The limited number of data sources used in this earlier work introduces questions about the comprehensiveness of intention categories and whether the linguistic patterns used to identify the categories are generalizable to developer discussion recorded in other kinds of software artifacts (e.g., issue reports). To assess the comprehensiveness of the previously identified intention categories and the generalizability of the linguistic patterns for category identification, we manually created a new dataset, categorizing 5,408 sentences from issue reports of four projects in GitHub. Based on this manual effort, we refined the previous categories. We assess Di Sorbo et al.'s patterns on this dataset, finding that the accuracy rate achieved is low (0.31). To address the deficiencies of Di Sorbo et al.'s patterns, we propose and investigate a convolution neural network (CNN)-based approach to automatically classify sentences into different categories of intentions. Our approach optimizes CNN by integrating batch normalization to accelerate the training speed, and an automatic hyperparameter tuning approach to tune appropriate hyperparameters of CNN. Our approach achieves an accuracy of 0.84 on the new dataset, improving Di Sorbo et al.'s approach by 171%. We also apply our approach to improve an automated software engineering task, in which we use our proposed approach to rectify misclassified issue reports, thus reducing the bias introduced by such data to other studies. A case study on four open source projects with 2,076 issue reports shows that our approach achieves an average AUC score of 0.687, which improves other baselines by at least 16%.

Keywords
  • Tuning,
  • Data mining,
  • Computer bugs,
  • Software,
  • Linguistics,
  • Training,
  • Taxonomy
Identifier
10.1109/TSE.2018.2876340
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Copyright Owner and License
Authors
Creative Commons License
Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International
Comments

Dataset available at https://github.com/tkdsheep/Intention-Mining-TSE

Additional URL
https://doi.org/10.1109/TSE.2018.2876340
Citation Information
Qiao HUANG, Xin XIA, David LO and Gail C. MURPHY. "Automating intention mining" IEEE Transactions on Software Engineering Vol. 46 Iss. 10 (2018) p. 1098 - 1119 ISSN: 0098-5589
Available at: http://works.bepress.com/david_lo/200/