Skip to main content
Article
An Empirical Study on the Adequacy of Testing in Open Source Projects
21st Asia-Pacific Software Engineering Conference (APSEC)
  • Pavneet Singh Kochhar, Singapore Management University
  • Ferdian Thung, Singapore Management University
  • David LO, Singapore Management University
  • Julia Lawall, INRIA, Regal
Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
12-2014
Abstract

During software maintenance, testing is crucial to ensure the quality of code as it evolves. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is an important metric to gauge the effectiveness of test cases and the adequacy of testing. However, what is the coverage level exhibited by large-scale open-source projects? What is the correlation between software metrics and the code coverage of the software?In this study, we investigate the state-of-the-practice of testing by measuring code coverage in open-source software projects. We examine over300 large open-source projects written in Java, coming from both the GitHub repository and the Debian Linux distribution, to measure the code coverage of their associated test cases. We analyse correlations between code coverage and relevant software metrics such as lines of code, cyclomatic complexity, and number of developers. We find that for the considered software developed under GitHub, average coverage is low, at 36%, and there is a significant negative correlation between coverage and both code size and complexity, i.e., coverage decreases with the increase in size and complexity of the software. The correlation between coverage and the number of developers is not significant. However, considering individual files in the GitHub software, coverage increases with the size and complexity, whereas the number of developers has no correlation with the code coverage. In contrast, the considered Debian software has a higher average rate of coverage, at 50%, but there is no correlation between coverage and any of the above metrics. Our results highlight the strengths and weaknesses of testing in open-source projects and make recommendations for future research.

Keywords
  • Code coverage,
  • software metrics,
  • software testing,
  • test adequacy
ISBN
9781479974252
Identifier
10.1109/APSEC.2014.42
Publisher
IEEE Computer Society
City or Country
Los Alamitos, CA
Copyright Owner and License
Authors
Creative Commons License
Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International
Additional URL
https://doi.org/10.1109/APSEC.2014.42
Citation Information
Pavneet Singh Kochhar, Ferdian Thung, David LO and Julia Lawall. "An Empirical Study on the Adequacy of Testing in Open Source Projects" 21st Asia-Pacific Software Engineering Conference (APSEC) (2014) p. 215 - 222
Available at: http://works.bepress.com/david_lo/176/