Skip to main content
Article
Code coverage and postrelease defects: A large-scale study on open source projects
IEEE Transactions on Reliability
  • Pavneet Singh KOCHHAR, Singapore Management University
  • David LO, Singapore Management University
  • Julia LAWALL, INRIA
  • Nachiappan NAGAPPAN, Microsoft Research
Publication Type
Journal Article
Version
acceptedVersion
Publication Date
9-2017
Abstract

Testing is a pivotal activity in ensuring the quality of software. Code coverage is a common metric used as a yardstick to measure the efficacy and adequacy of testing. However, does higher coverage actually lead to a decline in postrelease bugs? Do files that have higher test coverage actually have fewer bug reports? The direct relationship between code coverage and actual bug reports has not yet been analyzed via a comprehensive empirical study on real bugs. Past studies only involve a few software systems or artificially injected bugs (mutants). In this empirical study, we examine these questions in the context of open-source software projects based on their actual reported bugs. We analyze 100 large open-source Java projects and measure the code coverage of the test cases that come along with these projects. We collect real bugs logged in the issue tracking system after the release of the software and analyze the correlations between code coverage and these bugs. We also collect other metrics such as cyclomatic complexity and lines of code, which are used to normalize the number of bugs and coverage to correlate with other metrics as well as use these metrics in regression analysis. Our results show that coverage has an insignificant correlation with the number of bugs that are found after the release of the software at the project level, and no such correlation at the file level.

Keywords
  • Code coverage,
  • Computer bugs,
  • Correlation empirical study,
  • Measurement,
  • Open source software,
  • open-source postrelease defects,
  • software testing,
  • Sonar measurements,
  • Codes (symbols),
  • Computer software,
  • Correlation methods,
  • Defects,
  • Information dissemination,
  • Measurements,
  • Open systems,
  • Program debugging,
  • Regression analysis,
  • Software engineering,
  • Software testing,
  • Code coverage,
  • Computer bugs,
  • Empirical studies,
  • Open sources,
  • Sonar measurements,
  • Open source software
Identifier
10.1109/TR.2017.2727062
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Copyright Owner and License
Authors
Creative Commons License
Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International
Additional URL
https://doi.org/10.1109/TR.2017.2727062
Citation Information
Pavneet Singh KOCHHAR, David LO, Julia LAWALL and Nachiappan NAGAPPAN. "Code coverage and postrelease defects: A large-scale study on open source projects" IEEE Transactions on Reliability Vol. 66 Iss. 4 (2017) p. 1213 - 1228 ISSN: 0018-9529
Available at: http://works.bepress.com/david_lo/221/