Skip to main content
Article
Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems
22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER)
  • PAVNEET SINGH KOCHHAR, Singapore Management University
  • FERDIAN THUNG, Singapore Management University
  • David LO, Singapore Management University
Publication Type
Conference Proceeding Article
Version
publishedVersion
Publication Date
3-2015
Abstract

During software maintenance, testing is a crucial activity to ensure the quality of program code as it evolves over time. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is often used as a yardstick to gauge the comprehensiveness of test cases and the adequacy of testing. A test suite quality is often measured by the number of bugs it can find (aka. kill). Previous studies have analysed the quality of a test suite by its ability to kill mutants, i.e., artificially seeded faults. However, mutants do not necessarily represent real bugs. Moreover, many studies use small programs which increases the threat of the applicability of the results on large real-world systems. In this paper, we analyse two large software systems to measure the relationship of code coverage and its effectiveness in killing real bugs from the software systems. We use Randoop, a random test generation tool to generate test suites with varying levels of coverage and run them to analyse if the test suites can kill each of the real bugs or not. In this preliminary study, we have performed an experiment on 67 and 92 real bugs from Apache HTTPClient and Mozilla Rhino, respectively. Our experiment finds that there is indeed statistically significant correlation between code coverage and bug kill effectiveness. The strengths of the correlation, however, differ for the two software systems. For HTTPClient, the correlation is moderate for both statement and branch coverage. For Rhino, the correlation is strong for both statement and branch coverage.

Identifier
10.1109/SANER.2015.7081877
Publisher
IEEE
City or Country
Montreal, Canada
Creative Commons License
Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International
Additional URL
http://dx.doi.org/10.1109/SANER.2015.7081877
Citation Information
PAVNEET SINGH KOCHHAR, FERDIAN THUNG and David LO. "Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems" 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER) (2015) p. 560 - 564
Available at: http://works.bepress.com/david_lo/214/