Good Benchmarks are Hard To Find: Toward the Benchmark for Information Retrieval Applications in Software EngineeringInformation Retrieval in Software Engineering, International Conference on Software Maintenance (ICSM): Philadelphia, PA.
AbstractSeven to eight years ago, the number of applications of Information Retrieval (IR) methods in Software Engineering was close to zero. These days, IR and text mining methods are accepted approaches to analysis of textual artifacts generated during the software lifcycle. The incentive to try IR methods in such analysis is strong: the field comes with a reputation for proven industrial and academic success, and some important Software Engineering problems related to textual artifacts, can be translated into an instance of a standard IR problem in a reasonably straightforward manner. In this position paper, we observe that part of the success of IR as a field came from the use of established, well-maintained, and almost universally accepted benchmarks for testing the work of IR methods. We elaborate on the question “Is the field mature enough to talk about benchmarking?” asked by the working session organizers. Our position is that without robust, well-designed time-tested, and, eventually well-established and accepted benchmarks, research on application of IR methods to problems in Software Engineering will not reach its full potential.
Citation InformationAlex Dekhtyar and Jane Huffman Hayes. "Good Benchmarks are Hard To Find: Toward the Benchmark for Information Retrieval Applications in Software Engineering" Information Retrieval in Software Engineering, International Conference on Software Maintenance (ICSM): Philadelphia, PA. (2006)
Available at: http://works.bepress.com/dekhtyar/54/