Skip to main content
Unpublished Paper
Assessing Computer Searches Used for Systematic Reviews
  • Paul Fehrmann, Kent State University - Kent Campus
Systematic reviews (SR) are valued for increasing the rigor of and reader confidence in “reviews of the literature”. SR have been widely adopted in the health and social sciences. Computer searches are key for the information base of SR, and computer searches must be evaluated. The Computer Search Report Checklist (CSRC) is being developed to evaluate computer search reports. This paper has 4 objectives: 1. To report on development of the Computer Search Report Checklist (CSRC). 2. To report comparisons to AMSTAR, PRISMA, PRESS, and others. 3. To report on the inter-rater agreement for CSRC items. 4. To report on using the CSRC to assess evidence for the comprehensiveness and reproducibility of computer searches in SR. Methods to address paper objectives: 1. and 2. Major SR guidelines and assessment literature were evaluated recommendations used. Author experience was also used. 2. Item inter-rater agreement wre evaluated with psychology SR and Campbell Library SR. 3. Psychology SR and Campbell Collaboration SR were evaluated with CSRC items. Results: 1. A CSRC timeline is presented. 2. Guidance varies for assessing reproducibility and comprehensiveness. 3. Recent Kappas for 10 comprehensiveness items: 6 items moderate-perfect; 1 item poor; 3 items a constant value (so no Kappa). 4. Recent Kappas for 33 reproducibility items: 16 items moderate-perfect; 11 items fair-poor; 6 items a constant value (so no Kappa). 5. Article (2011) in Research Synthesis Methods : comprehensivness of computer searches in computer search reports of psychology SR and Campbell Collaboration SR. 6. Reproducibility in psychology SR and Campbell Collaboration SR: article in preparation. Key Conclusions/reccommendations: 1. Further testing can verify potential value of CSRC. 2. CSRC might be used as a supplement to existing SR assessment tools. 3. External input on CSRC design, use and value should be gathered.
  • systematic reviews,
  • assessment,
  • checklists
Publication Date
Summer August 8, 2015
Citation Information
Paul Fehrmann. "Assessing Computer Searches Used for Systematic Reviews" (2015)
Available at: