The study focuses on the effectiveness of the training procedure including web-based qualifying tests and the monitoring procedure using web-based refresher tests in rating the South Carolina Arts Assessment Program (SCAAP) Grade 4 music performances tasks. Five rater groups, comprised of two raters each, scored both performance tasks. Generalizability (G) theory was used to estimate inter-rater reliability. The researchers concluded that the training procedure and the monitoring procedure with four refresher tests were effective in establishing satisfactory inter-rater reliability estimates for each task. In summary, the researchers believe that a remote rating system is effective when sufficient training and stringent monitoring procedure are employed.