Visual assessment of objects is critical to many fields including metalcasting. While the output for such a task is often a simple attribute, theproblem studied here is for inspection tasks requiring an output that defines shape, size and location of anomalous areas, which are random and are not defined a priori. This paper defines a methodology to quantify the amount of repeatability and reproducibility variation. The application of the methodology for the visual surface assessment of steel castings reveals significant repeatability and reproducibility error. The work presented here is the impetus for current efforts that are defining the capabilities of the visual inspection process and ways to improve it through the selection,training and retraining of operators and through better control of the process.