Fast volumetric non-destructive testing methods are needed, especially for quality control in manufacturing lines. Ultrasonic testing with full waveform evaluation is a promising method for this. However, changes in coupling conditions or environmental factors can significantly alter the ultrasound signal, sometimes more than actual defects. This study investigates the effect of various factors on the ultrasound signal based on a Monte Carlo study with wavefield simulations. The test specimens comprise aluminium plates with holes of varying sizes and positions. Using both experimental as well as simulated data, the performance of two commonly used comparison metrics, namely the R2 score and the magnitudesquared coherence integral, for detecting defects in manufactured parts is evaluated. It was found that the magnitude-squared coherence integral is more robust against random influences than the R2 score. Additionally, factors influencing the entire plate exhibit the most significant impact on the signals. The hole positions and dimensions change the signals and the value of the comparison metrics significantly and are difficult to distinguish by one metric. A deep learning model, however, is capable of performing this task and it outperforms the comparison metrics in defect detection. The performance of the approaches is assessed with probability of detection curves.