We report precision determinations of the beam-normal single spin asymmetries (An) in the elastic scattering of 0.95 and 2.18 GeV electrons off 12C, 40Ca, 48Ca, and 208Pb at very forward angles where the most detailed theoretical calculations have been performed. The first measurements of An for 40Ca and 48Ca are found to be similar to that of 12C, consistent with expectations and thus demonstrating the validity of theoretical calculations for nuclei with Z≤20. We also report An for 208Pb at two new momentum transfers (Q2) extending the previous measurement. Our new data confirm the surprising result previously reported, with all three data points showing significant disagreement with the results from the Z≤20 nuclei. These data confirm our basic understanding of the underlying dynamics that govern An for nuclei containing ≲ 50 nucleons, but point to the need for further investigation to understand the unusual An behavior discovered for scattering off 208 Pb.