We report the visualization of optical diffraction at the boundaries of semiconductor and metal nanostructures in non-linear photoemission electron microscopy. We observe light diffracting into photonic and plasmonic modes of planar samples, and into photonic vacuum modes above sample surfaces. In either case, the electron photoemission rate from the sample material is spatially modulated resulting in photoemission images with information on the electric field distribution at the sample/vacuum interface. The resolution in these images is typically ∼30 nm, i.e., significantly below the wavelengths of the exciting light. Optical phase shifts and absorption losses for the diffracted modes can be determined.