This paper details lessons learned from human-robot interaction (HRI) research conducted with physical and virtual unmanned vehicle (UV) test beds. Two subject matter experts (SMEs) were interviewed regarding their experiences using the test beds, and content was categorized to address validity issues and practical concerns that researchers should consider when studying human operation of UVs. Interview data revealed that from a practical standpoint, virtual simulation is favored over physical simulation. Virtual simulation was also highlighted for permitting greater internal and construct validity than physical simulation. For tele-operation studies, however, physical simulation was identified as more externally valid than virtual simulation, but this advantage diminished for research at higher levels of autonomy. In general, this paper can serve as a resource for researchers who must choose between the two simulation methods.