In highly autonomous robotic systems, human operators are able to attend to their own, separate tasks, rather than directly operating the robot to accomplish their immediate task(s). At the same time, as operators attend to their own, separate tasks that do not directly involve the robotic system, they can end up lacking situation awareness (SA) when called on to recover from automation failure or from an unexpected event. In this paper, we describe the mechanisms of this problem, known as the out-of-the-loop performance problem, and describe why the problem may still exist in future robotic systems. Existing solutions to the problem, which focus on the level of automation, are reviewed. We describe our current empirical work, which aims to expand upon taxonomies of levels of automation to better understand how engineers of robotic systems may mitigate the problem.