Skip to main content
Article
The AGI containment problem
Faculty Scholarship
  • James Babcock, Cornell University
  • Jànos Kramàr, University of Montreal
  • Roman Yampolskiy, University of Louisville
Document Type
Conference Proceeding
Publication Date
7-13-2016
Department
Computer Engineering and Computer Science
Disciplines
Abstract

There is considerable uncertainty about what properties, capabilities and motivations future AGIs will have. In some plausible scenarios, AGIs may pose security risks arising from accidents and defects. In order to mitigate these risks, prudent early AGI research teams will perform significant testing on their creations before use. Unfortunately, if an AGI has human-level or greater intelligence, testing itself may not be safe; some natural AGI goal systems create emergent incentives for AGIs to tamper with their test environments, make copies of themselves on the internet, or convince developers and operators to do dangerous things. In this paper, we survey the AGI containment problem – the question of how to build a container in which tests can be conducted safely and reliably, even on AGIs with unknown motivations and capabilities that could be dangerous. We identify requirements for AGI containers, available mechanisms, and weaknesses that need to be addressed.

Comments
9th International Conference, AGI 2016, New York, NY, USA, July 16-19, 2016, Proceedings
DOI
10.1007/978-3-319-41649-6_6
ORCID
0000-0001-9637-1161
Citation Information
James Babcock, Jànos Kramàr and Roman Yampolskiy. "The AGI containment problem" (2016)
Available at: http://works.bepress.com/roman-yampolskiy/29/