This work reflects the opinions of the author and should not be taken as official position of the Future of Life Institute (though they are compatible; for its official position see this page), or any other organization with which the author is affiliated.
I’m grateful to humans Mark Brakel, Ben Eisenpress, Anna Hehir, Carlos Gutierrez, Emilia Javorsky, Richard Mallah, Jordan Scharnhorst, Elyse Fulcher, Max Tegmark, and Jaan Tallinn for comments on the manuscript; to Tim Schrier for help with some references; to Taylor Jones and Elyse Fulcher for beautification of diagrams.
This work made limited use of generative AI models (Claude and ChatGPT) in its creation, for some editing and red-teaming. In the well-established standard of levels of AI involvement of creative works, this work would probably rate a 3/10. (There is in fact no such standard! But there should be.)