Written by a DeepMind co-founder, which means the pessimism carries more weight than it would from an outsider.
The core idea is containment. AI and synthetic biology are both omni-use, cheap to copy, and improving fast, which makes them unlike any previous technology we have tried to govern. Nuclear containment worked partly because enriching uranium is hard. Nothing about model weights is hard in that way.
The phrase that stuck with me is pessimism aversion: the tendency of serious people to avoid saying the alarming thing because alarm is socially costly.
Where it is weakest is the prescription. The narrow path between catastrophe and surveillance gets described more than it gets mapped. Fair enough. Nobody has mapped it.
Read January 2024.