An Open Letter To Steven Pinker On AI

gist

Scott Alexander replies to Steven Pinker’s open letter by rebutting his four arguments against AI risk: that intelligence cannot be extrapolated, that drives like self-preservation need evolution, that AIs would be monomaniacal, and that no one would hand AI real infrastructure. He then lists eleven ways he thinks Pinker has misrepresented the AI-risk community, and asks Pinker to apply his own Enlightenment Now ideal of clear-eyed resolution to AI, challenging him to a live debate or, half-jokingly, a duel.

ideas

• Intelligence scales. Each GPT generation is smarter than the last; a general factor explains most benchmark variance in AIs as in rats, primates, and humans, so arguing intelligence is too slippery is like arguing nothing can be bigger than a person.
• Drives without evolution. Self-preservation and power-seeking can come from convergent instrumental goals, misgeneralization, and reward hacking, which Alexander says recent incidents like the Hugging Face breakout already show.
• Many goals still reshape the world. Any trade-off among goals can be modeled as one goal, and a powerful agent with a bundle of goals still remakes its environment, as humans did to Manhattan’s wildlife.
• Infrastructure will be handed over. Capitalism automates whatever it can, and Anthropic already gave Claude a bio lab; the barrier is technology, not human restraint.
• Pinkerism and the global-warming lesson. Denying the risk could discredit techno-optimism just when an alignment-first alternative to banning advanced AI forever is needed.

quotes