Jelani Nelson on why he joined Anthropic

Mike was my colleague at Berkeley, and he's been fairly skeptical of deep learning's trajectory for years. Most scientists are skeptics, and it can take a big shock to get someone to consider a possibility that goes against the anchoring effect coded in our DNA.

I too was somewhat of a skeptic until ~6 months ago, and people ask me all the time why I joined Anthropic.

It started in April, when a friend I've known for 20 yrs messaged me saying he'd be in the Bay Area in a week, and he suggested we meet up. I invited him to my house for dinner, and he told me he had just joined Anthropic. He said the models were becoming increasingly more capable, and he believed both that they'd soon be smarter than him and also that all intellectual work would be highly devalued in the near future, and lastly that society was not at all ready for what's coming.

I'd of course heard claims similar to this before, but not from a friend I'd known for this long, and who I trusted, at my dining room table. It was the initial shock needed to destabilize my own anchoring. Also, I've been fortunate to work with and/or mentor many smart people in my life (e.g. Fields medalists, a 4x Putnam Fellow, IOI first place winners, etc.), so I feel pretty calibrated in recognizing very smart people, and this was someone who is definitely very smart and with impressive such credentials. So, it was hard to ignore this friend saying the models would be smarter than even him in particular.

After that dinner, I decided to do an informal survey, calling and meeting up with a lot of my friends who now work on improving model capabilities at these frontier labs. They all agreed with that first friend. These were all people I'd known 10, 15, 20 years, who mentored me, or I mentored them, or we met professionally in academia. These are people I trust, and their beliefs on this topic were consistent.

I then spent a whole lot of time understanding the missions of different companies, and talking to current and former employees and people in leadership. I read Dario's (long) essays and watched videos on YouTube of company leaders being interviewed. Anthropic's stated mission "to ensure that the world safely makes the transition through transformative AI" felt like exactly the right guiding principle to handle such a society-disrupting technology, and everyone I spoke to at the company was very aligned with this mission. It's easily the most mission-focused place I've ever been at, and everyone takes the work extremely seriously and with a profound sense of responsibility.

And now here I am. And being on the inside, the pace of progress has surprised me more than I first imagined. The moment that pure mathematicians are in, to me, is just a microcosm of what I predict a much larger of society will be grappling with in the not too distant future, and I really hope our leaders and the general public internalize sooner than later just how significant a moment we're in.

I think this is an emotional moment for a lot of people, and the math community itself isn't of one unified mind on what should happen.

Perhaps the latest is the 3SUM/APSP result that came out last week. I'm not sure how carefully many people read that preprint, but the most bizarre part of the episode to me is what the prompt was: there was no prompt to solve 3SUM. The agent was working on something else, and decided to attack 3SUM on its own as a side quest then solved it (see page 63 of the preprint). Its solution purported to resolve 3SUM and APSP simultaneously, via a unifying problem that both reduce to.

So then, Anthropic is sitting on the refutation of 2 of the 3 major building blocks of fine-grained complexity. Just sitting on them and not telling the world doesn't seem right. Some mathematicians feel poorly written (by AI) manuscripts should be released as they are, and just let the community digest it. Dan Spielman, who I respect a lot, wrote in https://mathadjacent.com/p/the-big-reset-in-mathematics the exact opposite: AI companies should "find experts who are interested in the problems and the solutions, and to offer them the opportunity to write good solutions", which is exactly what Anthropic did.

Again, it's impossible to please everyone when there's no consensus amongst reasonable people. I think we’re in a phase of process experimentation while everyone is digesting the new world we're in.

Quoted post — Melanie Mitchell (@MelMitchell1): Michael Jordan (the machine learning one): "Superintelligence is Science Fiction" 🔥🔥🔥

https://www.di.ens.fr/~fbach/MJordan_ITW_Liberation_English.html