Inside Anthropic — AGI, Optimism, and Open Source

NEW 🚨 Inside Anthropic — AGI, AI Optimism, Regulatory Capture, Open Source & More!

@JTLonsdale sits down with Anthropic's @_sholtodouglas and @the_marwell

Have we reached AGI — and what does it mean? What are the most exciting breakthroughs at Anthropic right now? What are the biggest risks and concerns? What is their stance on open source? Is Anthropic angling for regulatory capture? Do AI researchers share same values as the heartland?

Episode 164 out now!

(00:00) Episode intro (02:00) Self-taught ML; Unlikely paths to Anthropic (04:30) What are the most exciting things you're working on? (08:04) Have we reached AGI — and what comes next? (10:10) How to measure AI progress (14:20) Is biology the next great frontier? (16:50) Career advice & the decade of the generalists (19:30) What could go wrong? What are the biggest AI risks? (23:11) Offense vs defense; how technology evolves (25:08) Answering the regulatory capture critique (30:45) Anthropic's stance on open source (31:50) Would you slow down and let China go ahead? (33:05) Anthropic's view on distillation (40:12) Is a post-scarcity world achievable? (48:54) How do we preserve values, tradition & community? (55:00) Predictions for 2028

models which are as or more capable than all humans are very likely to occur in the next couple of years. Each marginal unit of intelligence is [music] worth exponentially more than the unit that came before. You can keep increasing potentially doubling the productive intellectual and physical capacity of humanity in such a way that allows you to truly get to a post scarcity world. You guys are the edge of the AI frontier. What are the most exciting things you're working on? AI is going to be the the most important technology in the life sciences [music] certainly of our lifetime and maybe ever.

But it also comes with massive risks. We shouldn't be releasing these technologies into the world before we can keep the bad actors from doing a lot of effort. Tech leaders I know who are more skeptical who are saying oh well Dario is trying to scare everyone to create more rules to slow things down. So far we've slowed ourselves down much more than anyone else. Is there any scenario you'd slow America Nick Marwell and Schulto Douglas are key technical leaders at Anthropic, the fastest growing AI company in the world.

They're both very optimistic about the future despite some really serious threats we're facing as a civilization. What are these threats? How do we think about offensive versus defensive dynamics [music] in bio, in cyber, and other key areas? Some people are pretty angry at Enthropic for regulatory capture. What is their view on this? What is Enthropic trying to do? What do they think of the battles with distillation in China? If everyone's able to copy their models, how is Enthropic still going to be worth trillions of dollars?

And why? If you're a young person in the workforce, if you're trying to build your career, but you're not in the center of AI, what should you be doing for the next 10 years? An optimistic insiders view on one of the fastest moving companies in the world at a fascinating time. Excited for you to hear from them. Welcome to American Optimist. Really excited to have today two amazing talents from Enthropic. Nick, you're the key member of the technical staff, help lead RL science, which reinforcement learning science, and Schult Douglas, also a tech lead in reinforcement learning.

Is that right? Yep. And how how long have you guys been in Enthropic? Uh just over three years for me. The company's only how old though? It's five years or so. Yeah. It's amazing. It's like one of the most important biggest companies, fastest growing companies now in the world and it's only been around for 5 years. So this is a heady time here here in California. You're filming here in Napa today. Thanks for being here, guys. Well, thank you for inviting us.

It's going to be fun. Thanks for having us. Uh let's start with your backgrounds. Uh so where'd you grow up, Schult? Like where you from? Um I grew up in Sydney. uh I got very convinced that scaling and in general we were on track to get AGI uh in the 2020s in about 2020 uh reading a mixture of blog posts and papers um and I worked very hard on nights and weekends to prove that I could do the work of the quality bar that was demanded by a deep mind and anthropic and open eye and these kind of companies.

Had you studied computer science? I studied robotics in undergrad. Uh so I graduated in 2019 and then I uh was doing you know this my own research basically uh ended up working for DeepMind for two years and at anthropic for the last 18 months. Awesome. And Nick where you from? Yeah I'm actually from uh from San Francisco originally which is is is rare nowadays I suppose in in in folks in AI. Um I you know I grew up in a household with a a father in particular who had sort of spent the first half of his career doing sort of very traditional business things and

then had found this interesting path in in sort of the latter half of his his life and and sort of my more formative years where he had he had gone and and spent his time trying to figure out how to give back. Um and in particular figuring out how to leverage technology to to give back. Um, and I think a lot of sort of as I was thinking about what I wanted to do, I I was I was looking for things that sort of let me get this experience of uh of sort of both at the same time building the skills to sort of become

an effective uh sort of like capitalist or whatever you want to call it and also to to use this to to be able to sort of make a make social good. I think uh anthropic was sort of this this this very perfect uh coincidence uh where where I was getting very concerned about AI safety. I was actually a a resident working on a very different company at Thrive uh Thrive Capital at the time and it it it was just one of these things where it sort of came out of came out of nowhere and I was like I think this is the thing for me

to go do. Amazing. Well well since since you guys have joined Enthropic I think become has become the biggest of the AI companies that you guys don't release numbers officially but lots of stuff leaks. The stuff that leaks just for the audience it says it's you know here in August of 2026. I keep my face very still very still. Very still. No reactions. [laughter] You know it's like it's like supposedly doing over $80 billion of run rate, growing really fast, ahead of open AI, ahead of everyone.

Uh, like you guys are the edge of the AI frontier. Like what are the most exciting things you're working on internally right now that you can tell us about? What's what's it like to be at that cutting edge? Well, I think what I can talk about a lot is how the experience of programming has changed cuz I think this is the most visceral way that uh we felt progress. And 18 months ago, I was typing all of my lines of code by hand basically.

Then, you know, a few months into anthropic, I was guiding a model. I was intervening every few minutes. I'd say what I wanted it to do, write the next function or this kind of thing, but I'd have to check in and every few minutes because it would do it would make mistakes. It would go off course and this kind of thing. And where we're at now is I can ask models to do a day or two days of work independently and drive progress basically like a junior team member.

And that's just over the last 18 months, we've gone from what is really very much a tool where you have to stay in the loop to something that feels very much like a junior team member. So, as we're talking right now, they're the junior team members, the AI is working hard for you. Yeah. Yeah, they they are. There's a bunch of them working hard. What kind of stuff are they working you could say at is there any high level you could tell us?

Um, just a bunch of RL science stuff with with some [laughter] of these team members. Awesome. I I'll give a slightly different answer here. So I think I think a lot of uh sort of model development up to this point has really been about getting models to be able to to do tasks that humans are actually already quite good and capable at. Um and really what we're trying to do is it you know capabilities that bring lots of additional leverage to things humans were sort of already cognitively capable of achieving right building a piece of software that like a team of human developers could could

also have built in the same period of time. I think what you're starting to see now, and I think some of the the most recent sort of results in math, for example, are are a good example of this, is people trying to figure out how to use models to push past sort of the frontier of human intelligence and human achievement. This is like some of the recent math things. Yeah, there have been like a lot of new math uh math proofs and uh that have that have come out that are sort of things humans have been interested in for a long time and sort of

never been able to solve. Some of those it just seemed like a really smart mathematician realized that the computer is good enough to brute force something that a person couldn't have brute forced. Is you see what I'm saying? Like there some of them still seem like they're like good at specific things that humans are using or or is it is it really like a whole new theory that they're getting at or how do you think about that? I I think it it it is not sort of at least yet some some idea where like they're they're coming up with ideas that humans would have been

incapable of coming up with. But I actually think very little progress or or intellectual achievement ever looks like this in many ways. I think mo you know there there of course exceptions but I think most of the things that society has achieved are really just uh about sort of connecting the dots and building upon the prior intellectual achievements of things that came before. And I think that that what you're what you're seeing when you describe something like putting two ideas that different people had together to to find something new.

the foundation of like what humans do. And it's it's sort of the the important thing that's different now is it's it's really putting together sort of new ideas uh or sorry forming new ideas from combining uh other human ideas that that that weren't happening before. Whereas I don't think like writing another web app was actually some like new intellectual achievement for for humanity. And so you know we've never seen companies scale this fast in Silicon Valley. I think again not to 10x 10x 10x growing really fast obviously.

Uh we had another company here in Napa last night that I'm on the board of staying over and it's like it's like it's a new company every 60 to 90 days. There's so much happening, right? And and it's just I think this is like the experience of a lot of us. I think you're right in the core of like the fastest growing company. Like what's it what's it like versus like your life before? Like where do we go from here? Is it going to keep accelerating?

How do we think about this? I think one thing that's been really nice about anthropic has been that our ambitions are have sca been able to scale now with uh the with the acceleration of the company. you know, really felt like uh 18 months ago we had to be so insanely focused on code uh and really like proving out that we could be the best in the world at something. Um and now the mission of the company is so much grander than than make good models that are good at code.

And as the company has scaled, as the revenue has scaled, it's let us really start putting together the pieces to uh to start pushing on more of that. Let's talk more about that mission. I was with I was with another uh friend who's senior in AI, you know, earlier this week and I understand like he was saying how first you got to be the very best at code and math but and but then but then then there's going to be like new things it's going to be the very best at you can there's going to be all these other human skills you can get to

be A++ at is that the idea is bring in a bunch of new skills or like like what's the mission from here? Well, the mission from here is always we think that AGI is really is achievable in the next couple of years. Explain to our audience what you mean by that. Um, so we think that models which are as or more capable than all humans, uh, are very likely to occur in the next couple of years. So it's something that can do all of the things that a a human could do on a computer.

Um, or once we get sufficiently advanced robotics, all of the things that a human could do uh, in the physical world. So it's going to be like much better at the podcast. I could just let the model let the let the robot jump in. Well, you know, the podcast, [laughter] you know, let's go to the vineyard. It's more fun. Exactly. Um and we think this both has huge upsides, right? This it would allow you to multiply the intellectual uh like labor and physical labor done in the world today.

Uh it would allow you to get centuries of progress uh potentially uh much much much sooner. Yeah. But it also comes with ma massive risks and so really the goal of the company is shephering the world through those risks so that we can get the upsides on the other side. So let's step back on the AI wave in general like where we are on this CI wave. Obviously, you were saying in 2020, I think you realized you said the AGI was going to become possible.

I think that's right around GBT3 came out from OpenAI. I was looking back at some emails because you're doing like a tenure ADC book right now and I was making jokes even in 2020 about how we're going to replace all the associates and so it was it was kind it was kind of in the air that okay this is this is starting to become likely. Uh you know I think we talking to Dario earlier this year and by some metrics some capabilities were like doubling every every few months.

Like how do we think about quantifying progress? You talked about the fact you could just use the AI as opposed you know a junior employee. Is there any other way to think about like how much better it gets next couple years like metrics wise? Yeah. Uh I think there's a lot of evals and these are like basically exams where people test the models. Uh and on these progress is extremely rapid. We go from being able to do you know a great one is I don't want to lean too much on math as an example.

Um but math is maybe is the best in some ways. um where we went from 0% on a set of problems assembled by professors across different fields to well over 40 50 60% um just over the course of last year. Uh and that's an Evo called Frontier Math. Do you need you need something that's falsifiable, right? You need something iterate on. So could you like have it like talk to a girl on Instagram and like get a certain percent chance that it like gets her gets her want to go on a date with you?

Right. I mean is innocent enough to ask like that train on that. Right. [laughter] I'm married by not for me. We haven't trained on that. Um, but that's an example. Other sorts of things like that that'd be probably very useful for like especially some of our friends in San Francisco. Maybe there's two ways the models improve, right? One of them is via uh training on vast corpasses of text from the internet. And then the second form reinforcement learning what we both work on is doing these worked problems and checking whether or not you got the right answer.

Uh, and so this uh is what you're talking about this which is very easy for math and for computer science. It's pretty hard for romance or or poems or other things, right? Yeah. And that's why you've seen math and and computer science advance so much faster than everything else. But we actually think it's not that hard to put other problems in this in this frame. Yeah. I actually think there's like in in some ways a technical explanation for why math and computer science uh ca came first.

I also think to some degree there's like a social explanation for why this happened which is that you took a group of people who primarily came from backgrounds in math and computer science and you asked them to like make intelligence and of course they decided that the first things they wanted to do were to like make intelligence in the domains they understood um because they both were most interested in them. Um, and because it's much easier, you know, it's amazing how much time when you're trying to train good models, you actually spend like reading what the model is doing.

Um, and trying to understand as a result like uh how how capable it's becoming, what kinds of things you might need to focus on, sort of areas it needs to get better. And it's much easier to do this for domains where you're an expert, you understand what's going on. Do you anthropomorphize it when you think about it? I I don't think I I personally do. I think there are some people who who who who probably do and I'm sure some of them work at anthropic.

I mean I mean our brains are kind of programmed like there's whole there's whole parts of our brains that are set up to think of like the person we're looking at and mirror what we think their emotions are in order to talk to. So we have these all these things built in from evolution to like treat an intelligence as another intelligence and but it seems like it'd be hard to avoid doing that when you're working on this dayto-day. This is maybe one of the bigger debates in terms of how people try and frame these things as tools or as entities.

And I really think that entity is a useful frame because with a tool you are using it the entire time. Um, and with these models what you're really doing is you're you're taking this thing that is very intelligent and you're releasing it actually out into the world in some respects. Um, where it's going to take actions on your behalf. Uh, and so it's it's often more useful to think about these as as an entity for that reason. Are there categories or areas of things we should be having this intelligence do that we're not yet?

like there's things you guys are excited about are working on where like we're going to use this a lot more in these other areas like what are those areas? I think the big one's biology, right? Yeah. I think I think it depends if if you're asking a question that's about what are models already great at that we think are just underutilized skills versus where are we most excited about where models are going. I think I think on the latter question um and we can get more into why biology is the thing that I am I am most excited about over the next say 6 to

24 months in terms of where I think models will really begin to make an impact in the way that they have in software engineering but not just in the sense of that they'll be widely adopted but that the sort of societal positives that come out of this will be really diffuse um and really really powerful curing diseases and and making all of us healthier and longevity is It seems like there's all sorts of weird things about epigenetics that are tied to longevity that we're kind of just discovering through the Nobel Prizes starting 20 years ago and all of our friends have companies now in Bay

Area. There's billions of dollars trying to like figure out how to trigger these things and how to make part your body younger and stuff. So you think you think AI is going to be critical in advance? I think AI is going to be the the most important technology in the life sciences certainly of our lifetime and and maybe ever. In fact, I am I am confident to the point that I am I am almost more concerned that if we have the physical and lab infrastructure required to take advantage of this than I am about whether AI will will be able to to be a useful

intellectual uh three years ago we should have been probably building a lot more compute for example. We talk about this too much. Should someone be like putting creating a lot more like lab space for you to use or something right now? I think so. And I think I think that there are a couple of axes on which this is important. I think one axis on which this is important is is like literally how much lab space do you have. Um but I think that there are also a number of sort of metrics around quality of the labs um that we have in the US that

are clearly lagging behind places like China really. Um, and I think it is it is one of the the the great sort of societal infrastructure projects that people should be thinking about how to go solve is how to get the supply chain and lab complex in America back up to being sort of a gold standard best in the world. Yeah. This will be the right limiter on bi biology progress. Yeah. All right. Well, let's go let's go solve that after the podcast.

That sounds like a good one. You get some OSC money from the Department of War for Supply Chains and build some labs. uh just stepping back a little bit on where we are in the wave. Um what would you do as like a new college grad or someone who's in the workforce? You're not in anthropic. You're not in the middle of Silicon Valley. You might be a smart person who's ambitious. Like how should you think about your career given what's happening with you guys in the next few years?

So I think that there are a couple of different scenarios we could end up in. And anthropic, you know, sort of has a responsibility to speak about the risks that it's bringing into the world. And one of them is we're actually quite concerned about unemployment. Um, but I do think it's possible that this technology will take long enough to diffuse into the world. That there are a bunch of scenarios in which you have a a 10 or 20 year period uh where there are actually a ton of opportunities to to to to be part of the diffusion of this technology and sort of have that

be what defines your career. And a thing I've said to some folks, I mean I have a brother who's like in college right now and actually thinking about this exact exact question is I think that the next uh decade if if we are in such a scenario is really going to be one that sort of belongs to the generalists in a lot of ways. I think a lot of the way you sort of had a great career over the the the prior I don't know 20 or 50 years is you had a sort of set of skills that were very valuable and very marketable.

And it was it was very easy to have a set of skills Everyone has they have the power of a thousand person company at their back and what it's really going to uh mean is that the people who are capable of best understanding what are the problems to go out and be solved. what are the things that would make [clears throat] my community better? Whether it's from a a sort of like commercial standpoint, from a social good standpoint, etc. They'll have all of the tools they need to go to go affect that change.

And so, so sort of problem selection and things like that will be the defining skill that you said like what are the problems to [snorts] be solved? How do I make my community better? One of the one of the kind of fundamental backdrops to my view of the world and Isra Jewish is is you're supposed to like pretend you're like repairing the world six days a year and then you're pretend you're done on on the on the Sabbath and the day of rest which I think a lot of our major religions have something similar and and my optimism is that there's always going to be

things for each of us to kind of repair and help and fix with the people around us with the communities around us like unless you believe we're going to be in utopia there's going to be things to do for people right I think it's an optimistic framework absolutely yeah yeah there's a lot of work between us and utopia yes exactly so so so there's like the whole fundamental I think like the implication of the religion is it's not ever utopia here until like the Messiah comes or something and so therefore you have to like just keep fixing things right and that's like a positive thing.

Uh I know you guys are optimist there's lots of positive things. I want to go into some concerns though first of all because this is where a lot of people are very uh very curious about obviously on the coast I think we're seeing all these really positive things around us in our communities. We all have friends building companies faster than ever before. Uh it's just like this really positive energy. At least for me when I come out to San Francisco, when I go to New York, it's even things I'm doing in Austin, Texas.

But I think a lot of the rest of the country uh there feel like there's threats to their business, threats to their livelihood. It's scary. They don't know that you guys aren't just going to like be like some kind of wacky people who are going to like conquer with this or whatever. What are people missing like like what's what is there? Is there a plausible bad path that you worry about? Is the bad path they worry about wrong? Like like what's the how do you think about this problem?

I think the concerns are extremely reasonable. Um there's a couple of big categories that we're worried about. Um but all of them are ones that if we we think if we take the right actions over the next couple of years, we can end up with a radically better world in the 2030s. And so those categories are a big one we've mentioned already is unemployment, right? And we should have a long discussion around that and like why we think that's uh you know a risk and what we think you know the right things to do about that.

Uh there's also the more immediate near-term um uh concerns of bio and cyber risk that are literally happening like this year right now. Um there's there's all sorts of annoying things there where people would make fun of anthropic because you couldn't ask it about the mitochondria of a cell was a classic case and it's like well actually probably you should be able to do that and just maybe just don't help make a boweapon or something. You should definitely be allowed to do that.

Um but you know we're we're sort of putting in place the infrastructure so that we can feel confident that no one can make a boweapon out of them. Yeah. What's incredibly hard with things like bio and cyber is that these are inherently what's known as dualuse capabilities. So I think cyber is the easiest one to understand. If I prompt Claude and I say, "Here's a code base. Please find all of the vulnerabilities." Mh. It is just as likely that I am the owner of that codebase trying to harden my defenses as it is that I am the attacker trying to find my way in.

Yep. And this is what fundamentally makes things like cyber or bio which has many of the same sort of dual dual use characteristics so hard to police that you're trying to target something and you're in a in a in a cell and it's a really good chance probably like 99.9% chance you're trying to kill it to get rid of cancer or something but there's some tiny chance you're trying to target something terrible. Right. And so I think we're working on a number of things that are sort of meant to uh to to enable us to make sure that good actors get to use these technologies

and bad actors are kept. But I think that our our general feeling is we shouldn't be releasing these technologies into the world before we can keep the bad actors from doing there's there's there's definitely people and it's not even anthropic. I think they're using like open models from China uh with terrorist groups for example in Africa to figure out how to more easily make certain bombs that they didn't know how to make before and it's just smart enough to help them because they weren't very smart at that it is there's there's scary use but at the same time like I guess my bias is overall

with more intelligence more people can understand the scary uses and use to stop them as well right there's a trade-off and so so a big dynamic here that we should talk about is whether a area is offense or defense dominant. Yes. Right. Uh and right now uh it both cyber and bio are offense dominant but I think that over the next two years cyber becomes defense dominant especially if everyone has access all intelligence to to hack themselves and build it. Exactly.

Um because you can preemptively try and hack yourselves build up the defenses patch all the vulnerabilities and then eventually we'll end up in a much more cyber secure world where everyone's already has access intelligence they figure out how they're going to break in. So therefore that's that's what we're doing. That's what we're doing is we're ahead of time. I mean, I think one of my friends helped break in something at the Pentagon and the banks when they asked them and they showed them, but they're using new AI because AI is so good at it and so now everyone has to do it quickly.

Now, let's step back for a second. I think the offense defense thing is super interesting, right? This has been a concept for thousands of years in human history. It changes a lot. I'm a big fan of like the very free city states that used to compete in Europe and this was a really good thing and if one city state was doing something badly, you can go to the other one. And then unfortunately, these kind of jerks came along with cannons. they could just knock down the walls really easily and then they built these big empires and you couldn't just leave your city state anymore because

now you know now you're under them regardless and and so and so there's like different times when things have like moved between offense and defense and I agree I think cyber right now is super offense dominated to the point where like people should be scared by the way like anyone listening if you're not on the cutting edge using AI to challenge your systems they probably will be broken into by and it might be a very bad person so it's like so right now it's scary I I agree eventually cyber could become more defense dominated used correctly I I guess I guess there's a question with

AI like one of the fairs for me is like are there certain hidden things in the world that are just super offense dominated. So for example, and I don't believe this is the case, but for example, if you can make like a self-replicating nanobot with like enough intelligence that just eats the whole world and like one genius makes it and then eats the whole world and then we're all dead. And again, I don't think that's going to happen, but but I guess as a question like we I guess we have to make sure we're using the intelligence to find out whatever is super offense dominated

sooner. Right. Right. And and actually bio is super offense dominant until we put a huge amount of work into making the world defense dominant. There are ways you can make the world defense dominant against bio, but it's literally like it's going to require hundreds of billions of dollars of infrastructure. It's sort of become it's not unreasonable to do until you get dramatically advanced capabilities in robotics um in like the 2030s to uh help you build this out. So this is like a very scary period of a sense where like it's currently offense dominant.

We want to make it defense dominant. I guess the solution is to make sure these things are not too secretive and to maybe show them off and have lot like my view is you want lots of people using intelligence to challenge everything not just having like any single group doing secret. We very much agree on that note. Um, so Enthropic is uh generally like very admired. Also like any company I compared it to Microsoft in the 90s when you become like the champion people get jealous and attack you.

And like sometimes those attacks are because they're jealous and you have lots of aura and you're dominating. And sometimes there's certain things are legitimate as well. And those attacks um a lot of my very smart friends uh have impression that there's some kind of regulatory capture strategy is trying to do that's not that's like not a good thing and it's a bad thing. like like I'm curious we could talk more about like the details there but like how do you react to that like you know I think I think there's probably two things that are worth talking about here I think one is the sort

of rise of the regular cap regulatory capture narrative um tied to the open weights discussion that's been going on and then there's also probably just the like if you were trying to do regulatory capture in a very self-serving way would you have done it the way anthropic has I think on the first one it's not unreasonable sort of if you're if you're not in the details of what's going on to interpret something to the effect of open weights is is sort of an important public good technology that we want to keep around.

Um it is in some ways a natural opponent to parts of anthropics business. Y and anthropic wants to use uh any regulation it can to sort of push around uh the people who are are developing open weights because historically open weights or open source have have been communities that are are smaller, less wellunded and and have a hard time navigating the regulatory system. I think that's not what's going on with open weights here. If you look who the people who are pushing and developing open weights models and serving those models are it is largely the the the most gigantic corporations in the world right in

the US it's people like Nvidia and Amazon and Microsoft who are like pushing the frontier of open weights forward and then even uh with for example the Chinese labs these are extremely wellunded more often uh more and more often now public uh labs that are pushing this and so these are companies who are you Even [clears throat] though these are o openw weightights models, these are companies who are very capable of na navigating a regulatory environment. And so I'm not particularly concerned about their uh you know having some regulation of models both open and closed being something that slows down the developers of open weights

in a way that's dis disproportionate to the developers of closed weights. I think the other part of this is just like more directly if you were sort of doing regulatory capture in a way that was meant purely to advantage you. It's very unclear to me that anthropic would have done the set of things that it's chosen to do. Um I think that like if one one very good example of this and there are probably others and maybe fab uh maybe Schulto has some other things in mind is what happened with Fable, right?

If you think about Fable uh uh or or mythos before that um from a a a pure economic motivation standpoint, anthropic had this model that was far and away the best model in the world. Nobody had a model even close to it at the period of time. And Anthropic basically made a first decided to go work with the government to roll it out in a way the government was comfortable with when it could have chosen to take a multi-monthlong model lead and probably like put to bed any any questions of like could could other labs be competitive with the company.

And this was just a like decidedly bad economic decision for the company, but one that was sort of aligned with our with our values about how we think models should be deployed and regulated. And so it's it's not a strategy of regulatory capture for economic interests. It's a strategy of we think the government has a really important role to play in how critical and possibly dangerous technologies are developed and deployed. And we think it's our responsibility as someone at the frontier to to make sure that they're informed of that and that they participate in that process because otherwise who's going to represent everyone.

But but it's interesting too because obviously there's like a lot of people in the government actually there's quite a few smart people in this government. There's a lot of people who don't know what they're doing at all as well. And so you'll you'll end up with a silly circumstance where for example maybe you can't say it but I can say it which which is that like like anthropic would from my understanding would have had more cyber capabilities that people could have used for defense but then they were nerfed in order for the government to let it go out which I thought was a mistake.

I think that they are are leaning into the like classic American public private partnership that has been like such an important part like the it is not we we are not built in a way that the government has to be the most uh knowledgeable competent body about every subject in the world and what we want to be able to do is build like a great public private partnership where the government knows how to engage with anthropic with open AI with meta like whoever else it may be um to understand and then decide how to regulate into the people some of like maybe tech leaders I

know who are more skeptical who are saying who are saying just to put it out there like oh well Dario is trying to scare everyone to create more rules to slow things down to the because he's going to be one of the dawn people who owns that process like like what what's the answer to them so far we've slowed ourselves down much more than anyone else and in fact like if you take a look at um the the discussion around uh Demis' proposal for the the sort of closed model providers open AI Deep mind anthropic there to to coordinate in a way that they can

actually have uh basically thresholds like safety thresholds that we have to meet before we can release our models. This is us discussing slowing ourselves down because of our principles. Uh now I think also by the way we should clarify what our overall stance is on open source. Just what is that? What actually is it? Um at least personally we're very supportive of open source. like we genuinely believe that it's a very good thing for this to exist in the world. I learned how to you know do ML on open source models, right?

Um we also believe that there are a lot of risks uh and there will be increasing risks in future. Um so what we believe is that basically there should be thresholds for what society at large feels they're comfortable releasing. Um you know if a model can do cyber attacks, what threshold of capability are you comfortable with? And we believe that closed and open source models should have to meet the same bar. Um, other than that, people should be able to do what they want.

Is it worth even trying to do that if China's just going to be releasing random stuff that everyone could use anyway? Like what's the, you know, and similarly, what's the probability we overregulate at a fair and then China doesn't and then just there's a problem anyway and or they're ahead. Like how do you think about that? I mean, the ideal world is one where we can actually work with China uh such that they I mean, they also don't want cyber attacks on their country.

I think they want cyber attacks on us though is my impression. Maybe [laughter] they don't want cyber attacks on themselves. That's fair. The ideal world is one where we can work work together with them on that. Um but that's tricky, right? That's like that is going to be one of the most difficult parts of the next couple years is AI capabilities going to keep racing ahead. Yeah. Um this is going to make a lot of people very uncomfortable for many different reasons for buyer, for cyber, for employment.

Um and as a result you're seeing already for example like push back um of stopping the data sets. Uh and this is one of those worlds where or one one of those like debates where it becomes really tricky to pause unless you can arrange unless you can sort of coordinate with that with the other parties and trust the other parties to hold to that. China's not going to pause though. So right so is there any scenario you just slow America and let China go ahead?

Uh I don't think any scenario that we're comfortable with would involve slowing America and letting and and letting China go ahead. Yeah. Which seems to be the only option under our control right now. Yeah. For for any any slowing scenario, it would have to be fully coordinated in a way that we fully trust every counterparty. Basically, there's no way we'd slow America. You're surprisingly relative to other people think like more for open source. It's good thing it's affected your lives positively.

Um some of this open source stuff seems to be coming from people stealing it from you, which I would be annoyed by if I were you. And it's called distillation. And a lot of people say, "Well, why don't they just stop it by being really smart with their AI and watching everything?" And so, so, so, so why aren't you able to stop distillation? How do you feel about it? What's happening with that? Yeah. Well, on the one hand, distillation, let's there's a couple things to talk about here.

One of those is like why are we against distillation? Like, is distillation good or bad? And one of those is why can't you stop it if you think it's bad? Um, let's just close off the second, which is there's a bit of a cat-and- mouse game, right? Where actually people want a lot of access to their models. They want to be able to inspect the outputs. They want to run on their local computer. [snorts] they want to, you know, look at the thinking traces, but every additional bit of access you give makes it easier to distill.

And so there is this tension between how open your tool and product can be and how how easy it is to distill in effect. Yeah. Um and then I think on the first bit which is like you know why why should we be unhappy about distillation period. I think the way I I often think about it is this, right? You you you you want AI to make progress forwards because we think getting smarter and smarter models are going to do all of these great things for for for the world.

Um, and distillation is not a method that's capable of bringing you past the frontier of current just copying you. It's it's a copy. And the the the problem with this is that getting to the next frontier requires enormous upfront capital investments. We're going to be in regimes, forget about today, eventually we're going to be in regimes where there are 10 billion, hundred billion, trillion dollar training runs. And you can only support that kind of R&D investment if you're able to then make money selling the thing that you you invested in.

And the problem is if everyone else is able for a fraction of the cost to take your thing, copy it, and put out a copy in a week or two, they will actually be an economically advantaged competitor against you because they don't have this big R&D load. And so if you want progress on the frontier, then you want to protect the sort of IP and rights of people trading these models. I think I think that people also just to say one other thing often have compared this to sort of like drug development for example and I actually think AI is very interesting because I think

it it gets you a lot of like like protecting against distillation gets you a lot of the good things about uh how we protect IP and drug development and avoids some of the bad things. So what are the good things like we get more drug development in the US as a result of uh protecting IP from obviously I can't I can't invest in drug development if I can't make money on it. Exactly. Um some of the bad things are that like costs are extremely high as a result.

What's interesting about AI is unlike in drug development where like you get this protection for like a decade or more it's 20 or 30 years I think. um in AI like intelligence is it's it's incredibly deflationary because the shelf life of the frontier is like measured in months. Yeah. And so you imagine that like every 6 months yes the frontier is going to be expensive but everything behind the frontier which felt frontier 6 months ago is going to be made abundantly cheap by the fact that you push the frontier forward more.

This is sort of an artifact of how trading goes. So I think it gives you a lot of the good and actually like avoids most of the bad uh in sort of regimes uh that that we've been in and this I mean you like you can very concretely see this in the the cost of a given level intelligence. You can look at heaps of uh evad dashboards is one on artificial analysis that shows this exact trend line gets roughly 10x cheaper every year.

Um and it gets cheaper it gets cheaper after year. So this incredible deflationary extent um and by the way this is happening without distillation without distillation. just because of the natural progression of the technology. No, of course. And it's just intuitively unfair for someone to be able to copy and would and would break the whole system if everyone did it. Yeah. Maybe a follow-up question to you. You'd be like, well, hey, isn't that a crazy business model where like your thing gets obsoleted?

Well, you know, in some respects after like definitely a year, if not six months, the frontier seems like it has to keep moving forward for you to win. And the frontier feels like it has to keep moving forward if you win. And I think this is a very important part of like our [snorts] general worldview which is that actually AI has gone a very tiny amount into the global economy. It's gone you know like the sum total of AI revenues somewhere north of 100B right um into a tens and tens of trillions of dollars economy.

So and that's assuming the economy won't grow dramatically as a consequence of this technology which we expect it will and so there is this frontier in many ways you can think of like the level just behind the frontier like the lagging edge gets commoditized in many ways but then the leading edge has even more value. There's a huge amount more value at the leading edge. We often we often talk about this as sort of uh exponential economic returns to intelligence where each in each each marginal unit of intelligence that you're capable of is worth exponentially more than the the unit that came before it.

And and uh this is actually like you we've already seen this occur. So if you look at something like coding, right, like the first unit of intelligence in coding that was sort of like useful at all to people was tab autocomp completion in coding. And you moved from this into the next marginal unit of intelligence where you made agentic coding a reality in your terminal in your terminal with cloud code. And this was like a 1000,000x more valuable capability. And so even though tab autocomp completion got like commoditized behind it or something, this is like you can't even really have a company based on tab

autocomp completion at this point. Um these like next incremental units of intelligence are just extremely extreme. And this was very unintuitive to me when I was like thinking about anthropic a couple years ago cuz I cuz I would my intuition from like everything else I experience as a human being living in this world is that the exponentials don't go for that long. So I think okay well this is really valuable for a couple years but at some point there's not like an exponential going in some ways it requires you to be extremely optimistic right because you have to believe that like you're going to go

from tab completion and coding to agentic coding and but what could be more valuable than agentic coding I oh like maybe we can like cure cancer right and even between those it's like you have a fullon drop in software engineer that's better than any software engineer in the Y and that feels extraordinarily cheap. And you have an executive that helps you all sorts of things. And so by the time that people often talk about, oh, you know, like won't this just lead to commoditization?

By the time that any of those worlds you're even in like a hope of like complete commoditization, the entire economy is radically transformed. Literally the entire economy by that point is being done by AIS. Um both intellectual and physical. So yeah, and probably is a much much larger much larger economy. And if Enthropic is driving that forward, that's obviously worth trillions of dollars. So, yeah. Exactly. All right. So, we're all rich. Just kidding. [laughter] So, there's obviously a lot of concerns.

Uh but you're but you're optimist. So, so, so you're concerned like why like why are we pushing this ahead so fast? So, remind us again of that. What's going on here? What's the like what are we fighting for? Yeah. What are you fighting for? Um so, to come back to like why we're concerned because we're going to get something that's a drop in for a human, right? It's going to be as good as a human at everything they can do on a computer.

It's going to be once we have robots as good as a human at everything in the physical world. But at the same time, that unlocks massive like radical improvements. It means you can literally take technology that would have required the next couple of centuries to develop with our population and compress that into the next decade, couple decades. I'm already seeing this with like aerospace where there's airplane design stuff that's going so much faster that there's probably really advanced planes we would have gotten in like the 2040s that we're probably going to start seeing I think in the next few years.

It's really cool. And so you can keep increasing and potentially doubling the productive intellectual and physical capacity of humanity um in such a way that allows you to truly get to a post scarcity world like this is a world where literally everything is reduced to the cost of energy uh where building homes is reduced to the cost of energy. Yeah. The the middle class wall have giant homes if they want. Giant you could literally have whatever you want in that wall.

Yeah. You can go to the mountain and do like a 500,000 foot thing and it's a middle class thing. Exactly. Exactly. that would be middle class thing in the same way that things we have today are unimaginable to even the kings of like centuries ago, right? Um where we've cured every disease where people actually probably we've cured longevity. Um and so you're in this world where literally the the abundance everyone on earth has a level of abundance available only like not even to the richest people in the world today.

Um and that's the world we're fighting for. But it's a very tricky world because it means that you need to like one you have to sort of chart this course through to it and two you have to figure out well how do we even like share the proceeds of that world in such a way that you don't end up with massive inequality right that the returns don't just flow to people who h happen to have capital in the pre-existing world and instead everyone if they so want to um can have that kind of abundance and so there's a big social political question of of like

how do you uh share in the in the benefits Yeah, there's a lot of mistakes you can make by assuming we're already there though, too, right? Because the dignity of work right now is really and we're not there. We're not even close to there, right? Uh but if we invest in the technology in the right ways, then then we've got a shot of getting there. And so we think that like basically let's like draw the trend lines out a little bit from today, right?

over the course of the last four or five years, we've been 2xing or 3xing the amount of uh compute capacity devoted to AI uh every year. Uh and an interesting question will be and so I think that's roughly together the hyperscalers are spending about a trillion dollars this year on on capex related to to AI. A very interesting question will be will that keep that trend line hold right and so will it go to $2 trillion next year and then $4 trillion in 2028 and if that trend line broadly holds and you can sort of maybe expand that to encompass the broader robotics uh industry

and this kind of stuff um then that means that in the early 2030s or something like this uh you start to get to the point where you're actually effectively doubling the uh like the GDP of uh humanity um in the early 2030s, which again, a little bit of a ridiculous concept, but requires a lot of things to go right, but it see it seems like the productivity growth could just start productivity go really. I mean, yeah, and it feels like it should even be in the next couple years.

I think something we really notice like in the statistics. Kevin Walsh is the new chairman of the Fed. He is I don't know if he's AGI pill, but he's very bullish on AI productivity right now. Very smart friend of ours. Like it feels like we're going to start seeing that a lot more, but then that'd be like the kind of it feels like insane to say, but it could just accelerate much. It just accelerate even more. And I think like one of the interesting um charts I saw recently was this chart of uh of employment actually in the Philippines which went up.

Um I was surprised by this. I thought the BO was going to get really hit there. BO stand for business process outsourcing. They're doing like the low-end call centers but and yet they're still growing. And yet they're still growing and actually software engineering employment is up as well. Um and I think a lot of this is because all of a sudden people are dramatically more productive than they were before. Um, and a very interesting trend line would just be like how long does this bump continue where people sort of you're pairing in in the same way like there's so much more to do.

There's so much more to do. You're using this stuff with this positive sign. I I well at least in the short term it seems that way. But I do think I do think it's important that we uh we we we don't convince oursel too much that it's just like all going to be fine for it's going to be disruptive volatility right now. Right now, as Shto put it, you're in this in this it's actually possibly good moment where all your employees are still very important parts of taking advantage of AI.

Yes. And as a result, what happened is you got an enabling technology that made every one of them more valuable and so you wanted to hire more of them. But at some point, our belief is that we're going to go from a world where you need a human paired with an AI model to sort of get all the value out of it to a world where you don't. And I think that's the world where we start to get very nervous about the consequences down.

This is like with chess for 30 years it was like the best chess player was a human plus the machine and now the machines are just crashing and we think that's just a much shorter time period. Yeah. I think I think one thing that uh was a big update for me over the last three years was when I first started the maybe over the last five years when I first started thinking about sort of the rate of AI progress. Um, I I believed eventually we would sort of get to a place that looked like where we are today or where I think we'll be in

a year, but that it would take longer. And in large part, I thought it would take longer because I thought that scaling the data required to do this was just going to be an extremely extremely time and dollar uh uh dollar expensive pursuit. Um and the thing that happened that even by my sort of like lofty uh expectations at the time of AI progress uh was was was was unexpected was sort of that the the the ramp of the revenue and the scale of these businesses of anthropic and open AAI was so fast that all of a sudden it became completely plausible for the people

developing this technology to throw money at the problem of blasting through much faster. what we thought a lot of these bottlenecks might be. The compute ramp is another great example of this. It would have been inconceivable to most people 5 years ago. We would spend a trillion dollars in a year. It was inconceivable to people a year ago, right? It was a year ago. [laughter] And [snorts] so it it is kind of cool how it worked out how all these giant tech companies always had like hundreds of billions of cash they were hoarding and we kind of we criticized them for it and now they

actually were able to use it now right at the right time. It really worked out. It does seem like uh we are in a timeline where a lot of stuff that actually has is is sort of very large interaction term with each other is happening at once some good and some bad right we talk about uh the the the the the move into space for compute it's like kind of interesting that like AI is coming around at roughly the same time that SpaceX is coming into its own very worked out very well it it like the timing of that was like really interesting you could

imagine if if SpaceX had taken 10 years longer to get here that AI timelines might have might have existed. It's the only things to Elon too. He was like crazy enough to start it like way before anyone else. On the other hand, by the way, you have some much more concerning things that are happening. Like I don't think it's great for the world for all sorts of reasons, including race dynamics, including that AI I think fundamentally is a technology that can benefit uh authoritarianism more than uh more than most other forms of government.

That AI is arriving at a time when we're entering one of the like certainly the most bipolar world in my lifetime. Isn't that linked though? Like isn't isn't the Isn't there something about like social media and like how it uses machine learning and how it uses improving AI that does polarize us? Like is it something about the attention economy? I think when I'm saying bipolar, I'm really referring to sort of like the China US global picture. You don't you don't mean the left right polarization US.

No, I mean that's probably another thing that's worth thinking of. But I I I I really do think I would have updated us at least slightly more positively on like the prospects for AI going very well if if it was coming into the world say in like the early 2000s when you have this very unipolar more unipolar world it's more safer. Yeah. Yeah. It's like a tragedy worthy of a sci-fi novel that the most valuable resource in the universe is produced in this little island between the two world superpowers.

It's very is very it is very quite poetic in some respects. And so I mean I mean overall there's a lot of optimism in terms of where this goes for humanity if we get through this period. It seems to be seems to be the high level thing. One other question I want to ask I think a lot of our listeners are wondering about is it's one thing to have like extraordinary wealth but there's also uh things we we care about in our civilization.

There's there's virtues there's families there's there's like traditional values which a lot of people like base their lives on. There's a dignity of work obviously, but there's all these other things around that which is like our traditions that make us who we are and and I think there's some concern that because this is happening in San Francisco where those traditions maybe are not as valued on average that is there some kind of agency of the people in charge where now they're just going to like remake the world and like get rid of my like traditional sense of whether it's Christianity or whether it's it's like

you know monogamy or what whatever else like and like and like are there institutions we can be building to help preserve these things? How how do we think about that from the perspective of being the center of San Francisco and changing everything? I I think one it's interesting in San Francisco we actually expect a resurgence um in importance of these institutions because as work decreases in its primacy to people's lives and as like pure capitalism decreases in its primacy to people's lives because we sort of progressively have more and more and more abundance.

we think that the the important parts of people's lives are going to be tradition and ritual and providing values to other people and uh and your local community and sort of those bonds. Um I'm actually yeah I I also think like counterintuitively uh sort of sort of many many sort of intellectual pursuits and the the the sort of pursuit of figuring out how to become someone who knows how to think about the world and and and and is is is actually something that that perhaps grows in importance just not in a in a sort of economic sense right um I think that uh in a

in a world where you want for nothing it's sort of the the last thing that's that's left to to sort of entertain yourself to to feel part of community and and all of these things. So you you guys are both young and very successful. Let's just say Anthropic keeps working out. You're obviously everyone there is going to have resources and the whole world's got more resources. Like have you thought about building institutions to help with these things? There's something you're passionate about there to create.

Yes. So, I mean, Nick, you've got a great example with the early childhood stuff um that your dad's working on, but I think uh it's this is a very big topic of discussion at anthropic right now. You know, philanthropy is a very important part of anthropics culture. Um huge amounts uh I think uh there's a great blog post on this called like new wave of American philanthropy or something like this. um where people talk about how it's the philanthropic outpourings of opening ice uh foundation and the employee base of anthropic um would be like hundreds of arc institutes or like tens of like gates foundations

or something like this. No um which and there's there's all kinds of things that people are thinking of one thing uh foundation that I've donated to is a program that intends to try and end all viral disease. Uh and so like a lot of people are focusing on things which are are health related. One of my colleagues has donated to uh basically a not for profofit that's focusing on trying to figure out what the right way to do uh this like how to handle labor force impacts basically really study them figure out what's going on what should we be actually concerned about what's the real

picture what's the data on the ground and what policy should we be advocating for so that everyone makes it through this like transition in a way that they feel supported and they they're better off basically on the other side um and so uh yeah I mean I I am am certainly heavily influenced I think by by by my father in this regard who who spent probably the last 15 or 20 years of his life or at least the last 15 or 20 years uh focused on education.

I just talked about how I think you know education while the things we may want to learn I think will change a lot. Um it's unclear to me that you know we want to be going out and getting PhDs in in highly technical fields 20 years from now. Um I do think the sort of like pursuit of knowledge and the joy of learning and the ability to engage intellectually with one another um and and and frankly to understand the world that we're living in which is going to be a very confusing world for most of us if if things play out in sort of the

way we've been discussing is it is only growing in importance. Um, I think that uh one thing that many people in AI believe that I actually uh think they get wrong is that AI is just going to fix education. And I think that um if you want to understand like if you care about the education of like everybody and not just say like how do I give a really great education to a child with two parents who are in the 1% and who are highly educated themselves.

I think that there's two really important things to understand. The first one is um your education is a like deeply deeply compounding thing. Yes. And the so like a good example of this is the number one predictor of how well a student will do in math is whether they can read at grade level at the end of third grade because the problem is that when you get to the end of third grade this is where you transition from learning to read to reading to learn.

Yeah. And so and there are many things in education that are like this especially because children have and and many humans have this psychological thing where when they perceive themselves to not be good at something they lose motivation to keep doing it. Um, and and the problem, if you believe this, is that there's one thing that at least so far AI seems to be extremely bad at, and it is like computers cannot hold the attention of young kids. In fact, they seem to be an incredibly distracting force.

Well, that could be a whole new thing you train them on if you want. You got to figure out the game, although I'm not I'm not sure. And so while I think that there are are parts of the process of educating children that AI will help in in extreme ways in particular sort of making sure we're teaching kids the right thing for them at the right time. I I really don't think that sort of the uh there it's going to be this panacea that we just like get really smart AI and like all of the problems of education going.

Well, there's a lot of smart people working. I agree is really critical. So guys, flash forward to August 2028. You're looking back at this podcast. What [snorts] has to have happen for you to say like, "Wow, that went something went much faster than I expected." Much faster than we expected. [laughter] We expect things to go pretty fast. Much faster. I mean, I think timelines to get data centers in space is a big one. I think there's like wide error bars on this depending on who you ask.

And that's a really big part of it because there's so much more you can do up there. The compute ramp once you have that will be even faster to go faster and that's the main thing we need. Not going to surprise you. Maybe. Um I I broadly expect something like terraab to be online by then but TBD on the actual like ramp. Um and so some question of how much ah no okay I would be interested to see how many humanoid robots we have in homes in 2028.

So, I [snorts] think this is very likely to be like a whole bunch of early deployments, you know, maybe tens of thousands of robots in 28 um in homes doing laundry and like some basic cleaning and this kind of stuff. Uh but it could also be a fair bit faster than that. Yeah, we just put out an episode with generalists. It's like it's pretty amazing how good the world models are getting and it's faster than I expected. Like it seems like that could be that could be the thing that makes everything go a few years faster.

Exactly. And then once you get, you know, as I said before, once you get robots, it really compounds like progress goes fast. Yeah. I think there are some other things. I mean, I think a lot of the things here are are are data related in some ways. Like I think I think one of the slowest uh or sort of like the biggest regulators of of of progress in in AI capabilities are how fast can we scale the data in the domains that we care about.

And uh I think that this has already gone faster than I expected for sort of a reason I discussed earlier, which is there's been a lot more money to throw at the problem than I thought there would be because of because of what lab revenues have become. But I think we're my expectation is we we will be be slowed by sort of like some rate uh at which like humanity can produce this data. And if you told me, for example, that we we went on a crusade of like building tons of labs to like produce biological data for training models or something like that and

that started today and it was in full swing a year from now, I probably would update my timeline sooner. It's fun. Now we're focused on obviously all these types of data for this very specific task like coding and math and everything else. I wonder you think one day we'll be working on data for like shaping virtuous people or good character or mental health or something like like that. I mean obviously it's much more complicated but could could there be these other kind of interesting really complex problems the AI solves in the future?

Those problems I kind of want humans to solve to be honest. Well could wouldn't we use a tool to solve the problem though? Um I mean definitely I think AI will help us in those discussions right? Uh but I think that falls into the broad category of data which I I expect to be like both quite tricky to scale and extremely dependent on the person and just like just very uh that comes later than than we solve you know robotics you're still somewhat human it sounds like there's still still room for us in these I guess like one thing you know would I expect an

AI to win a pullet surprise in 28 it's like probably not well what if it didn't we didn't know we were giving it to the AI yeah that's true [laughter] um would I expect it to win a Fields medal in 28 quite quite possibly, right? Extremely likely. Nobel Prize also not crazy. Um, very feels very likely by before the end of the decade. Well, it's exciting to chat with you guys. We definitely have some challenges to work through, but I but I'm feeling very optimistic about the future of humanity if we get this stuff right.

Me, too. And thank you for joining us.