Post-AGI Statecraft

transcript

I think it's probably not controversial here, but it seems to be under-appreciated, at least in DC, that AGI is first and foremost going to be felt as an organizational shock — not just in terms of first-order risks like from CBRN or things like that. Neglecting this fact causes us to treat AI policy debates as if the basic institutional structure we're going to have through this transition is going to be relatively constant. But this is defied by basically every other major technological transition, which has always been accompanied by equally transformative institutional change.

The agricultural revolution didn't just create plenty and lots more food — it also moved us from flat nomadic societies to situated societies of hierarchies and divisions of labor. And those are both a cause and consequence of that revolution. You needed the innovations in livestock and agriculture to get off the ground in the first place, but then the new institutions sort of enabled it by having a class of agrarian workers and people who could do the record-keeping and everything else.

And so I think, as we think about AI policy more generally, we have to take into account that the institutional forms where we embed those policies are probably going to look very different. And this should be obvious because the companies are working towards this, right?

OpenAI's roadmap is right now — I guess, since we've just solved the narrow-ish problem — we're at the innovating AI. So the next stage is the organizational AI. I think you could then do another leap to the nation-state AI, because these are just organizations all the way down.

He thinks to himself, well, if a Martian — say they had some kind of dashboard, like a Google Maps that overlaid some social science on Earth — if they looked down, what they would see is basically organizations. They would see these clumps of firms and corporations and churches and community centers and lines connecting them, referencing the relations. And then within the firms, they would see lines representing the hierarchies. What they wouldn't see is just the market, right? The market isn't a thing. The market is like a subtraction that we overlay between the firms.

However, in traditional economics, we tend to take this market-first view, where companies are monopolists or perfectly competitive, or maybe they have some simple production function, but all the attention goes on the lines in between firms and to the market dynamics. Whereas reality is much more organizational.

And you can think of this as very similar to the boundaries of anything — the boundaries of a cell, the boundaries of a country, the boundaries of a firm. We have a kind of blanket that represents what's internal and what's external, and ways of doing inference, active inference across those boundaries. For cells, it's biochemical signaling or things like that. For companies, it's prices, it's marketing surveys, and then the outputs are production. And they have all this internal structure where all the interesting things happen.

Obviously, if we get AGI, the boundaries of these things are going to change. The internals are going to change.

And it's kind of blankets all the way down. We have cells that turn into larger organisms and those turn into people and turn into nation states. And so we should expect that — and we kind of already have this intuition that there's going to be a SaaS-pocalypse and other companies are going to get displaced by new kinds of companies. And we're going to have multi-agent teams that replace whole departments. But what does that mean at the level of our basic institutions?

And when you start applying this framework, you can import other ideas from information and systems theory, and you can get these kind of percolation events.

So this is just an arbitrary made-up system, but you can imagine if the alien was looking down, this would be like feudal Europe, and then some technology change, some parameter changes, and suddenly we start merging into larger nation states. And maybe in the 20th century, we thought we were going to go all the way and become some globalized super-state. But these things varied based on the technology.

And we see this even with mere information technologies, right? Could a mere information technology take down the nation state? Well, it kind of happened with the Arab Spring. And it kind of almost happened to us through very, very similar dynamics — basically, technology creating new kinds of long-distance correlations for people who have very different views, or minority views, able to coordinate with each other and mobilize.

And so I sometimes think of this in terms of a legibility arms race. I think of it as an arms race because we sometimes have this intuition that AI is going to make the world much less opaque and everything super legible. And there's ways in which that's true.

When Tesla recalled vehicles in 2022, it's because the Highway Traffic Safety Administration said that they were doing rolling stops. And they were able to tell that because they have telemetry on every single car, which is not normal. Human drivers don't have telemetry on all their cars that tell them when they break minor traffic laws. So that's an example of extreme legibility.

On the other hand, we have this crazy diagram, which comes from a GAO report of large multi-tiered partnerships. These are basically partnerships where every node in this diagram is some kind of different corporate entity. And they all have these intricate connections. They may actually only have six real employees, but they are structured like this to obfuscate their tax liability. In 2002, these were 1% of corporations with 20 or more tiers. Circa 2019, they had risen to 30% of all large partnerships — enabled, presumably, by tax software and information technology that made it much easier to obfuscate and do these kind of complex corporate arrangements.

Now, if we all have perfect tax accountants in our pocket — that we don't even have to have the idea of doing this, that just sort of inferred that this is a good way to find IRS code vulnerabilities — the world's going to get much less legible on those dimensions.

And part of this, I think, is a symptom of our broader regulatory paradigm.

You can think of the way we do regulation in the 20th century as the GOFAI version of regulation. It's formally verifiable, symbolic, rules-based. But by the same token, it's static, it's brittle, and overly procedural. So it doesn't generalize very easily if the conditions change.

And so I think we need to more broadly think about what comes next — moving from the good old-fashioned AI era to the deep learning era for regulation, for governance more generally.

These systems will be by nature more inscrutable, because they'll be basically learned or grown. But they'll also be generative. They'll be driven by outcomes rather than process. They'll be able to evolve and be continuous. They'll be more contextual. One reason we have rules and processes is because we're trying to remove human judgment, because human judgment can be partial. But if an AI can sort of pass the test of "I know it when I see it" — AIs have this rich semantic understanding — we could potentially import more judgmental forms of regulation.

So broadly speaking, I think of the kinds of governance we're going to need to move towards as ones where we borrow a lot of these analogies from deep learning — where we move from literal constitutions to things like hyperparameters, from statutes and legislation to loss functions, from rule-makings to reward models where we directly specify the thing that we care about, from technical standards to benchmarks, codes of conduct to evaluations, education to classifiers, and audits and inspections to different kinds of scalable oversight.

But the problem ahead of us is really hard. This is just an incomplete chart of the US federal agencies. They're all organizations full of people doing things in this 20th century way. And I think this is a bit of a risk for AI policy more broadly and for the stability of the world.

And I think we're in this Red Queen dynamic where we kind of have to run just to stay in place. A lot of my job is trying to get talent into government to work on these issues. But over the medium run, what's more important is to get procurement reform so we can get the smarter AIs into government to do this job faster. And the alternative is that we don't co-evolve with the technology and manage it, but end up basically being displaced.

And we kind of take this for granted with ride-hailing, because ride-hailing was a kind of regime change in micro. Not just of a company or technology, but regulatory commissions. Taxi commissions were a kind of 20th century public regulation that didn't evolve with the technology but instead got displaced.

So whether the 20th century nation state — hopefully not the way of Blockbuster. I think the goal that we should have is to try to advance this technology within government so that it can keep up and keep pace, or else suffer the fate of Blockbuster, which didn't really necessarily go under but rather was overwhelmed by everything growing up around it in the form of streaming services. Thank you.