Ben Buchanan
Appearances
The Ezra Klein Show
The Government Knows AGI is Coming
I think that's basically true. I think you need to have it. I firmly believe you need to have rights and protections that hopefully are pushing back and saying, no, there's key kinds of data here, including data on your own citizens. And in some cases, citizens of allied nations that you should not collect, even if there's an incentive to collect it.
The Ezra Klein Show
The Government Knows AGI is Coming
And for all of the flaws of the United States economy, intelligence oversight process and all the debates we could have about this, we do have those kinds of structures. And that, I think, is fundamentally more important for the reason you suggest in the era of tremendous AI systems.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, I would decompose essentially this question about AI and autocracy or the surveillance state, however you want to define it, into two parts. The first is the China piece of this. How does this play out in a state that is truly in its bones in autocracy and doesn't even make any pretense towards democracy and the like? And I think we can probably agree pretty quickly here. This
The Ezra Klein Show
The Government Knows AGI is Coming
makes very tangible something that, you know, is probably core to the aspirations of their society of like a level of control that only an AI system could help bring about that I just find terrifying.
The Ezra Klein Show
The Government Knows AGI is Coming
As an aside, I think there's a saying in both Russian and Chinese, something like heaven is high and the emperor is far away, which is like historically, even in those autocracies, there was some kind of space where the state couldn't intrude because of the scale and the breadth of the nation. And It is the case that in those autocracies, I think AI could make the force of government power worse.
The Ezra Klein Show
The Government Knows AGI is Coming
Then there's a more interesting question in the United States. Basically, what is the relationship between AI and democracy? And I think I share some of the discomfort here. There have been thinkers historically who have said, you know, part of the way is in which we revise our laws, our people break the laws, and there's a space for that.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think there is a humanness to our justice system that I wouldn't want to lose. and we tasked the Department of Justice in running a process and thinking about this and coming up with principles for the use of AI in criminal justice.
The Ezra Klein Show
The Government Knows AGI is Coming
I think there's, in some cases, advantages to it, like cases are treated alike with the machine, but also I think there's tremendous risk of bias and discrimination and so forth because the systems are flawed, and in some cases because the systems are ubiquitous. And I do think there is a risk of a fundamental risk
The Ezra Klein Show
The Government Knows AGI is Coming
encroachment on rights from the widespread unchecked use of AI in the law enforcement system that we should be very alert to and that I as a citizen have grave concerns about.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, there's a lot to unpack here, so let's just go in order. But basically, bottom line, I think I, in the White House and now post-White House, greatly share a lot of this discomfort.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think part of the appeal for something like the export controls is it identifies a choke point that can differentially slow the Chinese down, create space for the United States to have a lead, ideally, in my view, to spend that lead on safety and security.
The Ezra Klein Show
The Government Knows AGI is Coming
coordination and not rushing ahead, including, again, potential coordination with the Chinese, while not exacerbating this arms race dynamic. I would not say that we tried to race ahead in applications to national security.
The Ezra Klein Show
The Government Knows AGI is Coming
So part of the national security memorandum is a pretty lengthy kind of description of what we're not going to do with AI systems and a whole list of prohibited use cases and then high impact use cases. And there's a governance and you're not in power anymore.
The Ezra Klein Show
The Government Knows AGI is Coming
That's a fair question. Now, they haven't repealed this. The Trump administration has not repealed this. But I do think it's fair to say that for the period while we had power, the foundation we were trying to build with AI, we were very cognizant to the dynamic you were talking about, a race to the bottom on safety.
The Ezra Klein Show
The Government Knows AGI is Coming
And we were trying to guard against it, even as we tried to assure a position of U.S. preeminence.
The Ezra Klein Show
The Government Knows AGI is Coming
I don't know that it's pretty antagonistic to say we are not going to sell you the most advanced technology in the world. That does not in itself, that's not a declaration of war. That is not even itself a declaration of a Cold War. I think it is just saying this technology is incredibly important. Do you think that's how they understood it?
The Ezra Klein Show
The Government Knows AGI is Coming
This is more academic than you want, but my academic research when I started as a professor was based on the society's trap, or what in academia we call a security dilemma, of how nations misunderstand each other. So I'm sure the Chinese and the United States misunderstand each other at some level in this area. But I don't think they do.
The Ezra Klein Show
The Government Knows AGI is Coming
The plain reading of the facts is that not selling chips to them I don't think is a declaration of war.
The Ezra Klein Show
The Government Knows AGI is Coming
To some degree. I don't want to trigger this. I'm sure they do see it that way. On the other hand, we set up an AI dialogue with them. And, you know, I flew to Geneva and met them. And we tried to talk to them about AI safety and the like. So I do think in an area as complex as AI, you can have multiple things be true at the same time. I don't regret for a second the export controls.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think... Frankly, we are proud to have done them when we did them because it has helped ensure that here we are a couple of years later, we retain the edge in AI for as good as our talent as DeepSeek is.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, let's just take one step back so we're tracking the history of DeepSeq here. So we'd been watching DeepSeq in the White House since November of 23, or thereabouts, when they put out their first coding system. And there's no doubt that the DeepSeq engineers are extremely talented. And they got better and better at their systems throughout 2024.
The Ezra Klein Show
The Government Knows AGI is Coming
We were heartened when their CEO said, I think that the biggest impediment to what DeepSeek was doing was not their inability to get money or talent, but their inability to get advanced chips. Clearly, they still did get some chips that they some they bought legally, some they smuggled. So it seems.
The Ezra Klein Show
The Government Knows AGI is Coming
And then in December of 24, they came out with a system called version three DeepSeek version three, which actually I think is one that should have gotten the attention. It didn't get a ton of attention, but it did show they were making strong algorithmic progress and basically making systems more efficient. And then in January of 25, they came out with a system called R1.
The Ezra Klein Show
The Government Knows AGI is Coming
R1 is actually not that unusual. No one would expect that to take a lot of computing power just as a reasoning system that extends the underlying V3 system. That's a lot of nerd speak. The key thing here is when you look at what DeepSeek has done, I don't think the media hype around it was warranted. And I don't think it changes the fundamental analysis of what we were doing.
The Ezra Klein Show
The Government Knows AGI is Coming
They still are constrained by computing power. We should tighten the screws and continue to constrain them. They're smart. Their algorithms are getting better. But so are the algorithms of U.S. companies. And this, I think, should be a reminder that the chip controls are important. China is a worthy competitor here. And we shouldn't take anything for granted.
The Ezra Klein Show
The Government Knows AGI is Coming
But I don't think this is a time to say the sky is falling or the fundamental scaling laws have broken. Where do you think they got their performance increases from? They have smart people. There's no doubt about that. We read their papers. They're smart people who are doing exactly the same kind of algorithmic efficiency work that companies like Google and Anthropic and OpenAI are doing.
The Ezra Klein Show
The Government Knows AGI is Coming
I think two things are true here. The first is there will always be a frontier, or at least for the foreseeable future, there'll be a frontier that is computationally and energy intensive. And our companies, we want to be at that frontier. Those companies have very strong incentive to look for efficiencies and they all do.
The Ezra Klein Show
The Government Knows AGI is Coming
They all want to get every single last juice of insight from each squeeze of computation, but they will continue to need to push the frontier. And then in addition to that, there'll be a kind of slower diffusion that lags the frontier, where algorithms get more efficient, fewer computer chips are required, less energy is required. We need, as America, to win both those competitions.
The Ezra Klein Show
The Government Knows AGI is Coming
Every AI chip, every advanced AI chip that gets made will get sold. The market for these chips is extraordinary right now, I think, for the foreseeable future. So I think our view was we put the export controls on.
The Ezra Klein Show
The Government Knows AGI is Coming
We put the export controls on the first ones in October 2022. NVIDIA stock has 10x'd since then.
The Ezra Klein Show
The Government Knows AGI is Coming
Sure. I think the dynamic is right. I'm not suggesting they're, you know, if they had a bigger market, they could charge on the margins more. That's obviously the supply and demand here. I think our analysis was considering the importance of these chips and the AI systems they make to U.S. national security. This is a trade-off that's worth it.
The Ezra Klein Show
The Government Knows AGI is Coming
I met with him once. I don't know exactly, but I met with him once. Would that characterize the conversation he had with you? He talked about concerns related to startups and competitiveness and the like. I think my view on this is you look at our record on competitiveness, it's pretty clear that we want a dynamic ecosystem.
The Ezra Klein Show
The Government Knows AGI is Coming
So the AI executive order, which President Trump just repealed, had a pretty lengthy section on competitiveness. The Office of Management and Budget Management Memo, which governs how the U.S. government buys AI, had a whole carve-out in it or a call-out in it saying we want to buy from a wide variety of vendors. The Chips and Science Act has a bunch of things in there about competition.
The Ezra Klein Show
The Government Knows AGI is Coming
So I think our view on competition is pretty clear. Now, I do think there are structural dynamics related to scaling laws and the like that will force things towards big companies that I think in any respects we were pushing against. I think the track record is pretty clear of us in competition.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, so in the president's executive order, we actually tried to wrestle with this question, and we didn't have an answer when that order was signed in October of 23. And what we did on the open source question in particular, and I think we should just be precise here at the risk of being academic again, what we're talking about are open weight systems.
The Ezra Klein Show
The Government Knows AGI is Coming
Can you just say what weights are in this context and then what open weights are? Yeah, so when you have the training process for an AI system, you run this algorithm through this huge amount of computational power that processes the data.
The Ezra Klein Show
The Government Knows AGI is Coming
the output at the end of that training process, loosely speaking, and I stress this as the loosest possible analogy, they are roughly akin to the strength of connections between the neurons in your brain. And in some sense, you could think of this as the raw AI system.
The Ezra Klein Show
The Government Knows AGI is Coming
And when you have these weights, one thing that some companies like Meta and DeepSeek choose to do is they publish them out on the internet, which makes them, we call them open weight systems. And the crucial thing about an open weight system on the good side is that It's much easier to innovate with that system, to use it as a basis for future systems because you've got access to the raw thing.
The Ezra Klein Show
The Government Knows AGI is Coming
On maybe the riskier side is any safeguards that were built into that system. Refuse when a user asks you to help develop a biological weapon are pretty easy to remove. I'm a huge believer in the open source ecosystem. Many of the companies that publish the ways for their system do not make them open source. They don't publish the code and the like.
The Ezra Klein Show
The Government Knows AGI is Coming
So I don't think they should get the credit of being called open source systems at the risk of failure. Being pedantic. But open weight systems is something we thought a lot about in 23 and 24. And we sent out a pretty wide ranging request for comment for a lot of folks. We got a lot of comments back.
The Ezra Klein Show
The Government Knows AGI is Coming
And what we came to in the report that was published in July or so of 24 was basically there was not evidence yet to constrain the open weight ecosystem, that the open weight ecosystem does a lot for innovation and the like, which I think is manifestly true, but that we should continue to monitor this as the technology gets better, basically exactly the way that you described.
The Ezra Klein Show
The Government Knows AGI is Coming
I think I mentioned before this core intellectual insight of this technology for the first time maybe in a long time is a revolutionary one not funded by the government and its early incubator days. That was the theme from the labs, which is sort of like, don't you know we're inventing something very, very powerful?
The Ezra Klein Show
The Government Knows AGI is Coming
Ultimately, it's going to have implications for the kind of work you do in national security, the way we organize our society. And more than any kind of individual policy request, They were basically saying, get ready for this.
The Ezra Klein Show
The Government Knows AGI is Coming
The one thing that we did that could be the closest thing we did to any kind of regulation is one action, which was after the labs made voluntary commitments to do safety testing. We said, you have to share the safety test results with us and you have to help us understand where the technology is going. And that only applied really to the top couple labs. The labs never knew that was coming.
The Ezra Klein Show
The Government Knows AGI is Coming
Weren't all thrilled about it when it came out. So the notion of this was kind of a regulatory capture that we were asked to do this is simply not true. But I, in my experience, never got discrete individual policy lobbying from the labs. I got much more, this is coming, it's coming much sooner than you think. Make sure you're ready.
The Ezra Klein Show
The Government Knows AGI is Coming
To the degree that they were asking for something in particular, it was maybe a corollary of that, of we're going to need a lot of energy. And we want to do that here in the United States. And it's really hard to get the power here in the United States.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah. And this is actually, I think, an area of bipartisan agreement, which we can get to. But this is something that we really started to pay a lot of attention to in later part of 23 and most of 24 when it was clear this was going to be a bottleneck.
The Ezra Klein Show
The Government Knows AGI is Coming
And in the last week or so in office, President Biden signed an AI infrastructure executive order, which has not been repealed, which basically tries to accelerate the power development and the permitting of power and data centers here in the United States, basically for the reason that you mentioned.
The Ezra Klein Show
The Government Knows AGI is Coming
Now, as someone who truly believes in climate change and environmentalism and clean power, I thought there was a double benefit to this, which is that if we did it here in the United States, it could catalyze the clean energy transition and the like. And these companies, for a variety of reasons, in general, are willing to pay more for clean energy.
The Ezra Klein Show
The Government Knows AGI is Coming
And on things like geothermal and the like, our hope was we could catalyze that development and bend the cost curve and have these companies be the early adopters of that technology so we'd see a win on the climate side as well.
The Ezra Klein Show
The Government Knows AGI is Coming
So I think he is setting up a dichotomy there that I don't quite agree with. And the irony of that is, if you look at the rest of his speech, which I did watch, there's actually a lot that I do agree with. So he talks, for example, I think he's got four pillars in the speech. One's about centering the importance of workers.
The Ezra Klein Show
The Government Knows AGI is Coming
I think we are going to see extraordinarily capable AI systems. I don't love the term artificial general intelligence, but I think that will fit in the next couple of years, quite likely during Donald Trump's presidency. And I think there's a view that this has always been something of corporate hype or speculation.
The Ezra Klein Show
The Government Knows AGI is Coming
One's about American preeminence and like, those are entirely consistent with the actions that we took and the philosophy that I think the administration, which I was a part espoused, and I certainly believe. Insofar as what he is saying is that safety and opportunity are in fundamental tension, then I disagree.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think if you look at the history of technology and technology adaptation, the evidence is pretty clear that the right amount of safety action unleashes opportunity and, in fact, unleashes speed. So one of the examples that we studied a lot and talked to the president about was the early days of railroads.
The Ezra Klein Show
The Government Knows AGI is Coming
And in the early days of railroads, there were tons of accidents and crashes and deaths. And people were not inclined to use railroads as a result. And then what started happening was safety standards and safety technology. Block signaling, so that trains could know when they were in the same area. Air brakes, so that trains could brake more efficiently.
The Ezra Klein Show
The Government Knows AGI is Coming
standardization of train track widths and gauges and the like. And this was not always popular at the time. But with the benefit of hindsight, it is very clear that that kind of technology and to some degree policy development of safety standards made the American railroad system in the late 1800s. And I think this is a pattern that shows up a bunch throughout the history of technology.
The Ezra Klein Show
The Government Knows AGI is Coming
To be very clear, it is not the case that every safety regulation, every technology is good. And there certainly are cases where you can overreach and you can slow things down and choke things off. But I don't think it's true. There's a fundamental tension between safety and opportunity.
The Ezra Klein Show
The Government Knows AGI is Coming
The question then is, when you look at the actions we have taken on AI, are we strangling in the crib and have we taken actions that are akin to... I'm not saying that we've already done it.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, I think that's a profound question. I think what we tried to do in the Biden administration was set up the kind of institutions in the government to do that in as clear-eyed, tech-savvy a way as possible. Again, with the one exception of the safety test results sharing, which some of the CEOs estimate cost them one day of employee work.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think one of the things I saw in the White House when I was decidedly not in a corporate position was trend lines that looked very clear. And what we tried to do under the president's leadership was get the U.S. government and our society ready for these systems.
The Ezra Klein Show
The Government Knows AGI is Coming
We did not put anything close to regulation in place whatsoever. We create something called the AI Safety Institute, purely national security focused, cyber risks, bio risks, AI accident risks, purely voluntary. So, and that has relationships, memorandum of understanding with Anthropic, with OpenAI, even with XAI, Elon's company.
The Ezra Klein Show
The Government Knows AGI is Coming
And basically, I think we saw that as an opportunity to bring AI expertise into the government to build relationships between public and private sector in a voluntary way. And then as the technology develops, it will be up to now the Trump administration decide what they want to do with it.
The Ezra Klein Show
The Government Knows AGI is Coming
I think it's fair to say that there is a cultural difference with the Trump administration and us on some of these things. But I also, you know, we held conferences on what you could do with AI and the benefits of AI. We talked all the time about how you need to mitigate these risks, but you're doing so so you can capture the benefits.
The Ezra Klein Show
The Government Knows AGI is Coming
And I'm someone who reads an essay like Dario Amadei's Sea of Anthropics, Machines of Love and Grace, about basically the upside of AI. And so there's a lot in here we can agree with. President's executive order said we should be using AI more in the executive branch. So I hear you on the cultural difference. I get that.
The Ezra Klein Show
The Government Knows AGI is Coming
But I think when the rubber meets the road, we were comfortable with the notion that you could both realize the opportunity of AI while doing it safely. And now that they are in power, they will have to decide how do they translate Vice President Vance's rhetoric into a governing policy.
The Ezra Klein Show
The Government Knows AGI is Coming
And my understanding of their executive order is they've given themselves six months to figure out what they're going to do. And I think we should judge them on what they do.
The Ezra Klein Show
The Government Knows AGI is Coming
The sort of canonical definition of AGI, which again is a term I don't love, is a system... It'll be good if every time you say AGI, you caveat that you dislike the term. It'll sink in, right?
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, I think I basically agree with all of that. I think the conversation when we were in the government, especially in 23 and 24, was starting to happen. We looked at the clinical trials thing. You've read about healthcare for however long. I don't claim expertise on healthcare, but it does seem to me that...
The Ezra Klein Show
The Government Knows AGI is Coming
We want to get to a world where we can take the breakthroughs, including breakthroughs from AI systems, and translate them to market much faster. This is not a hypothetical thing.
The Ezra Klein Show
The Government Knows AGI is Coming
It's worth noting, I think, quite recently, Google came out with, I think they called it co-scientist, NVIDIA and the ARC Institute, which does great work, had the most impressive biodesign model ever that has a much more detailed understanding of biological molecules. A group called Future House has done similarly great work in science. So I don't think this is a hypothetical.
The Ezra Klein Show
The Government Knows AGI is Coming
I think this is happening right now. And I agree with you that there's a lot that can be done institutionally and organizationally to get the federal government ready for this.
The Ezra Klein Show
The Government Knows AGI is Coming
I'm trying to get it in the training data. A canonical definition of AGI is a system capable of doing almost any cognitive task a human can do.
The Ezra Klein Show
The Government Knows AGI is Coming
It feels kind of orthogonal from what I've observed from Doge. I mean, I think Elon is someone who does understand what AI can do, but I don't know how starting with USAID, for example, prepares the U.S. government to make better AI policy. So I guess I don't buy it, that that is the motivation for Doge.
The Ezra Klein Show
The Government Knows AGI is Coming
I couldn't agree more. I mean, the existence of my job in the White House, the White House Special Advisor for AI, which David Sachs now is, and I had this job in 2023, existed because President Biden said very clearly, publicly and privately, we cannot move at the typical government pace. We have to move faster here. I think we probably need to be careful.
The Ezra Klein Show
The Government Knows AGI is Coming
I don't know that we'll quite see that in the next four years or so, but I do think we'll see something like that, where the breadth of the system is remarkable, but also its depth, its capacity to, in some cases, exceed human capabilities, kind of regardless of the cognitive discipline. Systems that can replace human beings in cognitively demanding jobs.
The Ezra Klein Show
The Government Knows AGI is Coming
And I'm not here for stripping it all down, but I agree with you. We have to move much faster.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, I think the honest answer here is we had conversations with Europe as they were drafting the EU AI Act. But at the time that I was in, the EU AI Act was still kind of nascent and the act had passed. But a lot of the actual details of it had been kicked to a process that my sense is still unfolding. So speaking of slow moving. Yeah, I mean, bureaucracies. Exactly. Exactly.
The Ezra Klein Show
The Government Knows AGI is Coming
So I guess I didn't have from maybe this is a failing on my part. I did not have particularly detailed conversations with the Europeans beyond a general kind of articulation of our views. They were respectful. We were respectful. But I think it's fair to say we were taking a different approach than they were taking.
The Ezra Klein Show
The Government Knows AGI is Coming
And we were probably, insofar as safety and opportunity are a dichotomy, which I don't think they are a pure dichotomy, we were ready to move very fast in the development of AI.
The Ezra Klein Show
The Government Knows AGI is Coming
It's a vital question. I think we instantiate that in a couple of different principles. The first is that AI in the workplace needs to be implemented in a way that is respectful of workers and the like. And I think one of the things I know the president thought a lot about was
The Ezra Klein Show
The Government Knows AGI is Coming
it is possible for AI to make workplaces worse and in a way that is dehumanizing and degrading and ultimately destructive for workers. So that is sort of a first distinct piece of it that I don't want to neglect. The second is I think we want to have AI deployed across our economy in a way that increases workers' agencies and capabilities.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think we should be honest that there's going to be a lot of transition in the economy as a result of AI. I don't know what that will look like. You can find Nobel Prize winning economists who will say it won't be much. You can find other folks who will say it'll be a ton. I tend to lean towards the it's going to be a lot side, but I'm not a labor economist.
The Ezra Klein Show
The Government Knows AGI is Coming
And the line that Vice President Vance used is the exact same phrase that President Biden used, which is give workers a seat at the table in that transition. And I think that is a fundamental part of what we're trying to do here. And I presume what they're trying to do here.
The Ezra Klein Show
The Government Knows AGI is Coming
I think that is basically true, but I want to push back a little bit. So I do think we are going to see a dynamic in which it will hit parts of the economy first, it will hit certain firms first, but it will be an uneven distribution across society.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, we did talk to economists and try to texture this debate in 23 and 24. I think... The trend line is even clearer now than it was then. I think we knew this was not going to be a 23 and 24 question. Frankly, to do anything robust about this is going to require Congress, and that was just not in the cards at all. So it was more of an intellectual exercise than it was a policy exercise.
The Ezra Klein Show
The Government Knows AGI is Coming
Sure, yeah. I think that's fair. I think the advantage to AI that... is in some ways a countervailing force here, though I hear you and I mostly agree with your side's argument, is that it will increase the amount of agency for individual people.
The Ezra Klein Show
The Government Knows AGI is Coming
So I do think we will be in a world in which the 19-year-old or the 25-year-old will be able to use a system to do things they were not able to do before. And I think insofar as The thesis we're batting around here is that intelligence will become a little bit more commoditized. What will stand out more in that world is agency and the capacity to do things or initiative and the like.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think that could, in the aggregate, lead to a pretty dynamic economy. And the economy you're talking about of small firms and dynamic ecosystem and robust competition, I think on balance, at an economy scale, is not in itself a bad thing.
The Ezra Klein Show
The Government Knows AGI is Coming
I think where I imagine you and I agree and maybe Vice President Vance as well agree is we need to make sure that for individual workers and classes of workers, they're protected in that transition. I think we should be honest. That's going to be very hard. We have never done that well.
The Ezra Klein Show
The Government Knows AGI is Coming
So I think, yes, we were thinking about this question. I think we knew it was not going to be a question we were going to confront in the president's term. We knew it was a question that you would need Congress for to do anything about it. I think insofar as what you're expressing here seems to me to be like a deep dissatisfaction with the available answers. I share that.
The Ezra Klein Show
The Government Knows AGI is Coming
I think a lot of us shared that. You can get the usual stock answers of a lot of retraining. I share your sort of doubts that that is the answer. You probably talk to some Silicon Valley libertarians or something and they'll say, or tech folks, and they'll say, well, universal basic income.
The Ezra Klein Show
The Government Knows AGI is Coming
I think I believe and I think the president believes there's a kind of dignity that work brings and doesn't have to be paid work, but there needs to be something that people do each day that gives them meaning overall. In so far as what you were saying is like you have a discomfort with where this is going on the labor side. Speaking for myself, I share that. I guess I don't know the shape of it.
The Ezra Klein Show
The Government Knows AGI is Coming
Well, you know, we definitely didn't have Claude write up a brief because we had to get over government use of AI. See, but that's like itself slightly damning. Yeah. Yeah, I mean, Ezra, I agree that the government has to be more forward-leaning on basically all of these dimensions. It was my job to push the government to do that.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think on things like government use of AI, we made some progress. So I don't think anyone from the Biden administration, less of all me, is coming out and saying, we solved it.
The Ezra Klein Show
The Government Knows AGI is Coming
I think what we're saying is like we were building a foundation for something that is coming that was not going to arrive during our time in office and that the next team is going to have to, as a matter of American national security, and in this case, American security, economic strength and prosperity address.
The Ezra Klein Show
The Government Knows AGI is Coming
I think there should be an intellectual humility about before you take a policy action, you have to have some understanding of what it is you're doing and why. So I think it is entirely intellectually consistent to look at a transformative technology, draw the lines on the graph and say, this is coming pretty soon without having the 14 point plan of this is what we need to do in 2027, 2028.
The Ezra Klein Show
The Government Knows AGI is Coming
I think chip controls are unique in that this is a robustly good thing that we could do early to buy the space I talked about before. But I also think that we tried to build institutions like the AI Safety Institute that would set the new team up, whether it was us or someone else, for success in managing the technology.
The Ezra Klein Show
The Government Knows AGI is Coming
Now that it's them, they will have to decide, as the technology comes on board, how do we want to calibrate this under regulation? What are the kinds of decisions you think they will have to make in the next two years? You mentioned the open source one. I have a guess where they're going to land on that, but I think there's an intellectual debate there that is rich.
The Ezra Klein Show
The Government Knows AGI is Coming
We resolved it one way by not doing anything. They'll have to decide if they want to keep doing that. Ultimately, they'll have to answer a question of what is the relationship between the public sector and the private sector. Is it the case, for example, that the kind of things that are voluntary now with the AI Safety Institute will someday become mandatory?
The Ezra Klein Show
The Government Knows AGI is Coming
Another key decision is we tried to get the ball rolling on the use of AI for national defense in a way that is consistent with American values. They will have to decide what does that continue to look like? And do they want to take some of the safeguards that we put in place away to go faster?
The Ezra Klein Show
The Government Knows AGI is Coming
So I think there really is a bunch of decisions that they are teed up to make over the next couple of years that we can appreciate they're coming on the horizon without me sitting here and saying, I know with certainty what the answer is going to be in 2027. And then always our final question, what are three books you'd recommend to the audience?
The Ezra Klein Show
The Government Knows AGI is Coming
One of the books is The Structure of Scientific Revolution by Thomas Kuhn. And this is a book that coined the term paradigm shift, which basically is what we've been talking about throughout this whole conversation of a shift in technology and scientific understanding and its implications for society.
The Ezra Klein Show
The Government Knows AGI is Coming
And I like how Kuhn in this book, which was written in the 1960s, gives a series of historical examples and theoretical frameworks for how do you think about a paradigm shift. And then another book that has been very valuable for me is Rise of the Machines by Thomas Ridd.
The Ezra Klein Show
The Government Knows AGI is Coming
And that really tells the story of how machines that were once the playthings of dorks like me became in the 60s and the 70s and the 80s, things of national security importance. We talked about some of the revolutionary technologies here, the internet, microprocessors and the like, that emerged out of this intersection between national security and tech development.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think that history should inform the work we do today. And then the last book is definitely an unusual one, but I think is vital. And that's A Swim in the Pond in the Rain by George Saunders. And he's this great essayist and short story writer and novel writer. And he teaches Russian literature.
The Ezra Klein Show
The Government Knows AGI is Coming
And he, in this book, takes seven Russian literature short stories and gives a literary interpretation of them. And what strikes me about this book is fundamentally the most human endeavor I can think of. He's taking great human short stories and he's giving a modern interpretation of what those stories mean.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think when we talk about the kinds of cognitive tasks that are a long way off for machines, I kind of at some level hope this is one of them, that there is something fundamentally human that we alone can do. I'm not sure if that's true, but I hope it's true.
The Ezra Klein Show
The Government Knows AGI is Coming
I think that's right. I'm not a labor market economist, but I think that the systems are extraordinarily capable. In some ways, I'm very fond of the quote, the future is already here. It's just unevenly distributed. And I think unless you are engaging with this technology, you probably don't appreciate how good it is today.
The Ezra Klein Show
The Government Knows AGI is Coming
And then it's important to recognize today is the worst it's ever going to be. It's only going to get better. And I think that is the dynamic that in the White House world, We were tracking and that the next White House and our country as a whole is going to have to track and adapt to in really short order.
The Ezra Klein Show
The Government Knows AGI is Coming
And what's fascinating to me is that this is the first revolutionary technology that is not funded by the Department of Defense, basically. And if you go back historically, last 100 years or so, nukes, space, early days of the internet, early days of the microprocessor, early days of large-scale aviation, radar, GPS, the list is very, very long.
The Ezra Klein Show
The Government Knows AGI is Coming
All of that tech is fundamentally comes from DoD money. And it's the private sector inventing it, to be sure. But the central government role is... gave the Department of Defense and the U.S. government an understanding of the technology that by default, it does not have an AI, and also gave the U.S. government a capacity to shape where that technology goes, that by default, we don't have an AI.
The Ezra Klein Show
The Government Knows AGI is Coming
I do think there are profound economic and military and intelligence capabilities that would be downstream of getting to AGI or transformative AI. And I do think it is fundamental for U.S. national security that we continue to lead AI. I think the quote that certainly I thought about a fair amount was actually from Kennedy in his famous Rice speech in 62, the we're going to the moon speech.
The Ezra Klein Show
The Government Knows AGI is Coming
Aaron remembers it because he's saying we're going to the moon, but actually I think he gives the better line when he talks about the importance of space.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think that is true in AI, that there's a lot of tremendous uncertainty about this technology. I am not an AI evangelist. I think there's huge risks to this technology. But I do think there is a fundamental role for the United States in being able to shape where it goes. Which is not to say we don't want to work internationally, which is not to say we don't want to work with the Chinese.
The Ezra Klein Show
The Government Knows AGI is Coming
It's worth noting that in the president's executive order on AI, there's a line in there saying we are willing to work even with our competitors on AI safety and the like. But it is worth saying that I think pretty deeply there is a fundamental role for America here that we cannot abdicate.
The Ezra Klein Show
The Government Knows AGI is Coming
So I think let's look at just a narrow case of AI for intelligence analysis and cyber operations. This is, I think, pretty out in the open that if you had a much more powerful AI capability, that would probably enable you to do better cyber operations on offense and on defense. What is a cyber operation?
The Ezra Klein Show
The Government Knows AGI is Coming
Breaking into an adversary's network to collect information, which if you're collecting a large enough volume, AI systems can help you analyze. And we actually did a whole big thing through DARPA, the Defense Advanced Research Project Agency, called the AI Cyber Challenge, to test out AI's capabilities to do this.
The Ezra Klein Show
The Government Knows AGI is Coming
And I would not want to live in a world in which China has that capability on offense and defense and cyber, and the United States does not. And I think that is true in a bunch of different domains that are core to national security competition.
The Ezra Klein Show
The Government Knows AGI is Coming
What we mean canonically when we speak of hacking is finding vulnerability in software, exploiting that vulnerability to get illicit access. And I think it is right that more powerful AI systems will make it easier to find vulnerabilities and exploit them and gain access. And that will yield an advantage to the offensive side of the ball.
The Ezra Klein Show
The Government Knows AGI is Coming
I think it is also the case that more powerful AI systems on the defensive side will make it easier to write more secure code in the first place, reduce number of vulnerabilities that can be found, and to better detect the hackers that are coming in. And I think we tried as much as possible to shift the balance towards the defensive side of this.
The Ezra Klein Show
The Government Knows AGI is Coming
But I think it is right that in the coming years here, this sort of transition period we've been talking about, that there will be a period in which sort of older legacy systems that don't have the advantage of the newest AI defensive techniques or software development techniques will on balance be more vulnerable to a more capable offensive actor. Which is what most people use.
The Ezra Klein Show
The Government Knows AGI is Coming
I don't know if that's right, actually. I mean, you have an iPhone in your pocket that, or Google Pixel. People are often not that quick about updating. Yeah. I mean, the less tech literate you are, the more vulnerable you can be to this.
The Ezra Klein Show
The Government Knows AGI is Coming
legacy power systems and server mainframes and the like that could be two decades old that haven't been turned off. So that I think is where I feel the risk most acutely. I think for all of the risks that come with the monoculture of most people's personal tech platforms these days, one of the upsides is they do push security updates pretty regularly.
The Ezra Klein Show
The Government Knows AGI is Coming
They push them with new emojis that get people to download the updates. And on balance, I think people are probably better at patching their personal software now than they were 15 years ago.
The Ezra Klein Show
The Government Knows AGI is Coming
I've worked in this gift the last four years, a secure room where you can't bring your phone and all that. That is annoying. There's no doubt about it.
The Ezra Klein Show
The Government Knows AGI is Coming
Of AI labs. Yeah. I worry about it. I think it's a hacking risk here. I also, you know, if you hang out in the right San Francisco house party, they're not sharing the model, but they are talking to some degree about the techniques they use and the like, which have tremendous value. I do think it is the case to come back to this kind of intellectual through line of this is...
The Ezra Klein Show
The Government Knows AGI is Coming
national security-relevant technology, maybe world-changing technology that's not coming from the auspices of the government and doesn't have the kind of government imprimatur of security requirements. That shows up in this way as well. We, in the National Security Memorandum, the president's side, tried to signal this to the labs and tried to say to them, we are, as the U.S.
The Ezra Klein Show
The Government Knows AGI is Coming
government, want to help you in this mission. This was signed in October of 2024, so there wasn't a ton of time for us to build on that. But I think it's a priority for the Trump administration. And I can't imagine anything that is more nonpartisan than protecting American companies that are inventing the future.
The Ezra Klein Show
The Government Knows AGI is Coming
Yeah, I think we were pretty public about this. And the president signed a national security memorandum, which is basically the national security equivalent of an executive order that says this is a fundamental area of importance for the United States. I don't even know the amount of satellite images that the United States collects every single day, but it's a huge amount.
The Ezra Klein Show
The Government Knows AGI is Coming
And we have been public about the fact that we simply do not have enough humans to go through all of this satellite imagery, and it would be a terrible job if we did. And there is a role for AI in going through these images of hotspots around the world, of shipping lines and all that, and analyzing them in an automated way and surfacing the most interesting and important ones for human review.
The Ezra Klein Show
The Government Knows AGI is Coming
And I think at one level, you can look at this and say, well, doesn't software just do that? And I think that at some level, of course, is true. At another level, you could say the more capable that software, the more capable the automation of that analysis, the more intelligent advantage you extract from that data. And that ultimately leads to a better position for the United States.