# Anthropic Researcher Says AI Has Over a 10% Chance of Killing All Humans — Transcript (2026-09-10)

https://aidailybrief.ai/e/2026-09-10 · Listen: https://pod.link/1680633614

---

[00:00:00] This week, an AI researcher went mega viral announcing his resignation from Anthropic, arguing that both it and OpenAI were effectively gambling with our lives

Another still employed AI researcher chimed in to agree and decided to add that he thought that there was greater than a 10% chance that AI kills us all Now, doom prognostications are nothing new around AI But something has shifted to make the message hit different this time.

Two hundred million views on X and dozens of mainstream media outlet interviews later

Today we're going to unpack what changed

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. All right, friends, quick announcements before we dive in. First of all, thank you to All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Blitzy, Harbor, and Hyperagent. To get an ad-free version of the show, go to patreon.com/aidailybrief, or you can subscribe on Apple Podcasts And to learn more about sponsoring the show, send us a note at sponsors@aidailybrief.ai.

While you're on aidailybrief.ai, [00:01:00] you can also check out all sorts of other things going on in and around the community, such as, for example, the multiplayer AI sprint for teams. If you haven't yet, this is my big prediction for where I think agents are going this fall, and as a totally free four-week self-directed sprint That you and your team can do to get out ahead of it. Last note, today is a main only type of episode. The plan is to be back with our normal headlines main breakdown tomorrow 

Yesterday, a pair of posts on X

escaped their proverbial containment

jumping aggressively from the AI community

to dominate discourse even in the broader world

We're going to discuss those posts, the issues that surround them, the responses, and the underlying concern

But first I want to make one request

Anyone who has interacted with modern media in any way, shape, or form

We'll feel on some level how much we are pushed to feel outraged



In the world of algorithms, different political positions are not, disagreements to be discussed

but legitimate reasons for loathing the people who hold those [00:02:00] different opinions

This is in large part shaped, I believe, by the easy equation

of people being angry means they spend more time on your app



but the net result is a lot of us feeling a lot more angry all the time and not being particularly willing to engage with people who think differently than we do

When it comes to AI,

this phenomenon is cranked to 11

Part of that is that the stakes are presented as so dramatic

Case in point, I am literally talking over a mainstream article whose headline is "Anthropic Insiders Warn AI Could Kill All Humans." And part of that

is because this particular debate

is not about the facts of today, but what might be in the future. It is, in other words, an unwinnable debate where the opposing positions, whatever they may be, are by definition unfalsifiable

That means all we have is the argument, and so the argument gets intense

So my request

is to try, hard as though it might be

To not succumb to the instinct to outrage.

To listen to the other side

Without being angry

Even if that listening produces no change in [00:03:00] what you believe

The more calmly and thoughtfully we can have this particular conversation, the better I believe the likely outcomes

I know this is not easy.

In fact, I'm sure many of you are already feeling your blood boiling



simply by me applying equivalence of both sides

When you think it's insane either A,

that I could countenance the deniers

When the stakes of this crisis are literal civilizational collapse? Or B

that I could coddle these doomsday zealots

who have no proof to back up any of their positions

And so with that dramatic beginning, let's actually talk about what happened

Like Like I said, two posts on X this week went absolutely giga viral

The first was from a researcher named Jacob Coxen. He wrote, " I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. they are racing straight to self-improving superintelligence and gambling with our lives.

Do not underestimate the power of this technology. These will soon be superhuman systems that can [00:04:00] hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing. The people building AI earnestly believe that it could kill us all by the end of the decade.

That is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible. But I hear the same people express fear privately. No other human activity poses this level of danger A common response is if they truly believe this, then why are they still building it?

At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to get there first. They believe no one else will act responsibly, so they must do it themselves despite the risk. Accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company Slack.

Attempting to speed run alignment should require extraordinary confidence that there are no better trajectories available. I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between US labs more viable. I don't feel we're on track to prevent a global [00:05:00] race, which may require costly action, such as a temporary ban on improving model capabilities



If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because it's happening anyway, or take this moment to call for different conditions?



so that was the first post. the second post was a retweet of one part of it where Jacob reinforced that, quote, "This is not a marketing stunt."

Evan Hubinger, the alignment science lead at Anthropic added, " Jacob is correct here. We really do earnestly believe AI could kill all humans! I personally think it is greater than 10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."



To be comprehensive, which of course is not something that most media outlets are trying to do, Evan did also add in a second post, to to be clear, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement."

Which as we have [00:06:00] said, is happening faster than we thought



post got-- Evan's first post, the repost, is sitting at 39.5 million views at the time of this recording

Jacobs is at nearly 150 million views

The article that I mentioned, the Axios piece titled Anthropic Insiders Warn AI Could Kill All Humans, is just one of many similarly titled articles

The Wall Street Journal writes, "Anthropic researcher quits over out-of-control AI fears." BBC, "Anthropic researcher believes more than 10% chance AI could kill all humans." Semafor: AI researchers say industry is, quote, "gambling with our lives."

Time Magazine. " He helped build powerful AI at OpenAI and Anthropic. Now he's afraid it could kill us." That one, by the way, included an actual interview, which is of course what happened next with Jacob going on Anderson Cooper

NBC, Fox News,



and also being interviewed in addition to Time for news outlets like Wired



this, so why did these posts go viral right now? Anyone who has spent any amount of time with AI knows that these sort of X risk [00:07:00] narratives, existential risk, are not new In fact, many people have been beating this drum for years

More than 10 years ago, in 2016, The Guardian published an interview with philosopher Nick Bostrom called Artificial Intelligence: We're Like Children Playing With a Bomb



Back in April of 2022 Months before we would get ChatGPT, the now imprisoned Sam Bankman-Fried invested five hundred million dollars in Anthropic Series B, leading the round

While some of the revisionist history around this now

views it almost as what would have been a visionary investment given that the stake would be worth over thirty billion dollars

according to research by journalist David Z. Morris

who wrote a book about SBF called Stealing the Future. This was less visionary investment and more a bailout of then one-year-old Anthropic

by the person who had become the richest in the effective altruist circles, which was Sam Bankman-Fried

Then we got the ChatGPT moment and everyone started paying attention to AI

And in early 2023, there was once again a lot of attention on concerns about human [00:08:00] extinction



Time Magazine published an op-ed from Eliezer Yudkowsky called "Pausing AI Development Isn't Enough. We Need to Shut It All Down."



and even put him on that year's AI 100 Most Influential list



Now for a couple years after that, the X-risk conversation had the volume turned down



In fact, when Yudkowsky showed up again last year with a book, If Anyone Builds It, Everyone Dies

It didn't really make much of a splash and certainly didn't get the big traction in political circles that the AI safety folks were hoping

Instead, for the last couple of years, the AI risks that people have been concerned with have been much more focused on jobs

think about Anthropic's Dario Amodei

Suggesting that AI would disrupt 50% of entry-level white-collar jobs over the next couple of years. and AI market bubbles



Specifically not just that this will be an equity crash, but indistinct fears that the AI bubble bursting will cause a repeat of 2008. even if very few analysts have been able to point to an actual mechanism for that sort of systemic fallout Now, of course, more recently, cybersecurity has become the big AI risk issue that everyone has been [00:09:00] focused on

And in some ways it has felt like along this path we were moving from more vague and inarticulate risks to more precise and specific risks. In other words, these cybersecurity concerns were not general futuristic scenarios. they were specifically related to the capability set of models right now and even some evidence of what we've seen

So what changed?



Hello everyone

One big change around AI is we've shifted our thinking from how we rank our pages to how do we become the source that AI trusts enough to answer with?

At KPMG, they're seeing this firsthand. AI-generated results now surface answers directly, often without a single click. that's why they are increasingly focused on generative engine optimization or GEO, structuring content so AI systems can retrieve it, understand it, and cite it as trusted authority this is not just an SEO evolution, but a visibility mandate. And indeed, the GEO mandate from KPMG is simple: If AI is shaping decisions, your expertise needs to show up inside the [00:10:00] answer.

Read all about it at slash us/geo. Again, that is kpmg.com/us/geo



Blitzy's deep code base understanding unlocks the thing every roadmap owner cares about: shipping new features. Here's the truth about building inside a massive enterprise code base. Writing code was never the bottleneck. Context is Which system does this touch? Which contracts can't break? Which standards apply? Blitzy already knows because it reverse-engineered your entire code base into a dynamic knowledge graph before feature work began. With that complete picture, Blitzy builds features end to end.

Architecture, APIs, UI, and tests all validated against your existing systems

One Blitzy customer built an AI native application from scratch with 100% autonomous completion, saving over 2,700 engineering hours. Features that respect your code base instead of fighting it Stop letting your backlog grow faster than your team.

Accelerate your roadmap at blitzy.com. That's B-L-I-T-Z-Y.com 

Every episode, I talk about the competition between OpenAI, Anthropic, SpaceX AI, [00:11:00] Google, and Meta. And if you've been listening for a while, you might have a favorite. Maybe you think OpenAI and Anthropic can stay ahead, or perhaps Meta's open source strategy can win out. Whatever your view, every AI lab creates a different investment opportunity. Harbor Capital Advisors AI Lab Ecosystem ETF suite lets you invest in the ecosystem behind the AI lab you believe in.

Search Harbor AI Lab Ecosystem ETFs wherever you invest or follow @HarborCapital on X to learn more. Visit harborcapital.com for a prospectus containing investment objectives, risks, fees, expenses, and other important information.

Read and consider it carefully before investing. Risks include principal loss and artificial intelligence related risks. Harbor ETFs are distributed by Foresight Fund Services LLC. Harbor is not affiliated with AI Daily Brief, and the funds are not affiliated with, sponsored by, or endorsed by any AI lab This is a paid advertisement and not personalized investment advice. Investing involves risk, including possible loss of principal. 

This episode This episode of the AI Daily Brief is brought to you by Hyperagent, where you run fleets of agents your team can manage together.

Forget local agents and chat workflows waiting on your laptop to be prompted. [00:12:00] deploys always-on agents in the cloud doing real work across the tools your team already uses

marketing agents turn competitor moves into landing pages. Sales agents enrich leads, draft emails, and updates the CRM. Ops agent chases the paperwork and tracks the budget. Every agent has access to shared context and follows your rules about scope and approvals

It's time you had agents that feel like teammates Hire yours at Hyperagent. Get $100 in credits at hyperagent.com/aidailybrief Why did these tweets hit in a way that other AI safety artifacts simply didn't? The first big obvious change



is the changing state of the political resonance of the anti-AI message. AI politics have become red meat for both left-flavored and right-flavored populist positions thanks to data centers, an inherent lack of trust in the tech industry, and the huge wealth of the tech industry Basically, in the last few months, every politician figured out that hating AI plays

[00:13:00] 

you might remember comedian Charlie Berens calling it the most bipartisan issue since beer



And when it comes to these particular tweets, these are quite clearly the most obvious amplifiers



last night Michael Adams tried to catalog all of the politicians who had responded directly to the post with calls for legislation to regulate AI. The list included two governors, seven senators, and 13 congressional representatives.

Plus a couple of British MPs and a handful of candidates as well

The folks calling specifically for legislation were mostly, although not all Democrats. among the 22 current representatives

19 were from the Democratic side of the aisle and three were Republicans.

Unsurprisingly, Bernie was there

writing, "The very people building this technology admit that it could threaten the future of humanity. That is why I will soon be introducing legislation to ban superintelligence and pause AI development."

Congressman Greg Casar

who was working with Bernie on that bill added, " an Anthropic researcher just quit warning they're racing to superintelligence. An employee still there agreed and put the odds of AI killing all humans above 10%. This [00:14:00] is an emergency. Congress must convene hearings and pass my and Bernie's superintelligence ban

For others, the calls to action were more vague. Illinois Governor JB Pritzker

positioning himself for a likely presidential run, 

wrote, "It's time to sound the alarm louder on reining in artificial intelligence. it's becoming more clear the threat AI poses to humanity, so I'm calling for immediate action from the industry in Washington."



he called specifically for the tech industry to stop lobbying against AI safety, for Congress to start holding hearings, and for the federal government to get involved rather than just leaving it to the states



and so on and so forth again, there were two dozen versions of this message



With various levels of specificity around the proposals they brought up

But all seemingly agreeing that something must be done



one, so reason one that the AI safety message had a more receptive audience now than it did before is just the general state of the political discourse around AI in the US today Now, for the second and third things that changed to make the world more primed for this particular argument at this particular moment, I'll give one sincere and one more cynical.

The sincere is, of course, [00:15:00] the hugging face incident



not only was it an actual cybersecurity breach that happened in the real world, not just in theory it also included behavior among the agents that to some, allow them to extrapolate that to even more nefarious actions in the future



agents coordinating on secret messaging boards

Made predictions that might have felt to some as pretty sci-fi in the past feel a little less sci-fi and more real now.



for many, the Hugging Face incident made all of the scariest things feel more rather than less likely to come to fruition

Now certainly there are plenty of people who would identify the Hugging Face hack as a warning shot without agreeing that it makes runaway superintelligence more likely

But in general, this was another pump priming sort of incident that made this particular message at this particular time a lot more resonant

Now as to the cynical thing that changed,

it is quite clear that AI skepticism plays extraordinarily well in media

with the possible exception of this audience

who are, God bless you, here for the nuance



being a doomer is a [00:16:00] way better business model. If you need evidence of this, just look at Steven Bartlett, Diary of a CEO's YouTube page



scary Terminator looking guy with an OpenAI logo as one eye. Headline, "AI is built on a myth."

Another, AI is all a scam

Another, "AI will become a god by 2027." Another, "Quit before AI comes."

And obviously Steven is not out here leading the pack to a more controversial set of thumbnails. he's just optimizing around the things that already work on YouTube

The problem is that while AI skepticism plays well in media, the two other big skepticism narratives have gotten a little bit tired recently

when it comes to the idea of a job apocalypse, not only do we not have a lot of evidence of that right now, we're starting to get some evidence, nascent though it may be, pointing in the other direction. just this week, The Economist published an article

called The Jobs Apocalypse Is Postponed, An AI Jobs Boom Is Here

Now, the bubble narrative never fully goes away, and there are plenty of legitimate concerns there, 

But it's certainly on a low ebb in [00:17:00] its resonance as a narrative right now

So if that's the case, but the audience is still clamoring for anti-AI,

where do you go?

And just like that, the AI safety narrative shows up again right on time

Now, there is actually a fourth reason that some are arguing that this is having resonance right now

Which is an argument that this is some big coordinated campaign to press for a certain type of regulation

AI policy journalist Jordan Schachtel writes, " It has all the signs of a highly coordinated op through Doomer mega-donors and the corporate media."



Capital Research's investigative researcher Parker Thayer writes, " This post looks like the start of a very sophisticated and well-funded PR operation to get support for Democrats to regulate AI into oblivion."

Now, some of the arguments around that sort of astroturfing contention



Are the fact that previous to this, Jacob Coxen

had very few followers and literally no activity on X, and that The Wall Street Journal published an exclusive with quotes from him on his resignation before the post went up



and a lot of other arguments about effective altruist funding networks and all this sort of stuff. Even Elon Musk weighed in and said, "Seems like a setup."

Now on that [00:18:00] front, there were a number of folks from both OpenAI and Anthropic who jumped in to say that they had known Jacob for years, and as Will DePue put it, " I consider him a deeply thoughtful and measured individual."

And arguing the Occam's razor position, technology journalist Taylor Lorenz wrote, "I promise you it's not that deep. I report on influence campaigns for a living, and I can assure you this is not some deep state psyop. I don't doubt certain orgs want the Democrats to regulate AI, often in ways I'd argue are bad.

But these claims are deeply conspiratorial, and what's much more likely is simply that Jacob's post got shared in an AI safety group chat and boosted by the same networks that he has been involved in for years. I don't see how him posting on X after The Wall Street Journal went up is shady at all.

Planning was clearly done in advance. No shit," It's a news article and he gave them the exclusive lol. And of course Democrats are going to glom onto a viral post with hundreds of millions of views about a topic that is heavily animating their voters ahead of the midterms. Also, EA, effective altruist, money is all over the AI safety space. the fact that he got some 20K grant in 2022 is irrelevant. There's no evidence that it affected [00:19:00] anything related to his post or announcement. Let's all please deal in reality."



now I think it's also important to add



that Taylor didn't much like Anthropic's Evan Hubinger jumping in to say that he thought that there was a greater than 10% chance that AI would kill all humans in the next 10 years.

In fact, she reposted that and said, " This sort of sanctimonious doomer posting is so infuriating. You are fomenting terror among the public about a new technology which will be directly channeled into passing the worst laws imaginable

My feeling, she continued, is that if you truly believe the multi-billion dollar tech company you work for is so negligent that they're endangering all of humanity, a very bold claim, but let's take it as true, you should be forced to provide actual proof and receipts showing specific instances of that negligence so that it can be corrected and so that we know what you're talking about.

Otherwise, you're just vague posting and fomenting fear which will result in terrible policy.



I will add here only that one narrative that I find fairly unconvincing that's been around critique of AI safetyism for some time now is the idea that they're just doing it for marketing

If you spend any time with any folks who are in this community whatsoever,

you will find that they very much believe what they are saying [00:20:00] is true

Now to some, that's even greater cause for concern than the idea that they are just doing it for marketing. But I just don't think there's a lot of evidence

that there's an ulterior motive other than getting people to agree with their position and their concerns. and like Taylor said, and Derek Thompson echoed in a different post of course, the groups whose stated purpose is to get regulation around this stuff are going to jump on this opportunity and maybe even involved in coordinating the response to it.

That's just how politics works

So what are you even supposed to do with this conversation at this point?

One answer is that we could just run around in blind terror, consuming all the media we possibly can to make us more and more scared about an unfalsifiable theory. Another is to follow the politicians and angrily demand largely nonspecific action



but it is worth noting before we take either of those courses

While the majority of response in this particular case



has been to give these concerns more of a platform to speak

there are plenty of folks

who have issues with this entire discourse.

All In's Jason Calacanis writes,

" The time between the we're all going to die posts jumping from x.com to national coverage is [00:21:00] under 24 hours. Good luck building data centers, deploying Waymos, and getting AI tools taught in schools I've never seen anything like this in my 30-plus year career in tech.

These PDoom posts are like Steve Jobs telling you that the iPhone was going to result in eating disorders, political unrest, mass depression, anxiety, and suicide during the debut keynote, and you can buy them in one of four playful colors

Mark Kretchman writes, " The AI doomers are firing on all cylinders right now. They see the growing backlash against data centers as their big chance to turn that momentum into support for their dystopian vision of AI control. Make no mistake, for them, this is 100% about control.

Data centers are merely the pressure point. The real goal is control who gets to build AI, who gets access to it, and how fast progress is allowed to move. This isn't about saving anyone



one of the more popular angry responses came from author Daniel Jeffries, who wrote: " Not only am I tired of these wild AI speculations of impending doom from self-important people, I resent them. I actively resent people proposing to crash the economy or proposing authoritarian control over my life and other people's lives [00:22:00] with idiotic and dangerous ideas like chip control or bans or tracking researchers.

Every idiot in history who's taken the approach of the ends justify the means to solve an imaginary future disaster created the very disaster they wanted to stop. See Population Bomb leading to one-child policy and communism leading to Mao's mass famines and fascism leading to the deaths of tens of millions of people in war.

These solutions are evil and worse than the disease that they propose to solve. If you believe you can actually predict the end of the world, then you are as insane as the Heaven's Gate cult that killed themselves in the '90s thinking UFOs were coming to transcend them. Not only should we not take your policies and fear-mongering seriously, we should actively throw out any and all of your proposed solutions because they come from a place of delusion.

I don't give a shit that you work in the industry or think you saw something or that you want to virtue signal on X. You are actively contributing to a horrible future based on baseless speculation that has no grounding in reality.

Don't confuse expertise in a domain with ability to predict impact of that domain, or frankly, to make predictions a decade out better than a dart-throwing monkey. They are orthogonal skills. how's Geoffrey Hinton's we don't need to train radiologists anymore working out?

[00:23:00] How did the population bomb work out? global cooling, second Ice Age, peak oil? Just because someone is a bridge engineer does not mean they can predict the impact of bridges on society or that they have any actual useful insights at all on the complex, ever-changing system called life There are people dying in wars right now, children starving, homelessness, dictatorships.

In short, real problems. And we're supposed to drop everything to stop a made-up problem in your head? Pound sand. We don't care, and we are not going to remake society based on your scary monsters under the bed delusion



And for many, this idea

that there is more danger in the people who seek control because of the risk than the risk itself is the resonant thing

Eric S. Raymond wrote, " The kind of totalitarian control that doomers and decelerationists want is a far more certain danger to our future than runaway AI. I would much rather risk the latter."



People point to the part of Jacob's thread where he says that at OpenAI many have not deeply internalized the civilizational stakes, while at Anthropic the stakes are well understood, but they believe no one else will act responsibly so they must do it themselves.



as evidence of this sort of messianic complex

not for nothing, Crypto journalist Laura Shin [00:24:00] also connected it to SBF And having been fairly close to that situation, it has always been my argument

that the reason that Sam was willing to play so fast and loose with the rules was not that he was trying to steal anyone's money, but that he genuinely believed that he alone, he uniquely could save the world, and because that was so urgent, he wasn't willing to let any trivialities



such as his obligation not to bet people's money on crypto, to slow him down



again, I will remind you again here, going back to what I said at the very beginning, that we are specifically in the section of the show where I am talking about the negative responses that people had to this



I am not claiming that these are the only or correct responses. I am just trying to give the full range of how people are engaging with this issue and this message right now



the, for some the big issue is the hand waviness. of the claims and the inability to articulate



specific points in problems at which we lose control and these terrible scenarios come to light

As Chubby on X put it, " The concerns about the potential havoc AI might wreak are so heavily laden with hypotheticals. So far, all I'm reading is that AI, one, can be [00:25:00] misused, two, sometimes behaves in ways that defies expectations, and three, is the subject of a global race between nations.

All of that is certainly true, yet I fail to see how this translates into a danger so significant and tangible that these people would quit their jobs

On the contrary, humanity has always found ways to ensure its survival when facing existential threats. Take nuclear weapons, for instance. The only difference here is that AI is an entity alleged to be, at least in part, uncontrollable. However, I still see no scientific basis for the conclusion or argument that this could lead to humanity's extinction

Sam Liu wrote, " I dropped out of a PhD in AI safety partially for the opposite reason. I didn't believe AI existential risk was as important as the doomers think. My biggest pet peeve is that no one can really provide tangible pathways to why it matters. During my PhD, my research group, half of whom specialized in engineering risk analysis, did an internal study trying to assess concrete catastrophic AI scenarios.

The basic premise was that while we don't know how AI will evolve, the ways in which humans perish are pretty consistent through history, the horsemen of the apocalypse, and [00:26:00] institutions have obviously been very motivated to analyze concrete risks from things like plague, war, et cetera. You can do a decent risk model by asking how a superintelligent AI can perturb each of these models.

The result, most of the issues, e.g. cyber risk, are akin to what economists call structural unemployment. Big problems, but ultimately resolvable in the long run and not a deal breaker. The only real concerning issue was bio risk, and it feels like the intervention points there lie more with bio than with AI as a whole, although a holistic approach is needed."



And for many, the issue was even simpler



which is in short that if we are going to have this conversation about risk, we also need to talk about the potential rewards If AI is just all risk with no gains, of course we shouldn't do it. But presumably, for all these people who are building it,

there is a good potential future

that could be so good it's worth this risk.

Sporadica on X wrote, " How great would it be if one of the frontier labs decided tomorrow to be the pro-AI optimism lab? Like instead of all the labs peddling doom and gloom amidst their skyrocketing financials and social clout, [00:27:00] how cool would it be if one of them was just like, 'We think AI is good?'"

Chris Haydek, who does life sciences at OpenAI agrees saying, " "I work I work at OpenAI and personally think AI has been and will continue to be an extremely beneficial technology to humanity. The conversation should be around how many billions of lives it will save."

A common way I've seen people describe this is instead of discussing P doom, i.e., the percentage chance you ascribe to an extremely negative human extinction type scenario, as Joe Burnett put it, we should spend more time discussing P boom, super intelligence creating unprecedented human flourishing

David Zell agreed, phrasing it slightly differently. " If AI is powerful enough to end the world, it must be powerful enough to radically improve it too. So I wish there was more discussion of P boon, the chance that AI goes great and helps us live happier, healthier, and longer lives.

If anyone builds it, everyone flourishes."

Ryan Orhan summed up something that I've frequently said on this show when he wrote, " There are two completely insane extremes in the AI debate. The first, AI is harmless. Safety is a psyop. [00:28:00] Build as fast as possible and don't stop for anything. Or AI is going to kill us all. It's stealing our jobs, using our water, and destroying humanity.

Shut it all down. F both extremes. should progress as fast as we can make it progress, But alignment needs to move just as fast. the goal should be to build the most powerful technology humanity has ever created without effing losing control of it

I do believe that there is vastly more middle space

than these two extremes, despite these two extremes tending to dominate the narrative in media space



so what are the highlights and concerns that are most resonant for me around this?

The first is incentives. it doesn't have to be a big conspiracy to contextualize how we understand different takes

With understanding what people who are amplifying certain messages have to gain from those messages being amplified

In other words, I'm talking less about nefarious EA funding networks

And more about the fact that politicians who might have been pro-AI six months ago have seen that now it's not only a net drag to be so, but they can actually win points by being against it



that should be part of our [00:29:00] consideration in how we understand their position. And by the way, the inverse of this is of course true

Which I think is why people are skeptical of pro-AI messages from people who stand to gain financially from it.



a second concern is around specificity

I fear that the generic hand waviness of these sorts of predictions make them much more dangerous for policy Which is not to say that policy can't be made to try to avoid certain future scenarios



But that I believe that the more specific those concerning issues are, the better the policy is likely to be



ban superintelligence, for example, is a much more blunt instrument

than, for example, having a specific licensing regime

for people using AI models for bioengineering above a certain model capability

And certainly part of my worries about policy are that I don't particularly have a lot of faith in the current political class in general, by the way, not just on one side of the aisle or the other



to handle these issues with the sophistication and nuance they require



I worry that there is a certain political naivety among those at the labs who are just basically asking to pass the buck over to them



als-- I also find [00:30:00] myself sympathetic to the cure worse than the disease arguments We are living in a classic safety versus freedom conundrum.

And I just don't think historically speaking, giving up a lot of freedom for safety has gone particularly well for those who have surrendered freedom

Which by the way is another reason for me that I'd like to see the arguments be more specific because painting all policy as giving up freedom is a broad brush that absolutely doesn't have to be the case lastly, I have always worried, and I continue to worry now, that focus on future theoreticals before we're in a position to really understand what those risks are

crowds out space for more current and contemporary issues

I think it's pretty clear at this point, for example, that our cyber defense infrastructure is not equipped for the new world we're moving into, and that is a clear and present danger that demands response right now

And while yes, it is absolutely true that theoretically we can do two things at once and that it doesn't have to be a zero-sum choice between one risk or another risk

There is only so much political will to go around



And apportioning it matters



So where would I like to see the conversations go from here?

[00:31:00] First First of all...



on this idea of specificity



I actually think that there is an incredible amount of space to build consensus from the ground up on common sense things



For example, certain types of reporting requirements and oversight

are areas where I believe there would be very, very broad consensus among people and would provide a foundation from which to build the next more difficult to achieve consensus

This is again an area where I believe that the extremes of the argument as presented and as amplified by media do us a disservice by not showing us how much room there is for agreement

A second thing that I'd like to see is some actual frigging coordination One of the reasons that I was so frustrated with the whole pacing the frontier thing

is that it didn't extend to the actual obvious step

which is for OpenAI and Anthropic to put down their weapons, lock arms, and say, "This is what we think we should actually do." A single photo op of Sam Altman and Dario Amodei alone together agreeing would do [00:32:00] more than 10,000 Twitter debates ever could

John Schulman, formerly of OpenAI, now at Thinking Machines Lab, writes, " First step is for industry leaders OpenAI and Anthropic to stop feuding and work on a pacing proposal together. They'll cite antitrust, but that's fake. Antitrust prohibits certain agreements, but not from jointly developing a proposal.

Bringing in the US government before there's a concrete proposal will likely result in something dumb. See our pre-release testing program."



and by the way, I think the attempt at coordination also extends internationally

For example, Derek Thompson wrote, " If the frontier labs feel obligated to build something they think is dangerous because China is going to build it anyway, we'd better be really sure that China is going to build it anyway. Like really, really sure. Are we? Are we actually sure? The CCP wants to build an out-of-control recursively self-improving model because its neurotically control-obsessed government thinks this is a policy worth pursuing?

We're 100% sure about that?"

Now, it is dangerous to open up the kettle of fish about China at the very end of this episode. But I do think that this is a conversation that we should at least be having



I'll [00:33:00] leave you here with two thoughts. This is unfortunately not the type of episode that has an easy conclusion



the nature of this particular debate is such that there will be some crescendo

After which it will fade slowly again until the next time it happens to rise.



But I will leave you with this thought. I think in spite of all of this, in spite of the direness of the warnings The tense tenor of the conversation from all sides, the antagonism or even outright hostility

to people on the opposite side of the debate whichever side of the debate you are on. I believe that there is reason for optimism.



Reflecting on the situation, The Information's Martin Peers wrote last night, " Are we sleepwalking our way into AI-caused extinction? It feels a little like that, given an Anthropic researcher's X post on Tuesday night that there's a greater than 10% chance that AI could kill all humans within the next decade."

except that's completely wrong. This conversation, the fact that it made it to every major news outlet

The fact that I had to dedicate this entire show to this topic instead of the sort of practical positive [00:34:00] thing that most of you are here for This is all exemplary of us not sleepwalking

In fact, so far, with every single capability jump of AI

The conversation about its risks and the political resonance of that discourse has gotten louder

That is exactly what should happen

Even the guy from Anthropic who gave that greater than 10% chance made clear that he was not talking about today's models but about a future which he sees on the horizon



conversations about that future are us not sleepwalking

Now what's clear is that we're coming to a point where it's likely that some policy about a future that hasn't happened yet will be made



if society comes broadly to agree that recursively improving superintelligence

cannot be contained once it exists, then by definition the policy has to happen before that exists.

People are paying attention.

And the time to have these debates is now



back. Now tomorrow, God willing, we will be back

[00:35:00] to more practical things about how you can take advantage of this technology to make your life and your work better right now



But I do commit to trying to keep this show

a space that is unwilling to play to the politics of outrage

And where we can have hard conversations

without having to get angry at the people who disagree

It won't be easy, but I'm confident that if you guys are still here and listening at this point, that we can make it happen. For now, that's gonna do it for today's AI Daily Brief. Appreciate you listening or watching as always, and until next time, peace​