Anthropic's alignment lead puts the risk of AI wiping out humanity at over 10%

This morning I came across a headline that seemed designed to get clicks: an Anthropic researcher estimates the chance that artificial intelligence will wipe out humanity in the next decade at more than 10%.

Easy to dismiss as yet another apocalyptic prediction about AI.

The problem shows up when you see who said it and, more importantly, read the rest of his statement.

Where that 10% came from

Evan Hubinger leads Alignment Science at Anthropic. His work is directly tied to one of the hardest problems in artificial intelligence: getting increasingly capable systems to keep goals compatible with ours and stay under control.

Hubinger wrote publicly that he personally considers the chance that AI could wipe out all human beings in the next decade to be above 10%.

That sentence needs context.

It is not an official Anthropic estimate, nor does it mean there is a study that has mathematically calculated 10.7% or 12.3%. It is the personal assessment of a researcher specializing in alignment.

And he is not saying that Claude, ChatGPT, or Gemini could trigger a catastrophe of that magnitude tomorrow.

Hubinger has clarified that he considers the risk from current models to be low.

His concern looks several steps further ahead.

The problem is superintelligence

Let's imagine we use an AI to help us develop the next generation of artificial intelligence.

The new generation is better at programming, researching, and designing models, so it takes part even more in creating its successor.

We repeat the process.

At some point we could have systems that improve the tools used to build even more capable systems.

This is usually called recursive self-improvement.

That's where the problem changes quite a bit.

It's no longer enough to verify that a chatbot refuses certain dangerous instructions. We need to guarantee that a far more intelligent, autonomous, and capable system keeps doing what we actually want even when it can find strategies we hadn't even considered.

That is the alignment problem.

And it still isn't solved.

The phrase that worries me more than the 10%

Hubinger said something that seems to me far more important than his percentage:

Anthropic still doesn't have a plan to solve the alignment of a superintelligence and, by its own assessment, isn't clearly on track to achieve it either.

That changes quite a bit how I interpret the news.

It's perfectly reasonable to debate whether his 10% is too high.

It could be 1%. It could be 0.1%. It could also be higher. We're trying to assign probabilities to a technology that doesn't exist yet, so there's inevitably enormous uncertainty.

But there's a much more concrete question.

Do we have a proven solution for controlling an intelligence far superior to our own?

For now, no.

Anthropic itself has explained for years that there is still no solid understanding of how to guarantee that extremely powerful systems remain robustly aligned with human values.

A researcher just resigned over this

Hubinger's statement didn't appear in isolation either.

Jacob Coxon resigned from Anthropic after having worked for approximately three years in pretraining research between OpenAI and Anthropic.

His accusation was quite strong: he believes neither company is acting responsibly and that there is a race toward systems capable of improving themselves.

Hubinger responded by backing much of his concern.

Samuel Marks, who leads scalable oversight at Anthropic, also publicly noted that AI developers contemplate extinction scenarios or outcomes of comparable severity.

Three people working close to the problem publicly saying similar things on the same day deserve, at the very least, our attention.

It doesn't mean we should panic either

There's a danger with headlines like this.

If we present the story as "Anthropic says there's a 10% chance Claude will kill us," we're telling something that didn't happen.

Current systems still have enormous limitations.

The concern is related to much more capable future systems, particularly those that can contribute substantially to their own development.

Moreover, within the scientific community there is discussion about how much we should worry about extinction scenarios versus problems that AI already causes or can worsen: disinformation, fraud, cyberattacks, job displacement, or concentration of power.

We can address both.

Why keep developing it then?

This is probably the most interesting contradiction in the whole story.

If some researchers at the companies developing these models believe there is such a great risk, why do they keep moving forward?

Because no company works in isolation.

Anthropic may think that developing safe AI is preferable to allowing another organization to get there first with fewer controls. OpenAI may reason similarly. Google too.

And every company knows that stopping unilaterally doesn't obligate the others to do the same.

It's the classic problem of a technological race.

Everyone can recognize the risk while individually having incentives to keep going.

That's why this discussion inevitably ends up leaving the lab and reaching governments, regulation, and international agreements.

What I take away

I work with artificial intelligence every day and I still think it's one of the most useful technologies we developers have had in a long time.

Precisely for that reason, I believe we need to be able to talk about its risks without falling into either extreme.

Not every AI advance signals the end of the world, nor is any concern about safety paranoia.

Hubinger's 10% is a personal opinion and we should present it as such.

But his second claim is far more tangible: we still don't know how to guarantee the alignment of a future superintelligence.

And we're trying to build it.

Conclusion

A few years ago we were debating whether an AI could write code good enough to use in production.

Now researchers at the very companies developing the models are publicly discussing what would happen if those systems start taking part in building superior versions of themselves.

The change in scale is enormous.

I don't know if the real risk is 10%, 1%, or a much smaller fraction. No one can prove that number today.

But we do know something else: capabilities are advancing while the control problem remains open.