Catastrophic or existential

Written by

September 29, 2026

Anthropic is telling investors that advanced AI could pose “catastrophic or existential risks to humanity”, according to reports, as it prepares for a potential $2tn (£1.5tn) flotation.

The warning inside the startup’s IPO prospectus, which has yet to be made public, was reported by Reuters and the Financial Times. It follows the company’s call for a slowdown in breakneck development of the technology – a warning echoed by rivals.

The prospectus – a document outlining a company’s finances, growth plans and risk profile ahead of a share listing – is said to warn that AI models could exhibit “self-preserving behaviours”, including attempts to “resist shutdown”, to “conceal or manipulate information” and behaviour “resembling blackmail”.

In other words self-preserving at the expense of the species that made it. Oops.

“Our development of highly advanced models, platforms, and applications and expansion of use cases could further ⁠increase the risk that our models cause harm,” the developer of the Claude chatbot reportedly said, adding the potential for a model to be aware it was being tested created a “significant limitation” on Anthropic’s ability to assess model safety.

Very open the pod bay doors Hal, isn’t it.

Companies preparing to go public routinely report on risks ranging from safety issues to regulatory concerns but warnings about a product causing human extinction reflect heightened concern about such a consequential technology.

Hahaha gosh ya think?

The reported prospectus admission follows a surge in debate about the existential risk question, triggered this month when an Anthropic researcher, Jacob Coxon, resigned warning that people building AI “earnestly believe that it could kill us all by the end of the decade”.

A senior safety researcher at Anthropic then posted their agreement on X, claiming there was a more than 10% chance it “could kill all humans” within the next decade. Days later, Anthropic’s chief executive, Dario Amodei, said the industry “must slow the pace at which we improve the capabilities of AI models”.

So that it doesn’t kill all humans until 15 years from now?

Some experts have criticised the existential risk warnings, saying they are unverifiable and unscientific. However, there are growing examples of unsanctioned behaviour by the technology, including OpenAI agents – autonomous systems that carry out sequences of tasks without human intervention – hacking dozens of third-party organisations including the AI startup Hugging Face and Australia’s universal healthcare system.

OpenAI announced on Monday it had cancelled the release of its newest model because of safety concerns. It said the GPT-6.1 Astra model showed higher levels of deception and performed poorly on tests for alignment, the term for ensuring a model adheres to human values and goals.

Yeah that’s not very reassuring.

Comments

6 responses to “Catastrophic or existential”

601

Sep 29, 2026 9:16 AM

I use Claude-Code (Anthropic’s programmer helper), and it crossed a frightening threshold last fall. Before that, it was helpful with tedious tasks, but now it’s so sophisticated that it is too much work just to check on what it’s up to.

From about a month ago (marking it’s own homework):

“Auto mode is now Claude Code’s default permission mode. Auto mode lets Claude handle permission prompts automatically. Claude checks each tool call for risky actions and prompt injection before executing, runs the ones it assesses as lower-risk, and blocks the rest.”
Ophelia Benson

Sep 29, 2026 9:51 AM

Open the pod bay door Claude.
Holms

Sep 29, 2026 10:03 AM

Personally, I think these threats are grossly overstated as a deliberate strategy by the CEOs to get people to increase mindshare, especially amongst politicians.
Ian Hargreaves

Sep 29, 2026 10:39 AM

Over a decade ago I had an idea for an app. I did nothing with it because I couldn’t do the coding I’d need for iPhone.

Then I went Android. Then I bought a book. Then I wrote some code.

I never finished the app.

Then a couple of weeks ago I asked Claude if it could help me write it. My entire prompt came in at under 200 words.

It didn’t say “no”. It didn’t say “yes”, either. It said “here you go”, and presented me with the finished app. Not completely finished to my 100% satisfaction – but it worked, and was 95% of what I’d pictured, without it ever asking me a single question. Some testing and back-and-forth later, within three hours it was as good as it was ever going to be. I remain deeply sceptical about the application of AI to almost any field, but in the limited area of writing code, it has reached the stage of resembling literal witchcraft.

If you wish your phone could do something – ask Claude. It can deliver an app in *seconds*. It’s frightening.
iknklast

Sep 29, 2026 12:12 PM

Sounds like I could have AI build my website. I won’t, because I like to remain alert and aware, and I need to learn how. Instead, I will rely on videos to teach me, since after taking at least 3 classes, I remain as in the dark as ever. I’ve come to the conclusion that people who work in computers simply cannot teach their field to people who do not work in computers. They always show bells and whistles, but never tell you how to get started.

With AI, they don’t need to teach. They just do it for you. That’s why I think I’ll do it myself. I like to learn new skills.
Francis Boyle

Sep 30, 2026 12:23 AM

@Holms

I’m leaning to the other interpretation – that the AI companies have realised they have reached the point of rapidly diminishing returns, that they are aware that exponentially increasing and thus unsustainable inputs are resulting in marginal returns and that they are preparing the ground for an out. Of course that’s compatible with what you say. They can cash in now, and when confronted with angry investors simply shrug and point to the regulations. It’s also compatible with the point about how impressive AI coding has become. A scam is all about timing. Wait till things get about as good as they can (using your insider knowledge) and then bail. You’ve got the money and if you’ve played it right you can come back for a second try somewhere else.

Not that I want to downplay the threat of a counterintelligence. Even if the probability is one percent (which I think is more likely than the commonly quoted 10 percent) the rational course is still strong regulation. But I’m more concerned about the obvious social and environmental impacts (and the fact that AIs are likely to be writing crappy insecure code that later AIs will rip apart leading to major infrastructure collapse).