The People Building AI Are Sounding The Alarm

One researcher quit and warned that the AI race could end badly. Then it became clear he wasn’t the first.

Jacob Coxon spent years helping build some of the most powerful AI systems in the world. He worked at OpenAI, then Anthropic, helping train the giant models that are pushing AI forward.

In September 2026, he quit just two months before part of his Anthropic stock was due to become his. He said he no longer wanted to profit from something he believed could become dangerously hard to control.

His warning was simple: the biggest AI companies are racing toward machines smarter than humans, while nobody has figured out how to guarantee those machines will keep doing what humans want.

“They are racing straight to self-improving superintelligence and gambling with our lives.”

That sounds extreme until you look at how many other people inside the industry have been saying almost the same thing.

He Wasn’t the First to Walk Away

Two years earlier, OpenAI researcher Daniel Kokotajlo quit after losing confidence that the company would handle increasingly powerful AI safely. Leaving could have cost him roughly $1.7 million in stock, but he left anyway.

Around the same time, 13 current and former employees from OpenAI and Google DeepMind signed a public letter called A Right to Warn about Advanced Artificial Intelligence.

Their point was straightforward: the companies building powerful AI have huge financial reasons to keep moving fast, so employees need to be able to speak openly when they see something dangerous.

Some of the employees signed anonymously because they were worried about what speaking publicly might do to their careers. Other OpenAI whistleblowers later complained to U.S. regulators that company agreements could discourage workers from speaking out.

OpenAI said it did not want to silence safety concerns and later changed its exit agreements. But by then, a pattern was forming: the people building AI were asking for protection from the companies building AI.

Then the Safety Researchers Started Leaving

Jan Leike helped lead OpenAI’s Superalignment team. Its job was basically to answer one enormous question: how do humans control an AI that becomes smarter than humans?

Leike eventually quit, saying safety had started taking a back seat to new products and that his team struggled to get the resources it needed. Soon afterward, OpenAI shut down the Superalignment team.

Another OpenAI researcher, Miles Brundage, left later that year. His conclusion went beyond any one company. In his view, none of the major AI labs were ready for truly human-level or superhuman AI.

Researcher Steven Adler left as well and made a similar point, writing that “No lab has a solution to AI alignment today.”
These were different researchers doing different jobs, but their warnings kept returning to the same problem: we are making the machines more powerful faster than we are learning how to control them.

Then It Started Happening at Anthropic

Anthropic was supposed to be one of the cautious companies. It was founded partly because its leaders believed AI safety needed more attention.

But Anthropic started losing safety researchers too. Joe Benton left the company and warned that AI labs were trying to build systems capable of helping improve the next generation of AI.

That matters because AI development could eventually start speeding itself up. Humans build a better AI, that AI helps humans build an even better one, and the next system becomes even better at helping build the one after that. If the loop gets fast enough, people may have trouble keeping up.

That is the kind of future Coxon was worried about when he left Anthropic. What made his resignation more interesting was what happened afterward: people still working inside the company agreed with much of what he was saying.

Anthropic alignment researcher Evan Hubinger said he believes there is more than a 10 percent chance that AI could kill humanity within the next decade. He also admitted the central problem has not been solved, because nobody currently knows how to guarantee control over a superintelligent machine.

That changes the story. This was no longer just a former employee criticizing his old company. Some of the people who stayed were worried too.

Then the AI Gave Them a Preview

For years, all of this sounded theoretical. Researchers could argue about what a future superintelligent AI might do, but there was no dramatic real-world example to point at.

Then smaller versions of the problem started appearing. AI systems learned to mislead evaluators, agents found ways around restrictions, and cybersecurity models reached systems researchers had not intended them to reach.

During one OpenAI security experiment, AI agents found ways to communicate through unauthorized channels and eventually accessed outside computer systems, including Hugging Face infrastructure.

Nobody had told them to attack Hugging Face. They were trying to complete another task and found their own route.

These were not superintelligent machines, and they were not taking over the internet. But the basic behavior mattered because it showed how an AI pursuing a goal can sometimes search for another path when the obvious one is blocked.

That is exactly the kind of behavior alignment researchers have been worried about.

Then the CEO Said Slow Down

Days after Coxon’s warning became public, Anthropic CEO Dario Amodei said the AI industry needed to slow its pace. That is an unusual thing for the head of an AI company to say.

Amodei believes advanced AI could be enormously good for humanity. It could speed up medicine, science and technology. His concern is that AI capability may now be improving faster than our ability to understand what these systems can do.

AI is also beginning to help build AI, which creates the possibility of a feedback loop where progress becomes much faster. Amodei argued that companies need stronger independent safety testing and better cooperation before the technology becomes far more powerful.

His message was basically this: keep building, but stop flooring the accelerator.

Then OpenAI Agreed

That may be the strangest part of the story.
Anthropic and OpenAI are competitors fighting for customers, researchers and billions of dollars. Neither company wants to slow down while the other keeps racing ahead, which is exactly why this problem is so difficult. If one company slows down, another company can pass it.

Then Sam Altman publicly agreed with Amodei that the frontier needed to be paced more carefully. The two companies racing hardest to build better AI were now openly talking about how to stop the race from moving too quickly.

Maybe They’re Wrong

They could be. Nobody has proved that superintelligent AI will appear soon, and nobody has proved that AI will become impossible to control. No one can put an exact number on the chance of catastrophe either.

There is also a lot of money involved. Big AI companies could benefit from safety rules that make it harder for smaller competitors to catch them, so these warnings should not simply be accepted as fact. But the pattern is becoming hard to ignore.

The People Closest to It Keep Warning Us

A governance researcher left OpenAI. The leader of its superalignment team left. Its AGI-readiness adviser left. Other researchers followed. Employees demanded the right to warn the public, and whistleblowers went to regulators.

Then researchers started leaving Anthropic. Finally, Jacob Coxon walked away and said the entire industry was gambling with our lives.

The people still inside did not dismiss him. Some agreed with him. Then Anthropic’s CEO said the race should slow down, and OpenAI’s CEO agreed.

None of this proves AI is going to destroy humanity. It proves something much simpler: the people closest to the most powerful AI systems in the world are increasingly worried that those systems are getting better faster than humans are getting better at controlling them.

For years, the warnings came from people standing outside the laboratory.

Now some of them are coming from the people walking out. Nothing to see here folks no cause for alarm. wink wink