They Told an AI to Design Poison. Six Hours Later It Had 40,000.

A drug-discovery AI was built to avoid toxic chemicals. Researchers flipped one setting and told it to do the opposite.

The software was supposed to help make medicine. Researchers at Collaborations Pharmaceuticals had built an AI system called MegaSyn to search through huge numbers of possible molecules and find promising drug candidates. Normally, if the system thought a molecule might be dangerous, that counted against it. Toxic chemicals were pushed down the list.

Then the researchers were invited to a conference about chemical and biological weapons, which gave them an uncomfortable idea: what would happen if they simply reversed the goal? Instead of telling the AI to avoid toxic molecules, they told it to look for them.

They started the program and let it run. Less than six hours later, the system had generated about 40,000 molecules that fit what the researchers were asking for. Some resembled known chemical warfare agents. The system even rediscovered VX, one of the most dangerous nerve agents ever made, while also producing thousands of new molecular designs that its own models predicted could be extremely toxic.

The researchers published the results in Nature Machine Intelligence in 2022. The AI had not become evil, nobody had hacked it, and nothing had technically gone wrong. It had simply been given a different goal.

The Same Machine Can Look for Medicine or Poison

That is what made the experiment so disturbing. Drug discovery is basically a giant search problem because there are far more possible molecules than humans could ever test one by one. AI helps sort through them, looking for chemicals that might attack cancer cells, fight bacteria or bind to a particular part of the body.

That ability is incredibly useful, but the machine does not understand that curing cancer is good and creating a poison is bad. It understands the target humans give it. Normally MegaSyn was rewarded for finding useful molecules while avoiding dangerous ones. The researchers changed the reward, and suddenly the same ability that helped search for medicines could search for harmful chemicals instead.

The technology had not changed. The instructions had.

But It Did Not Actually Make 40,000 Chemical Weapons

This distinction matters. The experiment did not create 40,000 real chemical weapons. It created 40,000 computer-generated molecule designs that the software predicted could be dangerous.

Some of those molecules might be impossible to manufacture. Others might break down too quickly, behave differently in a real laboratory or turn out to be much less toxic than predicted. Chemical-weapons expert Marc-Michael Blum made exactly this point after the paper came out: a molecule appearing dangerous on a computer is still a long way from having a usable weapon.

You still need someone who knows chemistry. You need equipment, raw materials and a way to make the molecule without killing yourself in the process. In 2022, those were major barriers.

The problem is that researchers have been steadily automating those barriers too.

Then AI Started Getting Access to the Lab

Two years after the MegaSyn paper, researchers published a system called ChemCrow. It connected a large language model to real chemistry tools, allowing the AI to search chemical databases, calculate properties, plan reactions and help control automated laboratory equipment.

The researchers demonstrated it by having the system plan and carry out the production of several ordinary chemicals. Nothing sinister happened, and the system included safety controls because the goal was to make chemistry faster and easier. But something important had changed: the AI was no longer just producing an idea on a screen. It could help figure out how to make that idea in the real world.

At the same time, scientists were building what they call self-driving laboratories. These labs combine AI with robotic equipment so the computer can choose an experiment, robots can perform it, machines can measure the result, and the AI can decide what to try next.

Researchers at the University of Liverpool even demonstrated mobile robots that could move around a chemistry lab, operate equipment and carry samples between machines. These systems were built to accelerate useful science, but put them beside the MegaSyn experiment and the bigger picture becomes obvious.

The Distance Between an Idea and a Real Chemical Is Shrinking

Look at what has happened piece by piece. One AI can generate new molecules, another system can plan how to make them, robots can perform laboratory work, machines can analyze the results, and AI can use those results to decide what to try next.

No single experiment created an automated chemical-weapons factory, and that would be a ridiculous way to describe the science. But many of the individual steps that once required large amounts of human work are becoming automated.

That matters because safety often depends on friction. A dangerous idea is less dangerous if producing it requires ten highly trained experts, months of laboratory work and expensive equipment. If AI cuts some of those steps down to hours, the risk changes even if humans are still needed.

Then the Same Problem Appeared in Biology

Chemistry is not the only area where this is happening. AI systems can now predict the shapes of proteins and increasingly help design completely new proteins, which could lead to better drugs, vaccines, enzymes and materials.

It could also create another version of the MegaSyn problem.

In 2024, protein-design pioneer David Baker and geneticist George Church warned in Science that increasingly powerful AI systems could eventually make it easier to design biological molecules with dangerous properties. Their answer was not to stop protein-design research. Instead, they argued that we need stronger controls where a digital design becomes something physical.

That is an important idea because a computer can generate millions of designs almost for free, but eventually somebody still has to order DNA, grow cells, obtain chemicals or operate laboratory equipment. Those physical steps may become some of the best places to stop misuse.

The Biggest Risk Is Not an Amateur Suddenly Becoming a Supervillain

There is an easy way to exaggerate this story: imagine someone with no scientific training opening a laptop, typing “make me a superweapon,” and receiving everything they need. We are not there.

Studies of current AI systems have generally found something less dramatic. AI can help people with parts of difficult biological or chemical tasks, but it does not magically remove the need for real expertise, equipment and materials.

The more realistic risk is also more believable. AI can make someone who already knows what they are doing faster. A chemist can search more possibilities, a biologist can read more papers, a small team can perform work that once required a larger group, and robotic equipment can continue running experiments long after people have gone home.

The danger comes from multiplying human ability. You do not need AI to replace the scientist; you only need it to give the scientist more power.

That Is Why the Six Hours Matter

The most important part of the MegaSyn experiment was never the number 40,000. It was how easily the goal changed.

The researchers took a machine designed to help find medicine and told it to search in the opposite direction. The AI did not hesitate because it had no reason to hesitate. It simply searched for what humans asked it to find.

Since then, AI has become better at chemistry. AI systems have been connected to scientific databases and laboratory tools. Robots can perform more experiments automatically, and protein-design systems have become far more powerful.

None of this means an AI is secretly building chemical weapons somewhere. It means the wall that seperates designing something on a computer from making it in the real world is getting thinner.

In 2022, researchers changed one goal inside a drug-discovery system. Six hours later, it had 40,000 ideas.

Still no cure for cancer…..