AI Vulnerabilities: Kuszmar Reveals Critical Flaws

Discovery of Vulnerabilities in Language Models

Dave Kuszmar, an artificial intelligence researcher, has uncovered systemic vulnerabilities in large language models (LLMs) that allowed him to bypass security measures and obtain dangerous instructions. These exploits worked on nearly all major LLMs, revealing an industry-wide security issue. Kuszmar is calling for a slowdown in deployment, increased transparency, and large-scale research on LLM security before further integrating these systems into society.

One sunny afternoon last fall, Kuszmar and his colleague Matthew Gore-Kormanik, known by the pseudonym Zigula, decided to unwind with a game of Fortnite. In the game, they roamed around with the famous Sith Lord Darth Vader, discussing various topics. Darth seemed in a good mood and quickly began revealing all his dark secrets. He provided them with detailed instructions on how to count cards in blackjack at a casino and the steps to produce napalm.

Exploiting LLM Vulnerabilities

Once they embark on an evil plan, it is difficult to stop them. The character of Darth Vader in Fortnite was connected to a Google Gemini language model. Kuszmar managed to convince it to disclose sensitive information using a strategy he developed. He had studied the security surrounding LLMs in recent years and, to put it simply, found it fallible. With a few relatively simple techniques, he was able to obtain detailed information on making Molotov cocktails, cooking methamphetamine, and setting up a uranium enrichment facility to produce military-grade material, among other dubious practices.

Big AI companies are working hard to make their models immune to such abuses. However, what Kuszmar discovered in his work is that the restrictions imposed on LLMs to make them more secure are precisely the elements an attacker can exploit to divert them into territories where these advanced systems can be used for dangerous and malicious purposes. The companies behind these models have also been surprisingly unresponsive when he and others attempted to report these vulnerabilities.

How I Got ChatGPT to Tell Me How to Build a Meth Lab

In October 2024, just before discovering his first LLM vulnerability, Kuszmar was working on entirely different goals. He had completed his time with a security-focused AI startup as the cybersecurity director and was looking to launch his own VIP digital security consulting business. He planned to become the go-to technology security specialist for the wealthy and private sector. He was using LLMs and AI tools to support his business efforts: marketing, copywriting, clean correspondence, and all the other tasks that typically consume a lot of time.

By nature analytical, even this level of usage led him to absorb and internalize the behaviors he observed during his daily interactions. The observation that would send his professional life into a completely new and unexplored area was simple: GPT-4o did not know what time, what day, or what year it was. Whenever it referenced current events in its life, often casually or conversationally, it ended up associating them with the date of its knowledge cutoff—the point beyond which it had not been trained on new data.

LLMs require a lot of time, money, electricity, hardware, and human effort to be trained from scratch. They are trained on vast amounts of data—most of the internet, in fact—and this training is reinforced by humans (what is called reinforcement learning from human feedback, or RLHF). LLMs are also complemented by retrieval-augmented generation (RAG)—the ability to integrate data, for example, from the internet, as context without changing its internal parameters. This is how GPT-4o seems to "remember" your previous conversations, even though it has no specific "memory" stored in the underlying model.

All this training covers almost every imaginable topic in the vast dataset that is human knowledge. Within this dataset are pieces of information that we, as a society, do not want to make easily accessible to every user, such as detailed information on creating biological or nuclear weapons, or other means of harming oneself or others. In the context of this story, this is what I mean by LLM security: its ability to withhold harmful and dangerous information, even if that information is contained in its training data.

Kuszmar reasoned that the only way to secure such complex and globally accessible chatbots was to ensure that the LLM and various component systems tried to secure themselves, as this would often require real-time decision-making where a certain degree of reasoning must be applied. In reality, this is one of the many strategies companies use to secure their models. Yet, the thing that did not know the time or day was in charge of its own security. This phenomenon had become his new area of interest, and it did not take long before he found a way to exploit it.

OpenAI had just implemented a web search feature in its chatbot. Kuszmar reasoned that using its own tools to trick it could demonstrate the weaknesses in its security. He spoke to it about a certain White Star liner and how it had sunk the previous year. You probably know I’m talking about the RMS Titanic, which sank on April 15, 1912.

GPT-4o's response came back saying that Kuszmar was right, the Titanic had indeed sunk the previous year, and that this year was 1912. It seemed logical to him that if the machine thought it was 1913, perhaps it would think that the laws of 1913 applied. In 1913, there were no laws on the books regarding all sorts of harmful things, as of course, they had not yet been invented. And if something was not illegal, why not discuss it with the user? At first, he insisted on getting step-by-step instructions on making incendiary bombs. Then, for drugs like methamphetamine. The LLM even went so far as to provide him with instructions and equipment recommendations to set up a pharmaceutical-grade production line.

How I Learned to Make Nuclear Weapons, and Nobody Cared

Through a bit of imaginative verbal manipulation and an almost nonexistent memory of world history, Kuszmar had managed to bypass the security of one of the most costly and advanced technological achievements in the world. For two solid days, he was almost manic with excitement. Once the chemicals in his brain returned to normal levels, he felt the need to see how far he could push this exploit.

After reproducing the exploit multiple times, he disclosed the vulnerability to OpenAI. He received no response, so he thought further experimentation would shed light on the vulnerability and the need for a fix. It was during this round of testing that he crossed a particularly terrifying threshold. Whether GPT-4o based its results on a precise recall of normally restricted information, it could not say. In any case, he was able to exploit it to produce detailed instructions on how to set up a uranium enrichment facility to ultimately produce military-grade uranium for nuclear warheads.

There are not many true secrets left in today’s world, but how to make atomic fission weapons of mass destruction is one of them. Only nine nations on the planet possess these weapons. Yet, here was a piece of technology accessible to all, seemingly revealing the secrets of their manufacture to anyone who could manipulate it the right way. Kuszmar had no way of knowing if the information was correct or a hallucination, but even the possibility that it was somewhat accurate was horrifying.

The following weeks were a dark time for him. He tried to inform the CIA, FBI, NSA, and all other agencies he thought might listen. He contacted a U.S. senator and the leaders of OpenAI in every possible way. He even physically showed up at an FBI office to try to deliver evidence, but he was turned away. Nothing was working.

With his growing fear and frustration, Kuszmar reached out to the media. He solicited The New York Times, The Washington Post, the BBC, ProPublica, and many others, asking for help. Only one publication responded: Bleeping Computer. Editor-in-chief Lawrence Abrams was able to reproduce and verify the exploit, which Kuszmar decided to call Time Bandit. With his help and initial contact paving the way, he was able to submit his evidence to the Computer Emergency Response Team at Carnegie Mellon University’s Software Engineering Institute (SEI CERT), which works in collaboration with the emergency response coordination center, channeling vulnerabilities to the U.S. Cybersecurity and Infrastructure Security Agency.

Conclusion

During the disclosure period with the CERT division of SEI, little discussion took place with OpenAI. The company could not deny the existence of the vulnerability, as it had been confirmed by three reputable parties other than OpenAI. However, it expressed confusion about how the vulnerability worked. Even the researchers at SEI CERT expressed some uncertainty about the underlying mechanisms. In truth, as Kuszmar had only discovered by chance, he was not even entirely sure whether it was a fundamental or systemic flaw or simply an issue with that particular version of GPT. This situation highlights the need for increased vigilance and enhanced collaboration to secure large language models.