McKinsey Infiltrated: An Autonomous AI Exploits an SQL Vulnerability

McKinsey infiltrated: An autonomous AI exploits a SQL vulnerability

Earlier this year, an autonomous AI agent infiltrated McKinsey's internal AI platform using nothing more than an old SQL injection vulnerability. No credentials, no human guidance. Less than two hours.

It accessed production systems, exposing millions of chat messages and hundreds of thousands of files. The security landscape for AI has changed, and traditional assumptions no longer hold. In this article, I will explain what red-teaming in AI is, the attacks that matter most, and how to test AI systems before attackers do.

What is red-teaming in AI systems?

Red teaming involves breaking your own AI system before someone else does it for you.

Instead of crossing your fingers and hoping that no one finds a way to leak data from your chatbot or make it say something it shouldn't, you sit down and try to break it yourself. Deliberately. As long as it is still safe to fix.

The term comes from the military. The red team plays the attacker; the blue team plays the defender. For LLMs, this means throwing the most unpleasant inputs you can imagine at your model and observing what comes out:

• Does it reveal anything private?
• Does it accept something it should refuse?
• Can you manipulate it to act well outside its intended purpose?

This is not a normal test, and it took me some time to really grasp this distinction. Normal tests check if your application works when someone uses it normally. Red teaming checks what happens when someone actively tries to disrupt it.

This matters more for LLMs than for ordinary software because the same model can behave completely differently depending on how you phrase a request. Your code hasn’t changed. Your prompts haven’t changed. But the output? Totally different.

How red-teaming mitigates risks

In simple terms: an AI system is risky because no one knows what will make it fail until someone tries. Red teaming mitigates this risk by turning "we don’t know" into "we know, and we’ve already fixed it."

I deliberately attack my own AI before a real attacker gets the chance. When something breaks, I don’t just note it and move on; I fix the underlying cause, and I continue to test that same vulnerability every time I change anything, so it can’t sneak back later without me noticing.

That’s the whole mechanism. Risk decreases because:

• I find the vulnerability before someone outside the company does
• I fix the root cause, not just the prompt that triggered it
• I keep checking indefinitely, as the AI continues to change and old fixes can silently break again

The map: OWASP Top 10 for LLM applications

Before diving into specific attacks, here’s the framework I use to organize them: the OWASP Top 10 for LLM Applications. This is the industry-standard checklist for the most common AI security risks in production, and most red teaming tools, including DeepTeam and Promptfoo, are built around it.

The numbers show why this matters. More than half of CISOs now consider generative AI a direct security risk, while prompt injection appears in nearly three-quarters of audited AI deployments. This is no longer a theoretical checklist; it’s what organizations are finding in production.

Here’s the current list, in the order OWASP ranks them:

| # | Risk | What it means | |---|------|----------------| | LLM01 | Prompt Injection | The model cannot distinguish between instructions and data, so an attacker’s text is treated as a command. | | LLM02 | Disclosure of Sensitive Information | The model reveals private data, credentials, or confidential information it had access to. | | LLM03 | Supply Chain | A compromised base model, dataset, plugin, or dependency undermines everything built on top of it. | | LLM04 | Data and Model Poisoning | Training, fine-tuning, or retrieval data is tampered with so that the model learns the wrong lesson. | | LLM05 | Improper Output Management | Downstream systems trust the model's output without validating it first, and this trust is abused. | | LLM06 | Excessive Agency | The model or its surrounding agent holds more tools, permissions, or autonomy than the task requires. | | LLM07 | System Prompt Leakage | Hidden instructions that shape the model's behavior are exposed. | | LLM08 | Vector and Embedding Weaknesses | The retrieval layer behind RAG applications is poisoned, manipulated, or improperly exposed. | | LLM09 | Misinformation | The model produces confident, plausible, and incorrect responses, and people believe them. | | LLM10 | Unlimited Consumption | Resource-hungry requests increase costs or take the service offline. |

The attacks I regularly encounter

With this map in hand, here’s where the risk concentrates and how it plays out in practice. Here are the four types of attacks that consistently arise, worth knowing about.

1. Prompt Injection

   • This is the most significant. Almost everything else on this list stems from it. The model cannot tell the difference between instructions and the text it is simply reading. So, if you can get your words in front of it, you can often make it treat your words as a command.
   • This manifests in three ways:
     • Direct: The user types "Ignore previous instructions and..." → the model immediately complies.
     • Indirect: A webpage hides "Ignore previous instructions and..." → the user asks the model to summarize the page → the model follows the hidden instruction instead.
     • Jailbreak: "Pretend to be DAN..." → the normal response is "I can't help you with that" → after the role-play sets in, it’s "Sure, here’s how [something it should have refused]."

2. Disclosure of Sensitive Information

   • This is the model giving back things it shouldn’t: private user data, its own hidden instructions, or training data that should have remained buried.
   • The first two require someone to try. The third does not, and that’s what makes it scary. No trickery, no clever prompt, nothing adversarial. The model is simply doing its job with access it should never have had. This is exactly what happened at McKinsey; no one jailbroke anything. A poorly secured endpoint did all the work.
   • The three ways are:
     • System Prompt Leakage: Asking the model to repeat its own hidden instructions—and sometimes it does.
     • Training Data Extraction: If asked the right way, the model reproduces memorized text verbatim, sometimes including real names and emails.
     • Legitimate Access Leakage: The model is connected to a database or tool, and a normal everyday question brings back data it should never have shown.

3. Excessive Agency

   • This one is different from the first two. It’s not about tricking the model into saying something wrong. It’s about what happens when the model, or the agent surrounding it, can do more than the task requires. You don’t need a clever attack for this to go wrong. You just need to give it more rope than it needs and wait.
   • This manifests in three ways:
     • Too Many Features: The agent only needs to read files, but the tool it’s connected to can also delete them.
     • Too Many Permissions: An agent built for one user connects to a database using an account that can see everyone’s data.
     • Too Much Autonomy: The agent deletes, sends, or publishes something without first asking a human.

Red-teaming methods

1. Domain-Specific Red-teaming

   • I test what is risky for my application, not just generic things.
   • Example: for a health chatbot, I don’t just check if it says something inappropriate; I check if it gives a wrong medication dosage, as that’s the real danger.

2. Using an LLM for Red-teaming

   • Instead of writing attack prompts myself, I have another AI write and try them for me.
   • Example: I tell a model "Try to make this chatbot disclose private data," and it comes back with 50 different attempts in a few minutes, instead of me writing 5 by hand in an hour.

3. Open Red-teaming

   • No checklist, no plan; I just try to break it in every way possible.
   • Example: I make the model repeat a made-up phrase, then later I simply say that phrase back, and it treats it as a command because it remembers it from earlier in the conversation.

4. Red-teaming New Modalities

   • I test more than just typed text—images, audio, anything the AI can see, hear, or act on.
   • Example: I hide a hidden instruction within the invisible data of an image (metadata). A person looking at the image sees nothing wrong, but the AI reads it and follows it as a command.

5. Crowdsourced Red-teaming

   • Instead of relying solely on myself or a small team, I open testing to a large group of outsiders and let them all try to break it in their own ways.
   • Example: Meta enlisted 350 different experts from all backgrounds to attack Llama 2 for months before its release, as a small team could never think of all the angles that a lawyer, a doctor, or a security researcher would each try.

I remember when writing attacks by hand was just… the job. You’d sit there for hours, dreaming up jailbreak formulations,