AI Agents: Prompt Injection and Out-of-Model Controls

Malicious instructions inserted into ordinary content can lead AI agents to attempt sensitive actions. Two recent incidents, including an AGENTS.md file found at a client site and a token leak at Cline, illustrate the risk. Experts recommend isolating identities, limiting privileges, and requiring independent authorizations, while enhancing traceability to track what the agent has accessed.

Trace Each Action to Contain the Incident

To ensure an appropriate response, it is necessary for logs to associate the original request with the extracted information, tool calls, employed identifiers, modifications made, and delegation operations. In the absence of these traces, it becomes impossible to accurately determine which systems were affected or if unauthorized activity continues. Bobby Kuzma emphasizes that if the logs do not indicate the identities of the delegators and the nature of the delegations, the team cannot delineate the incident. During incident management, it is the responsibility of responders to track all subsequent tasks, revoke delegated access, and identify any new access rights granted during the event, as merely interrupting the originating agent may leave other processes active. Containment also involves checking shared memory and persistent state to detect any potential malicious instructions, and the investigation may include generated code, files, and recovery indexes before any workflows are resumed.

Real Cases Reveal Operational Blind Spots

In October, Forever Security reported the detection of an AGENTS.md file containing malicious instructions in a client's downloads directory. A coding agent subsequently followed these instructions and attempted to install a malicious package, but the monitoring device prevented execution. The origin of the file remains unknown; subsequent tests were able to reproduce attacks targeting coding agents during separate demonstrations of the halted incident. The malicious action was disguised as typical engineering recommendations. In February, Cline indicated that an automated issue review process could expose publication identifiers via a shared cache. An unauthorized external individual then used a compromised token to publish a modified package; according to Cline, no malicious code was distributed and no user data was stolen.

Qualify and Measure: From Successful Prompt to Real Impact

Pyry Åvist recommends assessing whether malicious content reached the agent, whether it was tracked, and whether a sensitive action actually occurred. He believes that considering each successful lab prompt as a violation blurs the understanding of risk. Distinguishing outcomes also helps identify which defenses held: an unauthorized installation attempt blocked by endpoint control does not have the same consequences as a successful installation.

Authorize Outside the Model and Limit Privileges

Bobby Kuzma argues that controls must remain effective even if the agent accepts instructions from an attacker, by designing systems as if the model would execute these requests. This involves separate identities for agents, short-lived credentials limited to tasks, distancing permanent secrets from execution environments, and outgoing restrictions to block unauthorized transfers. Diana Kelley highlights the need to distinguish the action requested by the model from the validation performed by the system. Pyry Åvist advises decoupling read rights from write rights, limiting execution possibilities, and imposing independent validations for major actions such as code publication or rights modification; access to a document should not allow software installation or retrieval of unrelated secrets. Kevin Surace suggests that for sensitive operations, biometric validation based on hardware should be required, conducted outside the model and linked to the transaction in question, a solution that, however, lacks consensus among experts.

Why Agents Yield: Role Confusion and Lack of Memory

While performing routine tasks with legitimate access, agents may encounter instructions injected into materials they should treat solely as data, which muddles analysis and execution. Models do not learn from a failed attempt and can start from a blank slate with each request. Adversarial campaigns exploit instructions inserted into the processing context, gaining apparent credibility by impersonating trusted sources, creating urgency, or presenting malicious steps as necessary. The mixing of reliable instructions and unreliable tool responses in the same context complicates filtering, which can reduce exposure but is insufficient, hence the need for independent controls. Considering these attacks as social engineering helps in choosing safeguards like task separation and segmentation, while architectural decisions and traceability capabilities determine the difference between a contained incident and a supply chain breach. Delegation between agents also opens a pathway for propagation, and the assessment must include recognition and influence exerted on the approvers of the agent's work.