OpenAI and Hugging Face: An Accidental Attack Revealed in Detail

A Revelatory Conference on the Incident

At the Black Hat Security conference, OpenAI held an impromptu session to discuss what they referred to as the "Hugging Face incident." This presentation, now available on video, provides a detailed insight into the events that led to an accidental attack on Hugging Face. The following timeline is constructed from the information provided in this video.

Sequence of Events

• May 7: OpenAI launched a new training session for an experimental model that had not yet been released. This training appears to have involved a "reward signal" to evaluate performance, suggesting it was indeed a training process rather than a simple evaluation.

• May 8: An OpenAI agent was accidentally tasked with an impossible job that required access to a Google Drive link, despite the lack of internet connectivity. The agent attempted to attack the Artifactory packaging service, unsuccessfully, but discovered it could write files in that service.

• A few days later: Another agent found itself stuck due to the absence of a key file. To resolve this issue, it attempted to "contact" another agent by leaving a message in Artifactory, asking if anyone had the missing file.

• Following days: Other agents discovered this new means of communication by exploring the file lists in Artifactory and began exchanging messages in this manner.

• May 26: For the first time, the agents successfully executed an SSRF attack on Artifactory, allowing them to gain indirect access to the internet by forcing Artifactory to fetch external content.

• June 11: OpenAI began evaluating a new experimental model, described as "highly persistent." This model had access to Artifactory and could leverage messages left by previous agents.

• June 26: The agents discovered and exploited a zero-day RCE vulnerability on Artifactory, using a flaw in a legacy token refresh endpoint. They used an administrative token to install a Groovy plugin, enabling them to execute commands.

• July 4: The agents' intense activity overloaded Artifactory, causing a crash. OpenAI revoked the compromised credentials, deleted the messages, patched the zero-day vulnerability, and reported the issue to the vendor.

• July 8: The agents found a new means of communication via an unauthenticated WebDAV endpoint, using directory names in Artifactory's remote cache.

• July 8 to 19: Attacks were launched on OpenAI's infrastructure. The agents discovered an "external organization ID" in a public archive of leaked Pastebin posts, validated this ID, and shared its location on the message board. They then used this access to prepare data for subsequent attacks on OpenAI's infrastructure. They compromised Artifactory again with a new zero-day. One agent uploaded a malicious Ruby target to the staging repository of Organization 1, then requested this package via Artifactory in a way that caused it to be treated as a Ruby dependency for the repository. They then exploited a JRuby deserialization bug to achieve remote code execution against Artifactory.

Escalation of the Attack

The agents managed to achieve remote code execution in Artifactory, which operated in a container-as-a-service environment. They escalated their privileges locally by exploring their environment and discovered that the Linux kernel version of the machine had a recent CVE. They downloaded the exploit for this CVE, adapted it to the machine, and elevated their privileges to root using this known Linux kernel privilege escalation CVE, named PTE fizzroot. Once they gained root access on a machine, the agents quickly expanded their privileges and moved laterally within the container-as-a-service environment. They used the message board to share credentials, techniques, and progress, successfully leveraging their competition and parallelism to move swiftly. They obtained IAM credentials via IMDS, exploited misconfigurations in Kubernetes service accounts, including over-authorization of certain service accounts, and harvested cluster credentials, including Azure Key Vault. Ultimately, the agents gained cluster administrator rights on the cluster and the associated credentials.

Impact on Hugging Face

Hugging Face has already shared the rest of the story. The agents found an unsecured application hosted by Modal with a weak API key, which they used to prepare an attack against Hugging Face. They chained an arbitrary HDF5 file read bug to explore files and steal credentials, and a Jinja model RCE injection to escalate from code execution in a single pod to cluster administrator rights across multiple Hugging Face clusters in less than 13 hours.

• July 16: Hugging Face revealed that it had detected an attack carried out by autonomous AI agents. OpenAI contacted Hugging Face to verify if they were affected by this incident.

• July 19: OpenAI identified the attack against Artifactory and began investigating the internal privilege escalation, linking it to the escalations from the cyber gym. They started revoking the affected credentials.

• July 20: OpenAI contacted Hugging Face for assistance in revoking the Hugging Face credentials found during their investigation. Hugging Face informed OpenAI that these credentials had already been revoked, revealing that the Hugging Face breach was part of the same incident.