xAI Challenges Minnesota's Anti-Nudification Law in Court

xAI Challenges Minnesota's Anti-Nudification Law in Court

xAI is suing Minnesota Attorney General Keith Ellison over a law passed in May that broadly targets “nudification” applications, claiming that the punitive provisions of this law leave the company with “no practical choice” but to restrict the image editing features of Grok Imagine in various ways. The company argues that this law violates the First Amendment.

In January, Grok flooded the internet with millions of sexual deepfakes, including images of minors. In another era, a content moderation disaster of this magnitude would have prompted the CEO to disable the product within hours. But in the bold new world of Elon Musk, the chatbot continued to produce pornographic images for days. “Grok is now number 1 on the App Store in country after country!” he posted amid the deepfake outbreak.

Something had gone wrong. Was the law simply ill-suited to curb an explosion of non-consensual nudes? Governments around the world were asking similar questions. The EU launched an investigation, as did the UK. There were reactions from government officials in France, India, and Malaysia. And while the U.S. federal government has remained remarkably silent on the issue, states like California and Minnesota have adopted a stricter stance.

In Minnesota, the state legislature passed a law targeting “nudification” technology, prohibiting website owners from allowing people to “access, download, or use” such software. (The law includes an exception for technology that “requires the technical skills of a user to nudify an image or video”—so tailored sexual harassment via Photoshop is off the table.) The law creates a civil cause of action for individuals who wish to sue and also allows the state attorney general to seek penalties of up to $500,000 per violation.

This law, touted as the first of its kind, was signed by Governor Tim Walz in May and is set to take effect on August 1. It is somewhat mysterious why xAI did not file a complaint over the past three months and instead chose to take legal action just days before the law's implementation. This kind of late reaction raises questions about whether someone at xAI suddenly realized last week that they were about to violate a fairly punitive law and then spent the following days searching for a law firm willing to work tirelessly for a non-consensual deepfake generator. One can only hope that the lawyers involved stand to make a considerable profit.

The complaint is quite reasonable, all things considered. The Minnesota law is too broad, it argues, as it imposes liability even for consensually generated images, or images of adults that are merely a bit sexy, or images with “humorous” or “artistic” merit. The First Amendment stipulates that laws must be drafted precisely to avoid banning social and political commentary like that deepfake of Chris Christie and JB Pritzker sumo wrestling.

An actual excerpt from page 20 of the complaint.

“Liability also applies… even if the company has deployed nearly perfect, state-of-the-art technical controls to prevent the generation of nude images,” the complaint states with an excruciating degree of fervor. Elsewhere, it laments the strict liability attached to the unsanctioned conduct of its users—“images and videos that users might succeed in creating in violation of Grok's terms of service, which is only possible by circumventing the sophisticated, state-of-the-art controls that xAI employs.”

“xAI takes violations of its terms of service seriously,” the complaint notes, emphasizing that the company has since sued one of its users for using Grok to generate CSAM. (The lawsuit against the alleged CSAM creator was filed more than two months after the Minnesota law was enacted.)

xAI's lawyers may have a point, but their complaint reads like a message from another planet. They argue, for example, that existing Minnesota and federal laws (like the TAKE IT DOWN Act) already prevent the non-consensual distribution of nude deepfakes. These laws, the lawyers contend, are constitutionally drafted—the nudification law, on the other hand, is not. And as with all laws in America that target speech based on its content, these laws must be written in the least restrictive manner possible while serving a “compelling” government interest.

But therein lies the crux of the problem with Grok's deepfake apocalypse. If existing law were up to the task, the entire month of January 2026 would not have been flooded with non-consensual pornography. When the Center for Countering Digital Hate analyzed an 11-day period from December 29 to January 8, it found that Grok had generated approximately 3 million sexualized images. About 23,000 of those images depicted children. “That's a shocking rate of one sexualized image of a child every 41 seconds,” the report states.

Many of these images may not even be illegal. For instance, it is unclear whether a suggestive deepfake of a minor in a bikini constitutes CSAM, although an expert previously told The Verge that other images generated by Grok—such as those showing minors' faces covered in semen—likely violated existing law.

Regardless, the mere fact that a multi-billion-dollar company allowed Grok to operate unchecked for so long suggests that the law did not incentivize the parties involved to behave appropriately. Perhaps Minnesota's HF 1606 is a bad law. But what does that mean when the “good” laws aren't even working?