Patreon and Cloudflare Block Predatory AI Bots

Patreon and Cloudflare Block Predatory AI Bots

Patreon, the membership platform for creators, is ramping up its fight against AI scraping of its content for training purposes. On Thursday, the company announced that it is collaborating with Internet infrastructure provider Cloudflare to directly block access to AI bots designed to train their AI models on creators' work without permission.

The enhanced measures were necessary as AI scraping has become more sophisticated since Patreon implemented measures to deter AI crawlers in 2023. Additionally, Patreon's paywall has long locked much of the creators' content, making it inaccessible to crawlers. However, more recently, the company has introduced new discovery tools, such as a redesigned Home Feed and its tweet-like Quips, which could expose more content to crawlers.

These changes come as more online publishers and content creators become aware of how AI ingests their work to make their AI models smarter. To combat this, Cloudflare is now offering tools that allow website publishers to restrict access to AI bots, including a marketplace that enables sites to charge AI bots for scraping, called Pay Per Crawl. Earlier this month, the company modified its policies so that so-called "mixed-use" crawlers, meaning those that index and train on a site's content, are blocked by default on all pages containing ads.

Patreon states that it is expanding its existing work with Cloudflare to utilize the company's AI Crawl Control technology to update its policies and enforcement tools regarding AI. The difference here is that instead of simply asking AI crawlers not to scrape content using robots.txt files—a standard method for providing instructions to bots on how they can use its site—Patreon is now actively blocking AI training bots.

"Consent should not depend on the behavior of a scraper," explains a blog post from Patreon, referring to the stricter measures.

During testing of the features, weekly attempts by AI training crawlers to access Patreon dropped from "thousands of attempts to zero," the post noted. This indicates that AI scrapers were ignoring Patreon's robots.txt file and continued to scrape the site despite its requests.

However, the company stated that it would allow bots that index pages and organize information that can be used to direct users back to Patreon.

"As AI agents become increasingly powerful and popular, creators deserve a say in how their work is used by AI companies," said Drew Rowny, product lead at Patreon, in the announcement. "Across most of the internet, creators have to accept that their work will be used for AI training just to reach and grow an audience. Patreon has a different vision: creators should be able to grow their audience and control how their work is used."