Cloudflare Charges AI Companies for Content

⚡

Key Takeaways

1Cloudflare will block mixed-use crawlers on sites with advertisements starting September 2026, unless settings are adjusted.

2Google, as cited by Cloudflare, has twice as much access to information as other AI companies, but offers opt-out options for AI training.

3Cloudflare is introducing a "Pay Per Use" model for publishers to charge AI companies based on the value created by their content.

💡Why it matters — This policy could transform access to web content for AI training, impacting the business models of AI companies and publishers.

Cloudflare Redefines AI Bots' Access to Web Content

Cloudflare recently announced a new policy that could shake up the AI industry. Starting September 15, 2026, web crawlers used for mixed purposes—meaning for traditional research and training AI agents—will be blocked by default on all web pages hosting advertisements. This measure will apply automatically unless site owners change their settings. This decision affects not only new Cloudflare customers but also new sites created by existing customers and all clients using the free version of the service.

This initiative could transform how AI model providers access online content for training their systems and operating their agents. Indeed, Cloudflare aims to protect the intellectual property of site owners who want their content to be accessible for research but not exploited for free by AI services.

Google in Cloudflare's Crosshairs

Cloudflare specifically mentioned the "largest search engine in the world," a clear reference to Google, which enjoys privileged access to about twice as much information as other AI companies. This is due to Google's difficulty in allowing sites to remain visible in search results without their content being used for AI training.

Google has already responded to such criticisms by highlighting the existence of Google Extended, a bot that allows site owners to opt out of having their content used for AI training without affecting their presence in Google Search. However, the main Googlebot continues to crawl the web for research, including for AI-integrated features like AI Overviews and AI Mode.

Towards a Sustainable Digital Ecosystem

Matthew Prince, co-founder and CEO of Cloudflare, emphasized the need for swift action to establish a sustainable digital ecosystem, as online traffic is now predominantly generated by bots rather than humans. This paradigm shift, although anticipated for next year, requires immediate measures to protect the interests of site owners.

Cloudflare is thus offering new tools and partnerships to provide site owners with better visibility and increased business opportunities. The goal is to encourage crawlers to clearly distinguish between research and the use of agents and AI training.

A Rethought Business Model for Publishers

In parallel, Cloudflare has developed a series of tools allowing publishers to better control their content in the age of AI. Among these tools is a marketplace that enables sites to charge AI bots for crawling, called Pay Per Crawl. This model is now evolving into a "Pay Per Use" system, allowing publishers to charge AI companies not only for content retrieval but also for the added value generated.

This approach could also help preserve publishers' bandwidth and computing resources, as Cloudflare's data shows that over 50% of AI bot traffic concerns pages that have not changed.

Strategic Partnerships for Initial Deployment

To implement these changes, Cloudflare is initially collaborating with Ceramic.ai and You.com. Publishers who choose to participate in this program will receive compensation when their content appears in AI search results from Ceramic or when You.com accesses their premium content.

Cloudflare specifies that other AI companies will be able to adapt this model according to their specific needs, thus offering flexibility in how content is monetized and utilized.