Cloudflare Reveals Anthropic as the Biggest Web Predator

Cloudflare Reveals the Practices of AI Giants

Cloudflare, a major player in web security and performance management, has recently published data on the crawl requests made by artificial intelligences (AIs). This information sheds light on how these companies interact with the web, particularly whether they take more than they give back. Among the companies studied, one stands out significantly for its behavior.

Every quarter, Cloudflare provides insights into how leading AI companies exploit the web. The latest statistics reveal a persistent trend: the intensive exploitation of web resources by these giants.

Anthropic Leads the Explorations

For the period from July 1 to July 7, Cloudflare's data shows that Anthropic, an AI company, stands out with its high crawl-to-refer ratio. Anthropic's bots crawled approximately 2,800 pages for each reference returned, a figure that places it far ahead of its competitors.

OpenAI, another major player, follows at a distance, with Perplexity in third place. Microsoft and Google come next, but with a significant gap compared to the top three.

DuckDuckGo, a Model of Balance

Among the companies analyzed, DuckDuckGo distinguishes itself by its ability to maintain an almost fair balance between what it takes from the web and what it gives back. According to Cloudflare, DuckDuckGo performs three scrapes for each reference, a ratio that sharply contrasts with that of Anthropic.

A Relative Improvement for Anthropic

Although Anthropic has improved its ratio from an impressive 8,800 to 1 recorded in early April, the company continues to scrape intensively. For instance, in early May, its bots crawled websites 24,700 times for each reference. This relative improvement may be due to a seasonal decrease in activity.

The Impact on the Web Economy

Cloudflare's metric is crucial for assessing whether AI giants respect the traditional economic agreement of the web. This agreement is based on the principle that sites allow free crawling in exchange for traffic, a model now threatened by the rise of generative AI.

AI response engines and chatbots now provide direct answers, thereby reducing the need for users to visit source sites. This dynamic diminishes the incentive to produce quality content on the web.

The Contradictions of Anthropic

Anthropic presents itself as an ethical AI company, but Cloudflare's data raises questions about this ethics. The company has criticized its rivals for using the results of its models to enhance their own products, labeling this as "distillation attacks."

However, this criticism could apply to Anthropic itself, which collects web content, often without the consent of the owners, to develop its products. This situation illustrates the complexities of the modern web, where data collection practices are often controversial.