Cloudflare: Bots Will Dominate the Web by 2027

Bots Set to Dominate the Web

Matthew Prince, the CEO of Cloudflare, recently shared a bold prediction at the SXSW conference in Austin. According to him, traffic generated by bots on the Internet will surpass that of humans by 2027. This assertion is based on the observation of the rapid evolution of artificial intelligence, which is driving an unprecedented increase in online bot activity.

Prince explained that the proliferation of bots is closely linked to the rise of generative AI. These bots have the capability to browse a significantly larger number of websites to provide answers to user queries. For example, when a user searches for a digital camera, they might visit five websites. In contrast, a bot could check up to 5,000 sites, generating a considerable volume of traffic that current infrastructures must manage.

The Evolution of Bot Traffic

Before the advent of generative AI, bot traffic accounted for about 20% of total Internet traffic. Google's crawler was the main contributor to this figure. However, aside from a few legitimate bots, most were used for malicious purposes. With the increasing data demands of generative AI, Prince anticipates that by 2027, bot traffic will exceed that of humans.

This transformation of the digital landscape will require the development of new technologies. Prince mentions the need to create "testing environments" for AI agents, which could be generated and deleted on demand. These environments would allow consumers to delegate complex tasks to AI agents, such as planning vacations, more efficiently. He envisions a time soon when millions of these "testing environments" for agents will be created every second.

Infrastructure and Technical Challenges

The increased use of bots will demand a robust infrastructure, including data centers and servers capable of handling this load. Prince recalled that during the Covid pandemic, Internet traffic experienced explosive growth, putting a strain on certain parts of the network, particularly among streaming giants like YouTube, Disney, and Netflix.

Unlike this sudden surge, the current growth in traffic is more gradual but ongoing. Prince emphasizes that there are currently no factors likely to slow this trend. This situation poses a major challenge for existing infrastructures, which must adapt to this new reality.

Cloudflare on the Front Lines

For Cloudflare, this evolution represents a strategic opportunity. The company specializes in optimizing website availability, loading speed, and protection against cyberattacks. Among its services are a content delivery network, advanced security solutions, and protections against DDoS attacks. Cloudflare also offers tools that allow businesses to filter out unwanted bot traffic. Notably, its "Always Online" technology enables serving cached versions of websites when the main server fails or disconnects.

Cloudflare's infrastructure and security services are used by one-fifth of all websites, giving it a privileged position to observe and adapt to the rapid changes on the web. Prince compares the impact of AI to a platform shift, similar to the transition from desktop to mobile. He believes that AI will radically transform the way information is consumed on the Internet.