Cloudflare launches one-click button to block AI bots
Great news for content creators, Cloudflare has unveiled a new feature that enables users to block all AI bots with a single click. This update is available to all customers, including those on the free tier, marking a significant step in preserving the integrity of original content on the internet.
“We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” Cloudflare announced in its blog post,
The surge in generative AI has heightened the demand for content used to train models and run inferences, often through web scraping. While some AI companies transparently identify their bots, others operate under the radar, scraping content without permission. Just recently, Perplexity made headlines for impersonating legitimate visitors to scrape website content.
The announcement comes after Cloudflare introduced a similar feature last year to block compliant AI bots that respect website protocols and don't use unlicensed content. The networking giant notes many Cloudflare customers opted to block these bots hence the decision to expand on the capability.
The new version allows users to block all AI bots, including those that operate dishonestly. Customers can activate this feature in the Security > Bots section of the Cloudflare dashboard by toggling the “AI Scrapers and Crawlers” option.
Cloudflare's decision comes amid growing concern about the volume of AI bot traffic. The company’s data shows that Bytespider, Amazonbot, ClaudeBot, and GPTBot are the top four AI crawlers, with ByteDance's Bytespider leading in both request volume and frequency of being blocked. These bots are primarily used for training large language models and indexing content for various AI applications.
In June, AI bots accessed approximately 39% of the top one million internet properties using Cloudflare, yet only 2.98% of these properties took measures to block or challenge the requests. This discrepancy highlights the need for more robust tools to manage AI bot activity.
Cloudflare’s advanced machine learning models have been instrumental in identifying and blocking AI bots, even those attempting to disguise themselves as legitimate browsers. The company’s global network, processing over 57 million requests per second, enables it to detect and respond to new scraping tools and techniques swiftly.
