Tue, 01 Jul 2025

Tue, 01 Jul 2025 Millions of websites to get 'game-changing' AI bot blocker

Publishers including Condé Nast and Sky News have welcomed the new tech from internet infrastructure firm, Cloudflare.
Cloudflare, a firm that hosts around one-fifth of the internet, has developed a system to block artificial intelligence (AI) bots from accessing websites without permission. The system targets AI firm crawlers that collect data and train systems on website content without paying or asking for permission. Websites such as Sky News, The Associated Press, and Buzzfeed will now be able to use Cloudflare's technology to prevent AI bots from scraping their content. This is a significant move as it aims to protect publishers and creators from having their work used by AI firms without permission or compensation. Cloudflare is also developing a "Pay Per Crawl" system that would allow websites to request payment from AI companies for accessing their original content. The company argues that this is necessary because AI crawlers collect data and generate answers without sending visitors to the original source, depriving creators of revenue. The move has been welcomed by publishers and content creators who have accused AI firms of using their work without permission or payment. However, some experts argue that stronger legal protections are still needed to protect creative rights. Cloudflare's technology is currently active on over a million websites, and the company claims it can handle up to 50 billion requests per day from AI crawlers. The system uses a "Labyrinth" of web pages filled with junk data to deter worst offenders. The move has sparked debate about the need for stronger protections for creative industries and the role of technology in preventing content theft. Some experts believe that only stronger laws can provide adequate protection, while others argue that Cloudflare's technology is a step in the right direction.
  >>