Cloudflare to block AI crawlers by default, introduces ‘Pay Per Crawl’ for publishers

Cloudflare

Cloudflare has announced it will now block AI web crawlers by default, escalating efforts to protect online content from unauthorised scraping. The move, revealed on Tuesday, aims to ensure creators are properly credited and compensated as generative AI tools increasingly draw from the open web.

With the new policy, all new websites onboarded to Cloudflare will automatically block known AI bots — even those that don’t respect robots.txt protocols. The company identifies such bots using its own internal list and now allows domain owners to choose whether they want to grant access to AI scrapers during setup.

In a notable shift, Cloudflare is also piloting a “Pay Per Crawl” initiative. This lets select publishers set a price for AI companies that want to crawl their content. Interested AI firms can view these rates and decide whether to proceed, potentially creating a commercial framework for AI training data access.

The program is currently limited to a group of major publishers and content creators, including The Atlantic, Fortune, Stack Overflow, and Quora — all of whom support tighter control over AI crawlers. These sites, like many others, are grappling with declining traffic as users increasingly get answers directly from AI tools instead of traditional search engines.

Cloudflare’s CEO, Matthew Prince, argued that unchecked scraping undermines content creators. “Original content is what makes the internet great. We have to protect it,” he said. In March, Cloudflare also introduced an “AI Labyrinth” — a technical deterrent to slow down bot activity from non-compliant crawlers.

Cloudflare plans to work with AI firms to certify their crawlers and publicly declare how they intend to use scraped content — whether for training, inference, or search.