Cloudflare to block AI crawlers by default, introduces ‘Pay Per Crawl’ for publishers

Cloudflare has announced it will now block AI web crawlers by default, escalating efforts to protect online content from unauthorised scraping. The move, revealed on Tuesday, aims to ensure creators are properly credited and compensated as generative AI tools increasingly draw from the open web.

With the new policy, all new websites onboarded to Cloudflare will automatically block known AI bots — even those that don’t respect robots.txt protocols. The company identifies such bots using its own internal list and now allows domain owners to choose whether they want to grant access to AI scrapers during setup.

In a notable shift, Cloudflare is also piloting a “Pay Per Crawl” initiative. This lets select publishers set a price for AI companies that want to crawl their content. Interested AI firms can view these rates and decide whether to proceed, potentially creating a commercial framework for AI training data access.

The program is currently limited to a group of major publishers and content creators, including The Atlantic, Fortune, Stack Overflow, and Quora — all of whom support tighter control over AI crawlers. These sites, like many others, are grappling with declining traffic as users increasingly get answers directly from AI tools instead of traditional search engines.