Cloudflare CEO urges UK regulator to break Google’s link between search and AI crawlers

Cloudfare

Cloudflare chief executive Matthew Prince has urged the UK’s Competition and Markets Authority (CMA) to impose stricter rules on Google, arguing that its control over search and AI gives it an unfair competitive edge. Prince, who is in London for discussions with the regulator, says Google’s dual use of its web crawler for both search and AI development effectively locks out competition and undermines content creators.

The CMA recently classified Google as having a “substantial and entrenched” position in online search and advertising. This designation enables the watchdog to extend its scrutiny to other areas, such as Google’s AI Overviews, AI Mode, Discover feed, and News tab. Prince believes this opens the door to much-needed reforms, especially as Google integrates AI deeper into its services.

Speaking at the Bloomberg Tech conference in London, Prince said Cloudflare is well placed to comment because, while it serves most AI companies through its network, it doesn’t directly compete in the AI space. “We don’t have a dog directly in the fight,” he said. “We’re not an AI company or a media publisher. But we’re the network that connects them—80% of AI companies are our customers.”

Prince argues that Google uses its long-established web crawler, Googlebot, to gather data for both Search and AI applications. This, he claims, allows Google to bypass the licensing costs that AI-focused companies like OpenAI or Anthropic would otherwise face. He contends that site owners are effectively forced to allow Google access to their content, as opting out of its AI crawler also removes them from Search, cutting off significant traffic and ad revenue.

According to Prince, this arrangement gives Google an unfair advantage and stifles competition. “Google acts as if it has a right to all the content in the world, even if it doesn’t pay for it,” he said. “It tells publishers that if they want to block AI, they must also block search, which isn’t feasible for anyone who depends on online visibility.”

Prince warned that blocking Google’s crawler also disrupts its ad safety systems, halting advertisements across multiple platforms. This bundling, he said, ensures Google retains access to data that rivals must pay for, effectively cementing its market dominance.

He believes the solution lies in creating an open marketplace where thousands of AI firms can buy content directly from publishers and small businesses, fostering genuine competition. Cloudflare has already supplied the CMA with technical data showing how Google’s crawler operates and why replicating its reach is nearly impossible for smaller players.

Prince’s comments echo those of Neil Vogel, CEO of Dotdash Meredith, who last month accused Google of being a “bad actor” for forcing publishers to participate in its AI data collection. Together, their remarks add pressure on regulators to act against what many see as a deeply unbalanced system.