Cloudflare has introduced a new tool designed to block AI bot crawlers from accessing website content without permission or payment, aiming to help site owners monetize their work amid growing concerns over content scraping by AI firms. Announced on Tuesday, the tool enables publishers to decide whether to allow AI crawlers on their sites and to set a price through a “pay per crawl” system, giving them more control over how their content is used and compensated.

As AI crawlers increasingly collect content without driving traffic back to the original sites, website owners have seen a decline in referral traffic that once generated significant ad revenue. Major publishers such as Condé Nast and the Associated Press, along with platforms like Reddit and Pinterest, are backing the initiative.

Stephanie Cohen, Chief Strategy Officer at Cloudflare, said the goal is to empower publishers and maintain a sustainable digital ecosystem. “Traffic patterns have shifted quickly, and a change was necessary,” Cohen stated. “This marks the beginning of a new internet model.”

Cloudflare cited internal data showing that Google's crawl-to-referral ratio has worsened, dropping from 6:1 to 18:1 in just six months—meaning more content is being indexed without a corresponding increase in traffic. Google's AI Overviews may be part of the reason, as users increasingly find answers directly in search results. Other AI companies, such as OpenAI, show an even starker ratio—1,500:1—highlighting how little traffic is being sent back to content creators.

For years, search engines have rewarded content creators by indexing and linking back to their sites. But AI crawlers now disrupt this model by harvesting content to train chatbots like ChatGPT, often without proper credit or compensation. Some AI firms are also bypassing standard protocols intended to block such scraping, while claiming their practices are legal.

In response, publishers have taken varied approaches—some, like The New York Times, have pursued legal action over copyright infringement, while others have signed licensing deals. Reddit, for instance, has sued Anthropic for allegedly scraping user comments to train its AI, while also entering into a content agreement with Google.