Header Image

Cloudflare Launches Tool to Block AI Crawlers and Help Publishers Monetize Content

Cloudflare Launches Tool to Block AI Crawlers and Help Publishers Monetize Content

Last Updated Jul - 02 - 2025, 03:17 PM | Source : Fela News

Cloudflare unveils a "pay per crawl" tool to stop AI bots from accessing content without permission, giving publishers control and revenue options. Supported by
Cloudflare Launches Tool to Block AI Crawlers
Cloudflare Launches Tool to Block AI Crawlers

Cloudflare has introduced a new tool designed to block AI bot crawlers from accessing website content without permission or payment, aiming to help site owners monetize their work amid growing concerns over content scraping by AI firms. Announced on Tuesday, the tool enables publishers to decide whether to allow AI crawlers on their sites and to set a price through a “pay per crawl” system, giving them more control over how their content is used and compensated.

As AI crawlers increasingly collect content without driving traffic back to the original sites, website owners have seen a decline in referral traffic that once generated significant ad revenue. Major publishers such as Condé Nast and the Associated Press, along with platforms like Reddit and Pinterest, are backing the initiative.

Stephanie Cohen, Chief Strategy Officer at Cloudflare, said the goal is to empower publishers and maintain a sustainable digital ecosystem. “Traffic patterns have shifted quickly, and a change was necessary,” Cohen stated. “This marks the beginning of a new internet model.”

Cloudflare cited internal data showing that Google's crawl-to-referral ratio has worsened, dropping from 6:1 to 18:1 in just six months—meaning more content is being indexed without a corresponding increase in traffic. Google's AI Overviews may be part of the reason, as users increasingly find answers directly in search results. Other AI companies, such as OpenAI, show an even starker ratio—1,500:1—highlighting how little traffic is being sent back to content creators.

For years, search engines have rewarded content creators by indexing and linking back to their sites. But AI crawlers now disrupt this model by harvesting content to train chatbots like ChatGPT, often without proper credit or compensation. Some AI firms are also bypassing standard protocols intended to block such scraping, while claiming their practices are legal.

In response, publishers have taken varied approaches—some, like The New York Times, have pursued legal action over copyright infringement, while others have signed licensing deals. Reddit, for instance, has sued Anthropic for allegedly scraping user comments to train its AI, while also entering into a content agreement with Google.

Share :

Trending this week

Thackeray Cousins’ Possible Reunion Heats Up Maharashtra Politics

Jul - 14 - 2025

As the Brihanmumbai Municipal Corporation (BMC) e... Read More

MCD Plans to Outsource Management of 300 Community Centres Across Delhi Under PPP Model

Jul - 14 - 2025

The Municipal Corporation of Delhi (MCD) is consi... Read More

WHO Places Saima Wazed on Leave Amid Corruption Probe

Jul - 14 - 2025

The World Health Organization (WHO) has placed Saima Wazed, daughter of former Bangladesh Prime Minister Sheikh Hasina, on leave following corrupti... Read More

Omar Abdullah Scales Wall to Pay Tribute at Srinagar Martyrs’ Memorial Amid Security Curbs

Jul - 14 - 2025

On Monday, Jammu and Kashmir Chief Minister Omar Abdullah bypassed security barricades and scaled the boundary wall of the Mazar-e-Shu... Read More

Hospital Attender Booked for Misbehaving with Woman in Hyderabad Ward

Jul - 14 - 2025

An attender at Durgabai Deshmukh Hospital in Hyderabad has been booked for allegedly behaving inappropriately with a woman who was att... Read More