Get the hottest Fintech Singapore News once a month in your Inbox
Cloudflare has introduced a new default setting that blocks AI crawlers from accessing website content without explicit permission or compensation, marking a major shift in how online content is protected from unauthorised use.
This makes Cloudflare the first major internet infrastructure provider to enforce such blocking by default.
The company said the change gives website owners more control by letting them decide whether to allow access and how their content can be used.
AI companies are now expected to identify their crawlers and declare whether they are used for training, inference, or search, giving site owners more information to make that decision.
The move comes amid growing concerns over AI models scraping web content without attribution or fair compensation.
While traditional search engines typically generate traffic and ad revenue for sites, AI-generated responses often bypass the original source, leaving creators without credit or income.
Cloudflare described this shift as a threat to the sustainability of the internet’s content ecosystem.
The company had previously launched a one-click AI crawler block feature in September 2024, which has since been enabled by more than one million customers.
Now, all new domains that sign up with Cloudflare will be asked whether they want to allow AI crawlers, shifting the default to a permission-based model.
Several publishers and technology companies have expressed support for the initiative, including Condé Nast, Dotdash Meredith, Gannett Media, Reddit, TIME, Pinterest, and others.
These organisations have called for clearer boundaries and fairer arrangements when their content is used by AI platforms.
Cloudflare is also contributing to the development of new protocols to help AI bots authenticate themselves and allow websites to better manage which crawlers are permitted.
The company said these tools aim to create a more sustainable model for content creators while still supporting AI innovation.
Matthew Prince
“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself. Original content is what makes the Internet one of the greatest inventions in the last century, and it’s essential that creators continue making it.
AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”
said Matthew Prince, Co-founder and CEO of Cloudflare.
Steve Huffman
“AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what.
The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare’s efforts are a step in the right direction for everyone.”
said Steve Huffman, Co-founder and CEO of Reddit.
Welcome Cloudflare: AI week 2025: Innovate with AI without compromising security
Featured image: Edited by Fintech News Singapore, based on image by ar_fp via Freepik
