Cloudflare Launches Content Signals Policy To Fight AI Crawlers and Scrapers

BrianFagioli shares a report from NERDS.xyz: Cloudflare has unveiled the Content Signals Policy, a free addition to its managed robots.txt service that aims to give website owners and publishers more control over how their content is accessed and reused by AI companies. The idea is pretty simple: robots.txt already lets site operators specify which crawlers can enter and where. Cloudflare's new policy adds a layer that signals how the data may be used once accessed, with plain-language terms for search, AI input, and AI training. 'Yes' means allowed, 'no' means not allowed, and no signal means no preference.

Matthew Prince, Cloudflare's co-founder and CEO, said: 'The Internet cannot wait for a solution, while in the meantime, creators' original content is used for profit by other companies. To ensure the web remains open and thriving, we're giving website owners a better way to express how companies are allowed to use their content.' Cloudflare says more than 3.8 million domains already use its robots.txt tools to signal they don't want their content used for AI training. Now, the Content Signals Policy makes those preferences clearer and potentially enforceable. Further reading: Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers

Read more of this story at Slashdot.