Could Cloudflare’s “pay per crawl” save news from AI?

AI is eating the news.

Publishers large and small are bracing for a grim reality that is starting to reveal itself: as readers increasingly turn to AI chatbots like ChatGPT for their queries, or skim over Google AI overviews, news publishers are seeing visits from search engines drop off a cliff.

AI companies have slurped up billions of web pages to train their models and fetch query results. Big publishers like The New York Times have filed suit against OpenAI and Microsoft, accusing the companies of stealing its content without permission or compensation. Many publishers have opted to cut deals with AI companies to license their work and appear in results.

While many of today’s chatbots surface citations with links in query responses, it generates a fraction of the traffic that traditional search engine results saw (and that was already in decline).

An unlikely company is stepping in with a novel solution to this problem that could provide a way for AI companies to crawl a publisher’s website with permission and pay for the access.

Cloudflare is a content delivery network — it ensures that customers’ websites, images, and videos will be accessible quickly around the world, sitting between the publisher and the web traffic hitting its site. That gives Cloudflare the unique ability to control who gets to see the content that it’s distributing. And while individual website owners can try to block AI bots from scraping their sites, Cloudflare can do it for billions of web pages at a time across 125 countries. The company says it serves about 20% of the web.

Cloudflare is introducing an experiment that it’s calling “pay per crawl,” which acts as a gatekeeper (and a toll booth) for AI bots crawling the web. Here’s how it works:

• Cloudflare detects traffic that is coming not from a human user, but from an AI crawler.

• Depending on a publisher’s choice, the “pay per crawl” system controls access to the site.

• The AI bot can be allowed to access the site for free, it can pay to access the site, or it can be blocked altogether. Publishers can also tailor this to specific companies.

• Cloudflare collects a micropayment from the AI bot, which it passes along to the publisher.

The pay per crawl plan is currently in private beta, and the company has also announced that all new Cloudflare customers will be set to block AI bots by default.

Cloudflare cofounder and CEO Matthew Prince wrote in a blog post declaring “Content Independence Day”:

“Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value. That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.”

The rub is that both AI companies and publishers need to opt in to the plan for payments to be processed, but several big publishers have signed up, including Condé Nast, Time, Associated Press, and The Atlantic, according to TechCrunch.