Cloudflare cofounder and CEO Matthew Prince (Noam Galai/Getty Images)

You down with PPC

Could Cloudflare’s “pay per crawl” save news from AI?

The novel plan would let publishers control AI bot access and collect micropayments to access content.

7/1/25 1:02PM

AI is eating the news.

Publishers large and small are bracing for a grim reality that is starting to reveal itself: as readers increasingly turn to AI chatbots like ChatGPT for their queries, or skim over Google AI overviews, news publishers are seeing visits from search engines drop off a cliff.

AI companies have slurped up billions of web pages to train their models and fetch query results. Big publishers like The New York Times have filed suit against OpenAI and Microsoft, accusing the companies of stealing its content without permission or compensation. Many publishers have opted to cut deals with AI companies to license their work and appear in results.

While many of today’s chatbots surface citations with links in query responses, it generates a fraction of the traffic that traditional search engine results saw (and that was already in decline).

An unlikely company is stepping in with a novel solution to this problem that could provide a way for AI companies to crawl a publisher’s website with permission and pay for the access.

Cloudflare is a content delivery network — it ensures that customers’ websites, images, and videos will be accessible quickly around the world, sitting between the publisher and the web traffic hitting its site. That gives Cloudflare the unique ability to control who gets to see the content that it’s distributing. And while individual website owners can try to block AI bots from scraping their sites, Cloudflare can do it for billions of web pages at a time across 125 countries. The company says it serves about 20% of the web.

Cloudflare is introducing an experiment that it’s calling “pay per crawl,” which acts as a gatekeeper (and a toll booth) for AI bots crawling the web. Here’s how it works:

Cloudflare detects traffic that is coming not from a human user, but from an AI crawler.
Depending on a publisher’s choice, the “pay per crawl” system controls access to the site.
The AI bot can be allowed to access the site for free, it can pay to access the site, or it can be blocked altogether. Publishers can also tailor this to specific companies.
Cloudflare collects a micropayment from the AI bot, which it passes along to the publisher.

The pay per crawl plan is currently in private beta, and the company has also announced that all new Cloudflare customers will be set to block AI bots by default.

Cloudflare cofounder and CEO Matthew Prince wrote in a blog post declaring “Content Independence Day”:

“Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value. That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.”

The rub is that both AI companies and publishers need to opt in to the plan for payments to be processed, but several big publishers have signed up, including Condé Nast, Time, Associated Press, and The Atlantic, according to TechCrunch.

South by Southwest Conference and Festivals

Gold Tesla Cybercabs are piling up, but they’re not picking up passengers yet

Low-volume production started in April. Now people are noticing them more and more in the wild.

Rani Molla14h

Millie Giles

TAKING ACCOUNTS

16h

Britain announces social media ban for under-16s starting early 2027

The UK government plans to use the same model for the restrictions as Australia — but how successful has that case study been so far?

UK Prime Minister Announces Under-16s Social Media Ban

Jon Keegan16h

Anthropic pulls Fable and Mythos access worldwide after Trump administration bars their use by foreign nationals

Only days after releasing two versions of its next-gen AI model, Anthropic has disabled them for users worldwide.

Anthropic says it received a Friday night order from the Trump administration to suspend access to the models for any foreign national (anywhere in the world) — a group that included some Anthropic employees. In response, the company turned off access to everyone.

Last week, the company released to the public its much-anticipated Claude Fable 5 model (and its restricted version Claude Mythos 5, which is still being tested with trusted partners). Anthropic said in a blog post announcing the action that officials cited national security concerns with the new models, while offering few specific details.

The post said that the government gave the company “verbal evidence of a potential narrow, non-universal jailbreak” of the public Fable 5 model. A jailbreak is a means by which users can evade restrictions built into the code to unlock prohibited functionality. Anthropic downplayed the significance of the attack, and said other major models, such as OpenAI’s GPT-5.5, could also be affected by the technique described.

Fears of these first Mythos-class models being misused are running high, after Anthropic warned the cybersecurity world in May that the advanced cyber capabilities of Mythos have rapidly discovered thousands of vulnerabilities in ubiquitous software, leading to the decision to restrict the full version of the model to a close group of trusted partners for testing.

This morning, Axios reported that Anthropic technical staff have flown to Washington to meet with White House officials to resolve the issue.

The Wall Street Journal is reporting that the Trump administration’s decision to take action against Anthropic was prompted by discussions that Amazon CEO Andy Jassy had with officials, including Treasury Secretary Scott Bessent. According to the report, Amazon researchers said they had been able to evade some of Fable 5’s security restrictions using specific prompts. Amazon is a major investor in Anthropic.

Anthropic is currently suing the US government to fight the Pentagon’s blacklisting of the company on national security grounds.

Statement on the US government directive to suspend access to Fable 5 and Mythos 5

This morning, Axios reported that Anthropic technical staff have flown to Washington to meet with White House officials to resolve the issue.

Anthropic is currently suing the US government to fight the Pentagon’s blacklisting of the company on national security grounds.

Rani Molla19h

Tesla used skewed data in push for European FSD approval, Reuters finds

Tesla has used highly questionable safety stats in an effort to win over European regulators and rekindle sales in the region, according to a Reuters investigation.

Tesla reportedly pitched regulators in Sweden and the Netherlands with claims that its Full Self-Driving (FSD) tech is over 7x safer than human drivers. However, independent researchers told Reuters that the stats are misleading because Tesla compares airbag-deployment crashes involving FSD-equipped vehicles with much broader US crash statistics, while also benchmarking newer Teslas against the entire US vehicle fleet, which is significantly older on average.

Despite the flawed metrics, the Dutch regulator approved FSD in April, saying its decision was based on its own “tests, analyses and verifications,” and Tesla is now pushing for EU-wide clearance. A version of FSD is currently available in five European markets.

Exclusive: Tesla presented misleading ‘Full Self-Driving’ safety data to European regulators

Rani Molla6/12/26

Report: Microsoft weighs Xbox spin-off amid major overhaul

Microsoft is reportedly considering spinning out or restructuring its struggling Xbox unit, per The Information. While new Xbox CEO Asha Sharma, who took over in February, is preparing for layoffs, she’s simultaneously planning to boost investment in its biggest franchises like “Halo,” “Fallout,” and “Minecraft.”

The latest potential shake-up comes as the gaming division battles major headwinds, following a massive 33% plunge in Q3 console sales and a recent move to slash Game Pass prices while removing new “Call of Duty” titles.