Cloudflare is changing how AI crawlers reach your site

Update, 10 September: where things stand

Five days out, we have found no announcement of a delay. Cloudflare still lists Googlebot, Bingbot and Applebot as multi-purpose crawlers and treats them by their most restrictive job, so the Google trap described below still applies.

Publishers are already drawing the line this guide recommends. In 4,223 robots.txt files from Cloudflare Radar's 31 August snapshot, OpenAI's training crawler GPTBot was blocked 2.33 times for every time it was allowed, while its search crawler OAI-SearchBot was allowed slightly more often than blocked. Anthropic's and Apple's crawlers showed the same pattern: training crawlers blocked far more often than the ones that fetch pages for answers.

We will update this page again after 15 September with what actually happened.

What actually changes

Cloudflare is splitting AI crawlers into three kinds and treating them differently: search crawlers, which index you so you can appear in results; training crawlers, which collect content to train models; and agent crawlers, which fetch a page because someone asked an assistant to go and look at it.

From 15 September, on pages that carry advertising, the default becomes: search allowed, training blocked, agent blocked. Cloudflare has not published how it decides that a page displays ads. Its stated reasoning is that an ad signals a page meant for people to visit, so this is aimed at publishers who sell ad space, not at business websites generally.

Who this applies to

Cloudflare's own announcement and changelog say the new defaults apply to new domains onboarding from 15 September. Some coverage reports that existing free-plan sites are included too. Cloudflare's published text does not say that, but it does say every customer can opt out before the date.

So the safe reading for a small business is this: if your site is on Cloudflare's free plan, check the setting rather than assuming you are exempt. It takes two minutes, and nobody reads the emails their DNS provider sends.

The mistake that costs you Google

This is the part worth understanding, and it is the opposite of what the headlines suggest.

Googlebot, Bingbot and Applebot are multi-purpose. The same crawler that indexes you for search also feeds AI features. Cloudflare judges a crawler that will not declare which job it is doing on a given request by its most restrictive behaviour.

So a site owner who reads about this, panics, and switches on a broad block of AI training crawlers can knock out Google and Bing at the same time. The damage is not that an AI assistant stops quoting you. It is that you disappear from search entirely, and it will not be obvious why.

If you change one setting because of this article, change it deliberately and check what else it catches. Blocking training crawlers broadly is the single most expensive mistake available here.

Which of the three cases are you in?

Almost every site falls into one of these. Finding out which takes a couple of minutes.

Should you allow AI crawlers at all?

For a publisher earning money from advertising, blocking training crawlers is a reasonable commercial decision: the content is the product and models are consuming it without sending anyone back.

For almost every other business, blocking is self-harm. An analysis of major news sites found 79% blocking training crawlers, and 71% also blocking the retrieval crawlers, which is what removes them from AI answers entirely. Blocking the search and agent crawlers is what stops an assistant recommending you. Blocking the training crawlers stops future models learning your business exists at all.

Our own position, for what it is worth: we allow every crawler that identifies itself, and our robots file names each one deliberately rather than relying on a wildcard. You can read it and compare it with yours.

What to do this week

• Find out whether your site is behind Cloudflare. Your developer knows, or a DNS lookup answers it in seconds.
• If it is, open Cloudflare and find the AI crawler controls. Note what they currently say, before the date rather than after.
• Decide what you actually want, rather than accepting a default chosen for publishers. For most businesses that means allowing search and agent crawlers.
• Check your robots.txt separately. Cloudflare is not the only place a crawler can be blocked, and we regularly find sites blocking AI crawlers there without knowing it.
• Confirm your pages are readable without JavaScript. No major AI crawler runs it, so a site that renders in the browser only is invisible to them regardless of any setting.

After 15 September

This guide stops being urgent once the date passes, but two things outlast it. The three-way split of AI crawlers into search, training and agent is how the whole industry is heading, and knowing which you allow is a permanent question. And Cloudflare's pay-per-crawl work points at a future where access to your content is priced rather than assumed.

We will update this page when that changes rather than leaving it to age.