Cloudflare Lets Sites Disallow AI Training Without Blocking Googlebot

Cloudflare's AI Training Opt-Out Still Runs on the Honor System

Cloudflare rolled out a new "Disallow AI Training" setting on September 15, 2026, replacing a plan it had floated in July that would have blocked Googlebot, Applebot and Bingbot outright whenever a site opted out of AI training. Under the new default, those three crawlers — plus any other crawler Cloudflare designates "Accountable" — keep full access for search, while training-only crawlers from the same companies get blocked. A separate "Block" setting still exists for publishers who want mixed-use crawlers gone entirely, including from search. Cloudflare says it created the Accountable label after talks with crawler operators that began in July, and that Amazon, Anthropic, Google, Meta, Microsoft and OpenAI all qualify because each now runs, or has committed to running, separate crawlers for search and for training, per Search Engine Journal.

What does "Accountable" actually certify?

A promise made to Cloudflare by the crawler operator, not an independent audit of that operator's actual behavior.

The label requires four things: a robots.txt-style training opt-out, an opt-out from AI summaries (live for some operators now, with the rest promised through 2027 by Cloudflare's account), URL-level reporting on what content was used for training, and an assurance that opting out of training won't hurt search ranking. Google, Apple and Microsoft reportedly meet the bar today; the others are counted Accountable on the strength of commitments with deadlines still ahead. That's a workable compromise for keeping search traffic flowing, but the label describes a relationship between Cloudflare and the crawler operator — not something a publisher can verify on their own logs.

Does this actually stop AI systems from using your content?

No — it publishes a preference in robots.txt, which compliant crawlers honor voluntarily and non-compliant ones can simply ignore.

For Google specifically, Disallow AI Training writes a Disallow rule for the Google-Extended token, which Google's own documentation says has no effect on inclusion in Search or on ranking. That's a different lever from the Search Console setting that controls whether a page can appear in AI Overviews, AI Mode or Discover's generative features. Publishers who assume "blocked AI training" covers both are likely to be surprised the next time their text turns up quoted in an AI Overview anyway.

What should you ask your CDN or robots.txt vendor now?

Ask exactly which crawlers move automatically, which stay blocked, and whether Accountable status can be revoked if a company misses its deadline.

Cloudflare says existing Block or Block on pages with ads selections migrate automatically to Disallow AI Training, and the old Block AI Bots toggle and Managed Robots.txt feature are being retired. If you'd set granular rules through either of those, check what you actually ended up with post-migration — the default outcome is more crawler access, not less, which is the opposite of what publishers who originally flipped "block AI training" were trying to achieve.

Frequently asked questions

Does this change how AI Overviews cite my site?

No. AI Overviews, AI Mode and Discover's generative features are governed by a separate Search Console setting, distinct from the Google-Extended training opt-out.

Which crawlers stay blocked under Disallow AI Training?

Training-only crawlers from Amazon, Anthropic, Meta and OpenAI remain blocked; their separate search crawlers, plus Googlebot, Applebot and Bingbot, keep access.

What if I want mixed-use crawlers gone entirely?

Choose Cloudflare's Block setting, which now stops Googlebot, Applebot and Bingbot completely, including for search — a visibility tradeoff worth weighing first.

Source: Search Engine Journal.