Cloudflare's "Disallow AI Training" Setting Won't Take You Out of Google's AI Overviews
On September 15, 2026, Cloudflare changed how its AI crawler controls work. Its one-click "Block AI Bots" switch is being deprecated in favor of separate Search, Training, and Agent controls, and a new Training setting called Disallow AI Training arrived with it. The email I got from Cloudflare says the dashboard update rolls out "over the coming week." It also says "Most customers won't need to change anything," and that if you had the old switch on, "it's worth a quick look at your new Search, Training, and Agent settings."
I took that look, with Cloudflare's, Google's, and Bing's own pages open beside it. Google's AI Overviews are controlled from a different dashboard than the Training row, and the Bing option Cloudflare points to costs you something inside Microsoft's Copilot assistant. Block, one of the old words on that page, now reaches Google's search crawler too.
The September 15 change
When your site's traffic runs through Cloudflare, Cloudflare sits between your website and whatever visits it. If a developer or your host set up your site, that may be the case even if you have never logged in to Cloudflare yourself.
The programs that read your pages are called crawlers. Some build a search index and some collect text to train AI models. Others only show up because a person asked a chatbot about your page.
Cloudflare's post frames the problem around one crawler doing two of those jobs at once. It calls that a mixed-use crawler and names three: Google's Googlebot, Microsoft's Bingbot, and Apple's Applebot. Cloudflare's old Block setting did not apply to those three, since "blocking them could also affect search discoverability."
Your robots.txt is a plain text file at the root of your site, like www.yourdomain.com/robots.txt, that tells crawlers what they may read. Every audit I run pulls a copy of it. It names crawlers by labels called tokens, and Google and Apple each give you a separate token for refusing training: Google-Extended and Applebot-Extended.
According to Cloudflare's August post, the Search, Training, and Agent rows have been in the dashboard since July 1, 2026. Each one covers a single behavior you can allow or refuse:
| Row | What Cloudflare says it covers |
|---|---|
| Search | Crawling to build a search index |
| Training | Crawling to train or fine-tune a model |
| Agent | "User-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents" |
And these are the settings you can pick:
| Setting | What Cloudflare says it does |
|---|---|
| Allow | Everything gets in unless another setting or a firewall rule (a custom blocking rule you write in Cloudflare) stops it |
| Disallow AI Training | Training row only. Cloudflare's Bot Preference Sync feature publishes a no-training preference in your robots.txt. The three named mixed-use crawlers stay allowed for search, and every other training crawler is blocked, including the training-only crawlers from OpenAI, Anthropic, Amazon, and Meta |
| Block on pages with ads | Crawlers are refused on pages Cloudflare detects serving an ad, and Cloudflare says this now reaches the three named mixed-use crawlers too |
| Block | Everything in the row is refused, the three named mixed-use crawlers included |
Cloudflare's August post says Bot Preference Sync would be on by default if you're a new customer, and that customers of its older managed robots.txt feature would get a prompt to confirm the switch. In the example that post publishes, Cloudflare's lines sit at the top of robots.txt between a "# BEGIN Cloudflare Bot Preference Sync" comment and a matching END line, and your existing Disallow lines stay put.
If your old Block AI Bots switch was on, Cloudflare's migration table gives you Search: Allow, Training: Disallow AI Training, and Agent: Block on pages with ads. If you never adjusted these settings, the email says you stay on Allow. Domains that already used the separate rows keep "the practical effect" of their choices, so an old Training: Block becomes Disallow AI Training.
Disallow AI Training and Google
Cloudflare's post names Google-Extended as Google's way to refuse training, so I went and read what Google says that token does. Google's crawler page lists two jobs for it: training future Gemini models, and grounding in Google's Gemini Apps and its Vertex AI developer platform. Grounding there means "providing content from the Google Search index to the model at prompt time." Further down, it says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
So a Google-Extended disallow covers Gemini Apps grounding too. If you want Gemini answers pulling from your pages, I'd weigh that first.
AI Overviews and AI Mode are the AI answers inside Google Search. Google handles your part in them from Search Console, Google's dashboard for site owners, under Settings, then Search generative AI. Google's help page says that control reached all websites as of August 31, 2026, that Include is the default, and that it "doesn't affect AI training." Choosing Exclude means, in Google's words, "You won't receive any traffic or impressions from these features."
Google's pages split the job up like this:
| What you want | Where Google says it's set |
|---|---|
| Your content stays out of Gemini model training and Gemini Apps grounding | Google-Extended in robots.txt |
| Your pages stay out of AI Overviews, AI Mode, and the AI features in Google Discover | Search Console, Settings, Search generative AI |
| Your pages stay out of Google Search completely | noindex, a tag that tells Google not to list a page |
Google names that Search Console control as the one for AI Overviews, so that's where I'd go to leave them. To stay in, leave it on Include; Disallow AI Training keeps Googlebot crawling for search either way.
The Bing tag and Copilot
Cloudflare's post says Microsoft is building support for a robots.txt no-training preference, "targeted for early 2027." Until then, it says, Disallow AI Training "will not automatically convey a no-training preference to Bing through robots.txt." In the meantime, the post points site owners to Bing's NOARCHIVE tag, a line of code you add to a page's HTML, among other Bing tools.
I read Bing's own help page for meta tags next. Among other things, it says NOARCHIVE means "Do not link in Chat and Copilot" and "Do not use the content for training Microsoft's generative AI foundation models." It also lists a second tag, NOCACHE, which will "Display only URL/Snippet/Title in Chat or Copilot" and use only those fields for training. Bing's 2023 announcement about these tags says pages with either one "will still appear in our search results."
As of this writing, every first-party AI citation report I can pull comes from Microsoft: Clarity's AI visibility report, which lists Copilot and partners as its data source, and the AI Performance report in Bing Webmaster Tools. So Microsoft's side is also where you could watch what NOARCHIVE costs you, and I'd be slow to add that tag on any page I wanted cited there.
Block reaches further now
Since September 15, if you pick Block or Block on pages with ads, either one reaches the three mixed-use crawlers. Cloudflare's post says either setting "impacts search as well as training," and that Block "will stop Applebot, Bingbot, and Googlebot from reaching your site," search included.
The migration moves old Block AI Bots users to Disallow AI Training, which Cloudflare says keeps those crawlers allowed for search. The exposure I see is a later edit by whoever reads Block the old way.
For example:
- Your site runs through Cloudflare, and after the move your Training row reads Disallow AI Training.
- A few months from now, your web person decides to "lock down AI" and switches Training to Block.
- From then on, Cloudflare turns away Googlebot, Bingbot, and Applebot across your site. Cloudflare's own post says that choice affects search.
On the Training row, Disallow AI Training is the setting Cloudflare says stops training and keeps search. Block is for when you want the three named mixed-use crawlers gone completely.
Agent: Block on pages with ads
If your old switch was on, your Agent row now reads Block on pages with ads. Cloudflare's post places Googlebot and the other two under search and training, and it defines the Agent row by user-directed agents. It explains the ads rule this way: "agents fetch the page with nobody there to see the ads."
If you run ads, think about a person who asks an assistant to open one of those pages. On a page where Cloudflare detects an ad, that assistant just gets turned away. If I ran an ad-funded blog, that might be a trade I'd take for some pages, and I'd still want to choose it per site.
My checklist
- Start in Security Settings for each of your domains on Cloudflare and read all three rows. Check whether the Block AI Bots switch is gone yet. If your host or developer runs Cloudflare for you, ask them to read you the three rows.
- Open the robots.txt at the root of your site and look for Cloudflare's block at the top. I'd read which crawler names are listed there. If Training reads Disallow AI Training and you see no Cloudflare block, check whether Bot Preference Sync is on, since Cloudflare's posts describe robots.txt as where that preference gets published.
- In Search Console, open Settings, then Search generative AI, and confirm it says what you want. Google's help page says the Generative AI performance report can give you "an idea of how changing your control may impact traffic to your site," so I'd look there before switching to Exclude.
- If Gemini Apps grounding matters to your business, check whether Cloudflare's block in your robots.txt names Google-Extended. If it does, Disallow AI Training works against grounding. With Training on Allow, Bot Preference Sync has no no-training preference to publish, for Apple or anyone else. And Google says Google-Extended has no separate user agent, so a firewall rule can't single it out. Cloudflare's August post says you can turn the sync off and write your robots.txt yourself. That's work for whoever has your Cloudflare login.
- Before you add NOARCHIVE for Bing, decide whether you can give up being linked in Copilot answers.
- Decide who is allowed to change your Training row, and tell them in writing what Block means now.
I sell SEO and AI-search audits. What this page says about Cloudflare, Google, and Bing comes from their own pages, linked below, and the email Cloudflare sent customers. I checked all of it on September 16, 2026. Cloudflare's September post lists AI summaries under "What's next," with a goal of early next year, which from that post's date means early 2027, so parts of this will move again.
Sources
- Cloudflare, "Have it both ways: stay discoverable in search while disallowing AI training," September 15, 2026
- Cloudflare, "Say it once: introducing Bot Preference Sync," August 21, 2026
- Google, "Google's common crawlers" (Google-Extended section)
- Google Search Console Help, "Search generative AI control"
- Google Search Central, "AI Features and Your Website"
- Bing Webmaster Tools, "Robots meta tags and attributes that Bing supports"
- Bing Webmaster Blog, "Announcing new options for webmasters to control usage of their content in Bing Chat," September 22, 2023
Run the free AI Readiness Scanner: paste in your robots.txt and your homepage source, get a scored report on whether ChatGPT, Perplexity, and Google AI can actually read your business. About two minutes, no signup, and nothing you paste ever leaves your browser.
Run the free scan → Get The Receipts free ▸
I run the SEO and AI visibility for my family's Shopify shop and publish the receipts, good and bad. Every number on this site comes from a named tool export, and corrections get published rather than edited away.