Cloudflare AI Crawler Block: Don’t Lock Out Google by Accident

The Cloudflare AI crawler block is about to do something most site owners never asked it to do. On September 15, Cloudflare changes how it treats crawlers that do more than one job, and a toggle a lot of teams flipped in 2024 or 2025 is going to start meaning something new. If you turned on “Block AI bots” when that felt like the responsible move, read this before Monday.

We’ve spent the last year telling clients the same thing about AI search: being findable matters more than being protected. llms.txt didn’t help. Building for agents did. This change is the first time a default setting can quietly undo that work.

What actually changes on September 15

Cloudflare is splitting AI traffic into three buckets: Search (crawling so you show up in results), Agent (a bot visiting on behalf of a real person, right now), and Training (a bot taking your content to teach a model). Fair enough. Different behaviors deserve different rules.

Two things happen on the 15th. First, for new domains joining Cloudflare, Training and Agent get blocked by default on any page that shows ads, while Search stays allowed. If your site doesn’t run ads, that default mostly leaves you alone.

Second, and this is the one that matters, multi-purpose crawlers get judged by everything they do, not just the friendliest thing. Googlebot crawls for Search and for Google’s AI products. Bingbot and Applebot do the same. Under the new rule, a crawler that does both is governed by the most restrictive rule you’ve set. Cloudflare says it plainly in its own announcement: customers who chose to block Training, whether through the new options or the legacy “Block AI bots” service, will block Googlebot, Applebot, and Bingbot.

Read that twice. The setting you turned on to keep your content out of a training set is about to keep you out of Google. Not because anyone changed their mind about you. Because the definition moved.

Why the old Cloudflare AI crawler block is the trap

When Cloudflare shipped its one-click AI block in 2024, it was a clean idea: one switch, block the scrapers. Thousands of agencies flipped it on client sites during a launch or a migration and moved on. Nobody’s been back to that screen since. That’s not negligence. That’s how infrastructure works.

The problem is that “block AI” in 2024 and “block AI” in September 2026 describe different sets of bots. Cloudflare’s numbers explain why they’re forcing the issue: more than half of all traffic on the internet is now non-human, and over 50% of AI crawler hits are re-fetching pages that haven’t changed. Cloudflare wants the AI companies to split their crawlers into honest, single-purpose bots. To force that, it’s making mixed-use crawlers pay the strictest price. Google is the biggest mixed-use crawler there is.

Cloudflare is squeezing Google. Your site is the collateral.

A default is Cloudflare’s opinion, not yours

We don’t think Cloudflare is wrong to push here. Publishers who live on ad revenue have a real grievance, and a toggle that says “index me, don’t absorb me” is overdue. But the default is written for a news site that monetizes attention. It’s not written for a manufacturer with 40 product pages or a B2B brand whose growth plan runs through search and AI answers.

For most of the businesses we build for, the math is simple. Training crawlers are a minor cost. Search crawlers are the business. Agent crawlers are the next customer, arriving early. Blocking the first is fine. Blocking the second by accident is a bad month. Blocking the third on purpose means the site you just paid for can’t sell to the buyer who’s shopping through Gemini or ChatGPT.

So decide. Don’t inherit.

What to do before Monday

This is a fifteen-minute job if you know where to look, and a very expensive one if you find out from your traffic report in October.

Find out if you’re behind Cloudflare at all. A surprising number of clients don’t know. Check the DNS or ask whoever set up the site. Free plans and recently added domains inherit the new defaults. Paid plans keep their existing settings, but the multi-purpose rule still applies to anyone who blocks Training.

Open Security settings and look for the AI traffic controls. If the legacy “Block AI bots” switch is on, you’re in the affected group. Cloudflare has put an opt-out in that same screen that confirms you want no change to Training crawlers that also crawl for Search. Use it, or switch to the new three-bucket controls and set them deliberately: allow Search, allow Agent, and make a real call on Training.

Put your preference in robots.txt too. Cloudflare is extending Content Signals with a use= field (immediate, reference, or full) that states how a bot may keep and reuse what it fetched. It’s a preference, not a lock, but it’s the layer that survives when you change CDNs. Google also offers Google-Extended for opting out of Gemini training without touching Search. Use the tools that separate the behaviors instead of the one that lumps them.

Then watch the crawl. Search Console’s crawl stats on the 16th will tell you in a day what an analytics report tells you in a month. If Googlebot requests fall off a cliff, you know where to look.

The bigger pattern

Every few months the AI conversation produces a new switch that promises to make the problem go away. Block the bots. Add the file. Opt out of the thing. The switches are easy, which is why they get flipped and forgotten. The decision underneath them, who gets to read your site and what they can do with it, doesn’t stay made. This week it moved again.

Twenty-six years in, we’ve learned that the settings nobody’s looked at since launch are the ones that cost the most. If you’d rather have someone check that screen for you before the 15th, that’s exactly what we’re here for.

llms.txt Is Everywhere. The Bots Aren’t Reading It.

Your SEO platform flagged a missing llms.txt file and called it a site issue. It isn’t one. Somewhere in the last year, a small Markdown file at the root of your domain turned into the thing every audit tool nags you about and every other LinkedIn post swears you need. The pitch is tidy. Robots.txt tells crawlers where they can go, sitemaps tell them what exists, so llms.txt must be the file that gets your brand into ChatGPT. That logic is clean. The crawler data says it’s also wrong.

What llms.txt actually promises

llms.txt is a proposal from Jeremy Howard, published in September 2024. The idea is reasonable. Raw HTML is a mess of navigation, cookie banners, pop-ups, and scripts, and a language model has to strip all of that away before it can read your actual content. An llms.txt file hands the model a clean, curated Markdown overview of your site so it can skip the cleanup. That’s the whole design. It helps a model use your site at inference time. It was never built to change your ranking inside an AI answer box.

Somewhere between the proposal and the pitch deck, that got lost. Tools started treating llms.txt like the next robots.txt. It reads like progress. It’s mostly a category error.

The crawler data is in, and it isn’t close

OtterlyAI ran the experiment everyone kept theorizing about. They put a correctly built llms.txt at the root of a site and watched the server logs for 90 days. Out of more than 62,000 AI bot visits, 84 touched the llms.txt file. That’s about a tenth of a percent. An average content page on the same site pulled roughly 265 bot visits in the same window, so the file the whole industry keeps nagging you about performed three times worse than a normal page, and no better than a stray PDF. The logs tell the story the theorizing couldn’t.

It’s not one study, either. Search Engine Land tracked llms.txt across ten sites and found two with AI traffic gains, neither traceable to the file. Google has said plainly that Search doesn’t use llms.txt, and John Mueller compared it to the old keywords meta tag, which is about the most polite way an engineer can call something dead. Adoption keeps climbing, north of a hundred thousand files indexed this spring, and the bots keep not reading them. Publishing a file isn’t the same as being read. The gap between those two is the whole story.

Where llms.txt actually earns its place

This is where the honest answer splits from the hot take. llms.txt isn’t useless. It’s just aimed at a different target than the one marketers keep firing at.

If you run documentation, an API, or a product that other people’s AI tools plug into, llms.txt is a genuinely good idea. Stripe, Vercel, Cloudflare, and Anthropic all ship one, and none of them are chasing a ranking. They’re making it cheap for a third-party AI assistant to pull a clean version of their docs instead of scraping and de-noising a page on every request. When a tool has to hit a search API at ten to fourteen dollars per thousand queries, a pre-built Markdown snapshot saves real money and returns better answers. That’s an integration decision. It’s an engineering courtesy to the developers building on top of you.

So the test is simple. If people build AI products on top of your content, llms.txt does something real. If you just want Perplexity to mention your brand, it does nothing. Same file, two very different jobs.

What actually moves your AI visibility

The uncomfortable part is that the things that get you cited by an AI engine are the same things that have always worked, which is why nobody’s selling them as a shortcut. Write content with a real point of view. Structure it so a machine can tell what the page is about. Mark it up with clean, honest schema. Earn references from places that already carry authority. We’ve made this argument before, from the strategy side: if your team can’t agree on what you stand for, neither can ChatGPT.

None of that fits in a file you drop at your root and forget. It’s slower, it’s harder, and it’s the actual work. A tidy llms.txt on a thin, forgettable site is a clean label on an empty jar.

If you want a site that AI engines can actually read and want to quote, that’s a content and build problem, not a config file. That part we can help with. Add the llms.txt too, if you like. Just don’t file it under strategy.