Cloudflare AI Crawler Block: Don’t Lock Out Google by Accident

The Cloudflare AI crawler block is about to do something most site owners never asked it to do. On September 15, Cloudflare changes how it treats crawlers that do more than one job, and a toggle a lot of teams flipped in 2024 or 2025 is going to start meaning something new. If you turned on “Block AI bots” when that felt like the responsible move, read this before Monday.

We’ve spent the last year telling clients the same thing about AI search: being findable matters more than being protected. llms.txt didn’t help. Building for agents did. This change is the first time a default setting can quietly undo that work.

What actually changes on September 15

Cloudflare is splitting AI traffic into three buckets: Search (crawling so you show up in results), Agent (a bot visiting on behalf of a real person, right now), and Training (a bot taking your content to teach a model). Fair enough. Different behaviors deserve different rules.

Two things happen on the 15th. First, for new domains joining Cloudflare, Training and Agent get blocked by default on any page that shows ads, while Search stays allowed. If your site doesn’t run ads, that default mostly leaves you alone.

Second, and this is the one that matters, multi-purpose crawlers get judged by everything they do, not just the friendliest thing. Googlebot crawls for Search and for Google’s AI products. Bingbot and Applebot do the same. Under the new rule, a crawler that does both is governed by the most restrictive rule you’ve set. Cloudflare says it plainly in its own announcement: customers who chose to block Training, whether through the new options or the legacy “Block AI bots” service, will block Googlebot, Applebot, and Bingbot.

Read that twice. The setting you turned on to keep your content out of a training set is about to keep you out of Google. Not because anyone changed their mind about you. Because the definition moved.

Why the old Cloudflare AI crawler block is the trap

When Cloudflare shipped its one-click AI block in 2024, it was a clean idea: one switch, block the scrapers. Thousands of agencies flipped it on client sites during a launch or a migration and moved on. Nobody’s been back to that screen since. That’s not negligence. That’s how infrastructure works.

The problem is that “block AI” in 2024 and “block AI” in September 2026 describe different sets of bots. Cloudflare’s numbers explain why they’re forcing the issue: more than half of all traffic on the internet is now non-human, and over 50% of AI crawler hits are re-fetching pages that haven’t changed. Cloudflare wants the AI companies to split their crawlers into honest, single-purpose bots. To force that, it’s making mixed-use crawlers pay the strictest price. Google is the biggest mixed-use crawler there is.

Cloudflare is squeezing Google. Your site is the collateral.

A default is Cloudflare’s opinion, not yours

We don’t think Cloudflare is wrong to push here. Publishers who live on ad revenue have a real grievance, and a toggle that says “index me, don’t absorb me” is overdue. But the default is written for a news site that monetizes attention. It’s not written for a manufacturer with 40 product pages or a B2B brand whose growth plan runs through search and AI answers.

For most of the businesses we build for, the math is simple. Training crawlers are a minor cost. Search crawlers are the business. Agent crawlers are the next customer, arriving early. Blocking the first is fine. Blocking the second by accident is a bad month. Blocking the third on purpose means the site you just paid for can’t sell to the buyer who’s shopping through Gemini or ChatGPT.

So decide. Don’t inherit.

What to do before Monday

This is a fifteen-minute job if you know where to look, and a very expensive one if you find out from your traffic report in October.

Find out if you’re behind Cloudflare at all. A surprising number of clients don’t know. Check the DNS or ask whoever set up the site. Free plans and recently added domains inherit the new defaults. Paid plans keep their existing settings, but the multi-purpose rule still applies to anyone who blocks Training.

Open Security settings and look for the AI traffic controls. If the legacy “Block AI bots” switch is on, you’re in the affected group. Cloudflare has put an opt-out in that same screen that confirms you want no change to Training crawlers that also crawl for Search. Use it, or switch to the new three-bucket controls and set them deliberately: allow Search, allow Agent, and make a real call on Training.

Put your preference in robots.txt too. Cloudflare is extending Content Signals with a use= field (immediate, reference, or full) that states how a bot may keep and reuse what it fetched. It’s a preference, not a lock, but it’s the layer that survives when you change CDNs. Google also offers Google-Extended for opting out of Gemini training without touching Search. Use the tools that separate the behaviors instead of the one that lumps them.

Then watch the crawl. Search Console’s crawl stats on the 16th will tell you in a day what an analytics report tells you in a month. If Googlebot requests fall off a cliff, you know where to look.

The bigger pattern

Every few months the AI conversation produces a new switch that promises to make the problem go away. Block the bots. Add the file. Opt out of the thing. The switches are easy, which is why they get flipped and forgotten. The decision underneath them, who gets to read your site and what they can do with it, doesn’t stay made. This week it moved again.

Twenty-six years in, we’ve learned that the settings nobody’s looked at since launch are the ones that cost the most. If you’d rather have someone check that screen for you before the 15th, that’s exactly what we’re here for.

Search Console’s New AI Report: Visibility Isn’t Value

Google spent two years pushing AI answers into every results page while giving you no way to see yourself in them. That era just ended. As of August 31, every Search Console property in the world has two new switches: a report that shows how often your pages appear in AI Overviews and AI Mode, and a toggle that pulls your content out of Google’s AI features entirely. One deserves your attention. The other deserves your suspicion.

We’ve spent most of 2026 telling clients that AI search visibility was real but unmeasurable. When we wrote about why brands are invisible to ChatGPT back in April, the honest caveat was that nobody could see the scoreboard. That caveat just expired.

The scoreboard everyone asked for

Google introduced its generative AI performance report in June and finished rolling it out to every site at the end of August. Open Search Console today and you can see the impressions your pages earn inside AI Overviews, AI Mode, and the AI features in Discover, sliced by page, country, device, and date.

If you’re a marketing director who has spent this year answering “how are we doing in AI search” with a shrug, this is a real gift. And yes… we have been asking ourselves this question in our own agency for our own Website. With that said, you can finally point at a number. You can watch it move when you publish something worth quoting. You can see which pages Google’s AI reaches for and which ones it walks past.

What it counts. What it hides.

Now the fine print. The report counts impressions and stops there. No queries. No clicks. No click-through rate, no position, no revenue trail. It answers the question you were asking last year: are we visible? It can’t answer the question your CFO is asking this year: is any of it earning anything?

That gap matters because AI search doesn’t hand off traffic the way blue links did. Someone asks a question, reads the answer, and leaves. If your brand shaped that answer, the report counts it. Everything that happens next, the branded search two days later, the direct visit, the mention in a sales call, lives outside the report entirely.

So treat the new number as a direction check, not a KPI. Pair it with the signals that show intent actually moved: branded search volume, direct traffic, and the “how did you hear about us” field your sales team keeps forgetting to read.

The off switch is a decision focused on business verticals.

The second switch is the one to handle with care. The same rollout gave every site owner a control to block their content from Google’s AI features. Flip it and your pages stop appearing in AI Overviews and AI Mode. Google is explicit about the cost: opted-out sites get no traffic and no impressions from those features. Your regular rankings stay untouched. The AI shelf just goes dark.

If you’re a publisher whose product is the content itself, run that math seriously. An AI answer that substitutes for your article is a real loss, and the toggle is a legitimate business lever.

If you sell anything else, the math points the other way. Your buyers are standing in the AI aisle asking the clerk what to buy. Blocking Google’s AI doesn’t punish Google. It takes your product off the shelf while your competitors stay stocked. Publishers can argue AI answers replace their product. For everyone else, AI answers are the shelf.

Read it monthly. Then get back to work.

Our advice after a few weeks inside these reports: check the trend monthly, note which pages earn citations, and resist the urge to build a dashboard shrine to AI impressions. Teams that chase impressions will produce the same thin, keyword-shaped sludge that chasing rankings produced. Same panic, new metric.

What earns the citation hasn’t changed. When we covered llms.txt in July, the point was never the file. The point was that AI systems quote brands with a sharp point of view, real proof behind their claims, and messaging their own team can repeat without checking notes. Now that work finally shows up on a scoreboard. The scoreboard didn’t change the game. It just turned on the lights.

If you’d rather your brand be the answer instead of the audience, that’s the work we do. Bring us the report. We’ll bring the plan.

llms.txt Is Everywhere. The Bots Aren’t Reading It.

Your SEO platform flagged a missing llms.txt file and called it a site issue. It isn’t one. Somewhere in the last year, a small Markdown file at the root of your domain turned into the thing every audit tool nags you about and every other LinkedIn post swears you need. The pitch is tidy. Robots.txt tells crawlers where they can go, sitemaps tell them what exists, so llms.txt must be the file that gets your brand into ChatGPT. That logic is clean. The crawler data says it’s also wrong.

What llms.txt actually promises

llms.txt is a proposal from Jeremy Howard, published in September 2024. The idea is reasonable. Raw HTML is a mess of navigation, cookie banners, pop-ups, and scripts, and a language model has to strip all of that away before it can read your actual content. An llms.txt file hands the model a clean, curated Markdown overview of your site so it can skip the cleanup. That’s the whole design. It helps a model use your site at inference time. It was never built to change your ranking inside an AI answer box.

Somewhere between the proposal and the pitch deck, that got lost. Tools started treating llms.txt like the next robots.txt. It reads like progress. It’s mostly a category error.

The crawler data is in, and it isn’t close

OtterlyAI ran the experiment everyone kept theorizing about. They put a correctly built llms.txt at the root of a site and watched the server logs for 90 days. Out of more than 62,000 AI bot visits, 84 touched the llms.txt file. That’s about a tenth of a percent. An average content page on the same site pulled roughly 265 bot visits in the same window, so the file the whole industry keeps nagging you about performed three times worse than a normal page, and no better than a stray PDF. The logs tell the story the theorizing couldn’t.

It’s not one study, either. Search Engine Land tracked llms.txt across ten sites and found two with AI traffic gains, neither traceable to the file. Google has said plainly that Search doesn’t use llms.txt, and John Mueller compared it to the old keywords meta tag, which is about the most polite way an engineer can call something dead. Adoption keeps climbing, north of a hundred thousand files indexed this spring, and the bots keep not reading them. Publishing a file isn’t the same as being read. The gap between those two is the whole story.

Where llms.txt actually earns its place

This is where the honest answer splits from the hot take. llms.txt isn’t useless. It’s just aimed at a different target than the one marketers keep firing at.

If you run documentation, an API, or a product that other people’s AI tools plug into, llms.txt is a genuinely good idea. Stripe, Vercel, Cloudflare, and Anthropic all ship one, and none of them are chasing a ranking. They’re making it cheap for a third-party AI assistant to pull a clean version of their docs instead of scraping and de-noising a page on every request. When a tool has to hit a search API at ten to fourteen dollars per thousand queries, a pre-built Markdown snapshot saves real money and returns better answers. That’s an integration decision. It’s an engineering courtesy to the developers building on top of you.

So the test is simple. If people build AI products on top of your content, llms.txt does something real. If you just want Perplexity to mention your brand, it does nothing. Same file, two very different jobs.

What actually moves your AI visibility

The uncomfortable part is that the things that get you cited by an AI engine are the same things that have always worked, which is why nobody’s selling them as a shortcut. Write content with a real point of view. Structure it so a machine can tell what the page is about. Mark it up with clean, honest schema. Earn references from places that already carry authority. We’ve made this argument before, from the strategy side: if your team can’t agree on what you stand for, neither can ChatGPT.

None of that fits in a file you drop at your root and forget. It’s slower, it’s harder, and it’s the actual work. A tidy llms.txt on a thin, forgettable site is a clean label on an empty jar.

If you want a site that AI engines can actually read and want to quote, that’s a content and build problem, not a config file. That part we can help with. Add the llms.txt too, if you like. Just don’t file it under strategy.