Cloudflare AI Crawler Block: Don’t Lock Out Google by Accident

The Cloudflare AI crawler block is about to do something most site owners never asked it to do. On September 15, Cloudflare changes how it treats crawlers that do more than one job, and a toggle a lot of teams flipped in 2024 or 2025 is going to start meaning something new. If you turned on “Block AI bots” when that felt like the responsible move, read this before Monday.

We’ve spent the last year telling clients the same thing about AI search: being findable matters more than being protected. llms.txt didn’t help. Building for agents did. This change is the first time a default setting can quietly undo that work.

What actually changes on September 15

Cloudflare is splitting AI traffic into three buckets: Search (crawling so you show up in results), Agent (a bot visiting on behalf of a real person, right now), and Training (a bot taking your content to teach a model). Fair enough. Different behaviors deserve different rules.

Two things happen on the 15th. First, for new domains joining Cloudflare, Training and Agent get blocked by default on any page that shows ads, while Search stays allowed. If your site doesn’t run ads, that default mostly leaves you alone.

Second, and this is the one that matters, multi-purpose crawlers get judged by everything they do, not just the friendliest thing. Googlebot crawls for Search and for Google’s AI products. Bingbot and Applebot do the same. Under the new rule, a crawler that does both is governed by the most restrictive rule you’ve set. Cloudflare says it plainly in its own announcement: customers who chose to block Training, whether through the new options or the legacy “Block AI bots” service, will block Googlebot, Applebot, and Bingbot.

Read that twice. The setting you turned on to keep your content out of a training set is about to keep you out of Google. Not because anyone changed their mind about you. Because the definition moved.

Why the old Cloudflare AI crawler block is the trap

When Cloudflare shipped its one-click AI block in 2024, it was a clean idea: one switch, block the scrapers. Thousands of agencies flipped it on client sites during a launch or a migration and moved on. Nobody’s been back to that screen since. That’s not negligence. That’s how infrastructure works.

The problem is that “block AI” in 2024 and “block AI” in September 2026 describe different sets of bots. Cloudflare’s numbers explain why they’re forcing the issue: more than half of all traffic on the internet is now non-human, and over 50% of AI crawler hits are re-fetching pages that haven’t changed. Cloudflare wants the AI companies to split their crawlers into honest, single-purpose bots. To force that, it’s making mixed-use crawlers pay the strictest price. Google is the biggest mixed-use crawler there is.

Cloudflare is squeezing Google. Your site is the collateral.

A default is Cloudflare’s opinion, not yours

We don’t think Cloudflare is wrong to push here. Publishers who live on ad revenue have a real grievance, and a toggle that says “index me, don’t absorb me” is overdue. But the default is written for a news site that monetizes attention. It’s not written for a manufacturer with 40 product pages or a B2B brand whose growth plan runs through search and AI answers.

For most of the businesses we build for, the math is simple. Training crawlers are a minor cost. Search crawlers are the business. Agent crawlers are the next customer, arriving early. Blocking the first is fine. Blocking the second by accident is a bad month. Blocking the third on purpose means the site you just paid for can’t sell to the buyer who’s shopping through Gemini or ChatGPT.

So decide. Don’t inherit.

What to do before Monday

This is a fifteen-minute job if you know where to look, and a very expensive one if you find out from your traffic report in October.

Find out if you’re behind Cloudflare at all. A surprising number of clients don’t know. Check the DNS or ask whoever set up the site. Free plans and recently added domains inherit the new defaults. Paid plans keep their existing settings, but the multi-purpose rule still applies to anyone who blocks Training.

Open Security settings and look for the AI traffic controls. If the legacy “Block AI bots” switch is on, you’re in the affected group. Cloudflare has put an opt-out in that same screen that confirms you want no change to Training crawlers that also crawl for Search. Use it, or switch to the new three-bucket controls and set them deliberately: allow Search, allow Agent, and make a real call on Training.

Put your preference in robots.txt too. Cloudflare is extending Content Signals with a use= field (immediate, reference, or full) that states how a bot may keep and reuse what it fetched. It’s a preference, not a lock, but it’s the layer that survives when you change CDNs. Google also offers Google-Extended for opting out of Gemini training without touching Search. Use the tools that separate the behaviors instead of the one that lumps them.

Then watch the crawl. Search Console’s crawl stats on the 16th will tell you in a day what an analytics report tells you in a month. If Googlebot requests fall off a cliff, you know where to look.

The bigger pattern

Every few months the AI conversation produces a new switch that promises to make the problem go away. Block the bots. Add the file. Opt out of the thing. The switches are easy, which is why they get flipped and forgotten. The decision underneath them, who gets to read your site and what they can do with it, doesn’t stay made. This week it moved again.

Twenty-six years in, we’ve learned that the settings nobody’s looked at since launch are the ones that cost the most. If you’d rather have someone check that screen for you before the 15th, that’s exactly what we’re here for.

The Agent-Ready Website Is Mostly the Web Done Right

An AI agent booked a hotel room last week without a human ever looking at the website. It read the page, found the form, filled in the dates, and moved on to the next task in its queue. That is not a preview of where the web is going. That is a Tuesday in 2026.

So here’s the question on the table for anyone who owns a site. When an agent shows up to read your content or complete a purchase on someone’s behalf, can it actually do the job? Most teams answer that by bracing for a rebuild. They have it backward. An agent-ready website is rarely a new website. It’s the fast, structured, semantic web you already owed your human visitors, plus two or three genuinely new hooks that are still early enough to be optional.

We build sites for a living, and we’ve watched every “the web is changing, tear it all down” cycle since 2000. Most of them were vendors selling a rebuild. This one is different, because the change is real and the fix is mostly boring. That’s a good thing. Boring is cheaper than panic.

What “agent-ready” actually means

For most of the last two years, browser agents worked like a person with bad eyesight and no patience. They took a screenshot, guessed which pixels were a button, clicked, read the screen back to a model, and hoped. It was slow, and it broke constantly, because your site was never talking to the agent. The agent was reverse-engineering your interface every single time, and a redesign could blind it overnight.

Agent-ready flips the direction of that conversation. Instead of the machine guessing at your interface, your site tells the machine what it can do and what each action needs. Two very different things are happening under that one label, and they get conflated constantly. Agents that read want to find you, understand you, and cite you when a person asks a question. Agents that act want to finish a task on someone’s behalf: compare three vendors, fill the quote form, add to cart, check out. Reading is a visibility problem. Acting is an infrastructure problem. Your site has to be good at both, and they are not the same work.

We’ve written before about why your brand is invisible to ChatGPT, and that piece was about the reading half: getting understood and quoted when an assistant answers a question. This is the acting half. The stakes are higher, because a purchase is worth more than a mention, and the plumbing is harder, because completing a transaction reliably is a lot to ask of a machine that’s poking at a page built for a human thumb. You can read about the visibility side in our take on AI search visibility. This article is about the side where the agent has a credit card.

The numbers stopped being hype

It’s fair to be skeptical of any trend that shows up with a fresh acronym every quarter. This one has receipts. Adobe Analytics measured AI-referred retail traffic growing close to 400% year over year in early 2026, and that traffic converted meaningfully better than ordinary search traffic once it landed. The IBM Institute for Business Value found that 45% of consumers already use AI for some part of the buying journey. McKinsey put a number on the destination: somewhere between $900 billion and $1 trillion of US retail influenced by agentic commerce by the end of the decade. Shopify has already exposed more than a million merchants to agent-driven checkout, and Amazon’s shopping assistant is serving hundreds of millions of people.

Now the part the trend roundups leave out. The infrastructure isn’t ready, and the data says so out loud. When Walmart let people buy inside a chat interface, those purchases converted about a third as well as sending the same shopper to Walmart’s own site. Read that twice. The demand is real. The plumbing is not. That gap between what shoppers want to do and what merchant sites can actually support is the entire opportunity, and it’s a build problem wearing a marketing costume.

Most of an agent-ready website is the web you already owe your users

Here’s the deflating truth for anyone hoping to buy a shiny new layer. The single best thing you can do to prepare for AI agents is the thing you were supposed to do for humans all along. An agent parses your page using the same signals an assistive screen reader and a search crawler have always relied on. Clean, semantic HTML. Headings that actually describe the content beneath them. Forms with real labels tied to real inputs. Buttons that are buttons, not a div wearing a click handler. Structured data that states, in a language a machine can read, what this page is and what lives on it.

We made the case a few weeks ago that the prettiest 2026 web design trends are also the slowest, and that argument holds double here. A three-second hero animation that hogs the main thread annoys a person. It defeats an agent outright, because the agent times out before your beautiful thing finishes painting. Core Web Vitals were never just an SEO scoreboard. They’re now the line between a machine finishing a task on your site and quitting halfway through. If you want the full version of that argument, we wrote it up here.

Accessibility tells the same story through a different door. The markup that lets a blind user move through your checkout with a keyboard is nearly identical to the markup that lets an agent move through it with no eyes at all. A labeled field, a logical tab order, a form that announces its errors: a screen reader needs those, and so does a machine. Do the fundamentals and you’re most of the way to agent-ready before you touch a single new standard. Skip them, and no protocol on earth bails you out. The agent is only as capable as the page is honest about itself.

Isn’t this just SEO with a new coat of paint?

Reasonable question, and the honest answer is: partly, and that’s the point. A lot of “optimize for AI agents” advice is the structured-data hygiene good developers have preached for a decade, repackaged with urgency. If someone is selling you an agentic web package that turns out to be schema markup and a sitemap, you’re not being lied to, exactly. You’re being charged a premium for fundamentals.

But the acting half is genuinely new, and it’s where the coat of paint stops explaining things. Search optimization was about being found. Agent readiness is about being operated. A crawler indexed your content and left. An agent wants to submit your form, apply your coupon, and complete your purchase, and it wants to do that reliably enough that a person trusts it with a card. That’s not a ranking problem. It’s an API problem, and it’s why the next section matters even though the last one told you to fix your HTML first.

The genuinely new hooks: WebMCP and llms.txt

Two things are actually new, and they’re worth understanding before you decide how much to bet on either.

WebMCP is the bigger one. It’s a proposed browser standard, backed by Microsoft, Shopify, and Booking.com among others, that lets your site register its own tools through a new browser API. Instead of an agent screenshotting your checkout and guessing at the fields, your site declares its capabilities out loud: here is a “checkout” tool, here is a “filter_results” tool, and here are the exact inputs each one expects. Google opened a WebMCP origin trial in Chrome in the spring of 2026, which means developers can switch it on for real users and test it in the wild. You can read Google’s own writeup on the Chrome for Developers WebMCP origin trial. There are two ways to expose those tools: annotate the HTML forms you already have, or define functions in JavaScript with a schema describing their inputs and outputs. If you’ve ever built an API, the mental model clicks instantly. You’re handing the agent a clean API to the thing your page already does, instead of making it infer that API from pixels.

The honest caveat matters more than the feature. An origin trial is not a finished standard. It can change shape, and it can be pulled. Building your entire storefront around it today would be a bet, not a plan, and we’d talk a client out of it. But if you run a real transactional flow, a booking, a quote request, a cart, it’s worth a pilot right now. The teams that understand WebMCP while it’s still experimental are the teams that ship it well the day it becomes table stakes. Early is cheap. Late is a scramble.

llms.txt is the low-effort one, and there’s almost no reason to skip it. It’s a plain markdown file at the root of your domain, the same basic idea as robots.txt or sitemap.xml, that tells language models what your site is about and which pages actually matter. Adoption sat around 10% of sites by the middle of 2026, so it’s early, but it costs an afternoon and it can’t hurt you. If you’d rather hand the models a curated map of your site than let them guess from your navigation, you write that map yourself. Most companies haven’t bothered. That’s a cheap edge sitting on the table. You can read the llms.txt proposal and have a draft live before lunch.

Agentic commerce commoditizes the middle. Brand is the defense.

Now the part that isn’t a developer problem at all, and the reason this belongs in front of whoever owns the brand and not just the codebase. When an agent does the buying, it optimizes on whatever it can measure. Price, specs, availability, star ratings. If the person only said “find me a good laptop under a thousand dollars,” the agent runs a spec comparison, and the product with the cleanest data and the lowest number tends to win. That’s a race to the bottom, and it’s a race you cannot win by being slightly cheaper than the next feed in the list.

There’s exactly one instruction that changes the whole game, and it comes from the human, not the code. It’s the person naming you. “Order me another one of those from Aerie.” “Book the Kimpton, not the cheapest room on the block.” The instant a human asks for you by name, the agent stops comparing and starts fetching. Distinctiveness isn’t an AI default. It’s the thing that survives the agent. We argued that taste is the only real moat in the AI era, and agentic commerce is where that argument gets tested for actual money. A brand strong enough to be requested by name is a brand the agent has to honor. A brand that only competes on the spec sheet is a line item the agent optimizes away without a second pass.

So the technical work and the brand work turn out to be the same project aimed at the same threat. Being readable gets you into the agent’s consideration set. Being memorable gets you requested by name and lifted out of the comparison entirely. You need both halves. Clean markup with a forgettable brand is a site that agents can use to shop you into a commodity. A beloved brand on a site an agent can’t operate is a name people ask for and then can’t get served. The failure modes are different. The fix is one coordinated push, not two.

Where to actually start

You don’t necessarily need a rebuild. You need a sequence, ordered by what pays off soonest.

Fix the fundamentals first, because they serve people and machines in the same stroke. Semantic HTML, labeled forms, honest structured data, and a site fast enough that an agent doesn’t give up waiting on it. If your Core Web Vitals are in the red, start there and close the trend-piece tabs. Then write your llms.txt, because it’s an afternoon of work almost nobody has done. Then, only if you run a genuine transactional flow, pilot WebMCP inside the origin trial and learn how it behaves before anyone makes it mandatory. Watch it closely. Don’t wager the business on an experiment.

Notice what’s not on that list. Ripping out your site. Buying an “AI-ready” template that looks like every other AI-ready template. Bolting on a widget that promises to make you agent-friendly overnight, the same way overlay widgets once promised to make you accessible overnight and mostly didn’t. The shortcut is the trap. It was the trap with accessibility, and it’s the trap here, wearing new branding.

The agentic web rewards the boring virtues. Structure. Speed. Clarity. A brand worth asking for by name. We’ve been building on those four for twenty-six years, which is a strange thing to feel vindicated by, but here we are. If you’d rather not sort the real signal from the acronym noise on your own, that’s the kind of build we do. Start with the fundamentals this week. The standards will keep. Your Core Web Vitals won’t fix themselves while you wait for them to.