Website owners will soon be able to tell most large AI companies, with one line in one file, not to use their content to train AI models. Google, Microsoft, OpenAI and Anthropic each said at the end of October that they will honor the same signal, and the setting you choose may affect where your business shows up in AI tools. This article explains what changed, how the new opt-out works, what it does and does not affect, and how to decide what to do.

What changed at the end of October

“AI training” means feeding large amounts of text, such as web pages, into an AI model so it learns to write and answer questions. Until now, a site that wanted to opt out of AI training had to block each AI company’s crawler, the automated program that reads web pages, by name. That list kept growing.

On October 30, several companies backed one shared signal instead:

How the new AI training opt-out works

Every website can have a small text file called robots.txt. It sits at the root of your site (for example, yoursite.com/robots.txt) and gives instructions to crawlers. Your web developer or hosting company can edit it in a few minutes.

The new signal is a shared label, called a policy token, that any AI company can recognize. It comes from a proposal at the Internet Engineering Task Force (IETF), the group that writes many of the technical standards the internet runs on. To opt out of AI training for your whole site, the file would include two lines:

User-agent: __usage_ai-training
Disallow: /

You can also opt out for only part of a site. For example, a manufacturer could keep its technical manuals out of training data while allowing its press releases.

Google says it will read this the same way it reads Google-Extended, its existing setting for whether your content trains Gemini, Google’s AI assistant. Company-specific settings still work, so you can set one general rule and make exceptions for individual companies.

What it does and does not affect

This is the part most likely to cause confusion.

  • Regular search results are not affected. Google says Google-Extended “does not impact a site’s inclusion in Google Search.” Microsoft says its new preferences keep sites eligible for Bing search results.
  • Some AI answers may be affected. Google’s documentation says Google-Extended also covers grounding, which is when an AI tool looks up current web pages to answer a question. Sites that allow Google-Extended can appear, with links, in Gemini’s answers. Opting out of training through the shared signal is treated as opting out of Google-Extended.
  • It is a request. The proposal describes a preference about how content is used, which companies choose to honor. Companies that have not agreed to honor it may ignore it, and the proposal says private licensing deals can override it.

We wrote about a related setting in September, when Cloudflare’s AI crawler block began affecting Google. The same advice applies: keep the setting that controls search separate from the setting that controls training.

What is still unsettled

The IETF proposal is a draft submitted by individuals. It has not been formally approved, and participants say parts of it are still disputed. Google’s timeline is early 2027, OpenAI’s is “the coming months,” and other AI companies have made no public commitment. For now, adding the line may have little effect, and it is reasonable to watch for updates before relying on it.

Your three options

  1. Allow AI training. Do nothing, or state it plainly. Your content may help AI tools describe your business and products, and you stay eligible for linked mentions in Gemini. The trade-off is that you have no say in how your writing is reused.
  2. Opt out of training for part of your site. Keep original material, such as guides you sell or research you paid for, out of training while leaving product and service pages open. This takes a little more setup and review when your site changes.
  3. Opt out of training everywhere. This is the strongest statement of your preference and does not affect your regular search rankings. It may reduce how often you appear in some AI answers, such as Gemini’s.

Any of these can be the right choice for a given business. A company that sells its expertise as content may weigh this differently from a company that wants to be recommended as widely as possible.

Where to start

Google and Microsoft will not read the new signal until early 2027, so there is time to decide. A reasonable first step looks like this:

  • Look at your current robots.txt file. Visit yoursite.com/robots.txt and note any lines that mention AI companies or “Google-Extended.”
  • Check your hosting and security settings. Services such as Cloudflare may already block AI crawlers on your behalf.
  • Decide which content you consider most valuable. List the pages you would least want reused, if any.
  • Pick one of the three options and write it down. Then ask whoever maintains your site to apply it once support begins.

If you’d like help with any of it, that’s work we do every day.