On October 30, 2026, the four largest Western AI companies agreed on one signal for website owners. OpenAI, Google, Microsoft, and Anthropic said they will honor a single shared robots.txt token that lets a site opt out of AI model training across all four companies at once. Instead of maintaining a separate rule for every company’s crawler, a publisher will be able to state the preference once, in one place, and have it understood the same way by each participating lab.

If you run a website, this matters even if you never train a model yourself. Your content is the raw material these companies want, and until now the only way to say no was a growing list of company-specific instructions that changed every time a new crawler appeared. The new signal does not solve every problem publishers have with AI — it deliberately does not cover AI search answers, and it is not yet live at most of the companies backing it — but it is the first coordinated attempt to make the choice simple.

What was announced on October 30

The announcement, first reported by The Verge and confirmed in separate posts from the companies involved, centers on a new “policy token” called __usage_ai-training. It lives in robots.txt, the plain-text file that has told web crawlers which parts of a site they may visit since the 1990s. A site owner who wants to opt out of AI training adds two lines:

User-Agent: __usage_ai-training
Disallow: /

That is the whole mechanism for a site-wide opt-out. The draft also allows partial opt-outs: a site can disallow training on most paths while explicitly allowing specific folders, such as a press kit, using Allow: lines. The point is that one entry replaces the current patchwork, where a publisher refusing training must name each company’s crawler separately — GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic — and keep that list current as new crawlers launch.

The specification behind the token is an Internet-Draft titled “Machine-Readable Training Preference Signal,” authored by Kevin Kelley, Anthropic’s technical lead for standards, and dated October 30, 2026. Two cautions matter. First, it is an individual draft, not a formal standard: the IETF datatracker notes that an individual Internet-Draft “is not endorsed by the IETF and has no formal standing in the IETF standards process.” Second, the draft expires on May 3, 2027, unless it is updated or advanced.

Who is backing it, and when it actually takes effect

All four companies have committed to honoring the token, but their timelines are uneven — and that is the most important practical detail in the announcement. Nothing changes overnight for most sites.

Company Status at announcement What it has said
Anthropic Implementation begun The draft’s author; has started implementing support, with rollout across its services taking time
OpenAI Coming months Aims to implement the signal in the coming months; existing crawler controls keep working alongside it
Microsoft From February 1, 2027 Will begin honoring the preferences on February 1, 2027, as pages are recrawled
Google Aimed for 2027 Plans to support the token in 2027; no more specific date given yet

In other words, Anthropic is the only participant that has started. For Google, Microsoft, and OpenAI, the shared token is a promise about the near future, not a control that works today. Until each company’s implementation lands, the company-specific rules you already have remain the controls that actually work. Do not delete them when you add the new token.

One notable absence: Meta. Meta’s communications head Andy Stone called a common signal an important step, according to The Verge’s reporting, but Meta did not confirm that it will honor the token. The same week, Meta separately delayed enforcement of its updated commercial terms, which expand permitted content use for AI training, from October 30 to November 20 after pushback from publishers and advertisers. So the “shared” signal currently covers four labs, not the whole industry.

What the signal does — and what it deliberately does not do

The most common misunderstanding about this announcement would be to read it as a way to disappear from AI entirely. It is not. The draft is explicit that the token “is not a directive for crawling behavior”:

  • Training only. The token states a preference about using content to train models. It does not block crawling, search indexing, or caching. A bot may still visit a page that carries the token.
  • Not an AI-answers opt-out. Opting out of training will not keep a page out of AI-written search answers or chatbot responses that quote live web content. That is a separate use, governed by separate controls — and it is the use currently costing publishers traffic, as we covered in our guide to AI search traffic and Reddit’s referral losses.
  • Not retroactive. Adding the token does not remove content already collected into existing datasets or already baked into trained model weights. It governs future training use, not the past.
  • Voluntary. The mechanism binds only the companies that choose to follow it. It carries no enforcement power of its own, and a crawler that ignores robots.txt altogether will ignore this token too.

Microsoft has gone furthest in spelling out finer-grained choices. Alongside the shared token, it plans a page-level training setting and a separate “AI Grounding” setting — grounding being the retrieval step that pulls live web content into chat answers, rather than training model weights. A third option, DisallowExtended, would allow grounding in Microsoft’s own products but not in grounding services it sells to others. Microsoft also says opting out of training will not affect Bing search eligibility, and it is blunt: “Neither choice guarantees inclusion, traffic, or payment.”

Google’s position has its own wrinkle. Google’s John Mueller wrote on the Search Central blog that if a site uses the shared token to opt out without explicitly naming Google-Extended, Google will treat that as if Google-Extended had been disallowed.

Why publishers pushed for this

To understand why four fierce competitors agreed on anything, look at the publisher’s side of the desk. A decade ago, robots.txt was mostly about search engines. Then every AI lab launched its own crawler with its own name and opt-out syntax, and a small publisher had to track all of them — a missed name meant silent inclusion. The shared token is, above all, an admission that this fragmentation had become unworkable.

The timing is not accidental. Publishers are fighting on two fronts: training use of their archives, and AI answers that summarize their pages without sending visitors. Our earlier coverage of websites blocking AI agents showed the blunt end of that frustration — sites shutting the door entirely. A standard training preference is the more surgical alternative: it lets a publisher say “do not train on this” without also saying “do not index me, do not send me traffic, do not quote me.”

What website owners should do now

The honest answer is: prepare, but do not overreact. The token is published and adding it costs nothing, but it changes nothing at Google, Microsoft, or OpenAI today. Here is a practical sequence:

  1. Keep your existing crawler rules. Your current GPTBot, Google-Extended, ClaudeBot, and other company-specific entries are still the controls that work. The shared token is an addition, not a replacement, until each company confirms its implementation.
  2. Add the shared token if you want out of training. If your decision is to refuse training use, add the two-line block to robots.txt now so it is in place as support arrives. If you only want to exclude part of a site, use Allow: lines for the exceptions.
  3. Decide training and AI answers separately. Because the token does not cover AI answers, make that choice explicitly too, using each company’s existing search and grounding controls. Do not assume one setting covers both uses.
  4. Mind Microsoft’s March 1, 2027 deadline. Microsoft says the new preferences replace its older NoArchive and NoCache tags. Sites relying on those tags should switch to the new preferences by March 1, 2027, after which the old tags will no longer be read as training or grounding preferences.
  5. Check how your robots.txt is served. On WordPress, robots.txt is often generated virtually or managed by an SEO plugin rather than edited as a file. Whichever route you use, fetch your live robots.txt afterward and confirm the new lines actually appear.
  6. Do not expect traffic changes. Adding a training opt-out should not change search rankings or search traffic, and Microsoft explicitly says it will not change Bing eligibility. If traffic moves afterward, look for another cause first.
  7. Revisit the decision when implementations land. The draft expires in May 2027, Google and Microsoft have only committed to 2027 in general terms, and Meta has not signed on. Re-check who honors the token once the 2027 rollouts begin.

For small businesses, the decision itself deserves a moment’s thought. Opting out of training is a statement about how your content may be reused, not a growth tactic. If your content is your product — original research, reporting, paid courses — refusal protects an asset. If your content is primarily marketing that exists to be discovered, the AI-answers question (which this token does not answer) is the one that affects your traffic. Many sites will reasonably choose different answers for different sections, and the token’s per-folder Allow: mechanism exists for exactly that kind of split.

The trade-offs nobody should gloss over

Three tensions will decide whether this becomes a real standard or a polite gesture. The first is coverage: a token honored by four companies but not Meta, xAI, or the long tail of scrapers is a partial shield. The second is verification: the announcement does not explain how a publisher can confirm that every service has applied its preference. Robots.txt has always run on trust, and this token inherits that weakness. The third is the answers question: as long as opting out of training leaves a site exposed in AI answers, publishers who feel their traffic is being taken may still reach for the blunt instrument — blocking crawlers entirely — rather than the surgical one.

There is also a quieter trade-off for the AI companies themselves. A shared, machine-readable preference makes it harder for any one lab to claim it could not understand a publisher’s wishes, which is useful if regulators come asking.

Frequently asked questions

Does adding the token remove my site from Google or Bing search?

No. The token governs AI training only, not crawling or search indexing. Microsoft states that opting out of training does not affect eligibility for regular Bing results.

Will it stop AI chatbots from quoting my pages?

No. Quoting live content in answers is grounding or retrieval, not training. Microsoft is building a separate grounding control for that use. The shared token alone does not cover it.

Do I still need GPTBot, Google-Extended, and ClaudeBot rules?

Yes, for now. Only Anthropic has begun implementing the shared token. Until Google, Microsoft, and OpenAI confirm their implementations, the company-specific rules remain the ones that actually work.

Does it delete my content from models already trained?

No. The token governs future training use. It does not remove material already collected into datasets or trained into existing models.

What if I only want to protect part of my site?

Use the token with path-specific rules: disallow the sections you want protected and add Allow: lines for the folders you are happy to include, such as a press kit.

The bottom line

A single line in robots.txt that four of the biggest AI companies will honor is a genuine improvement on the mess it replaces — one choice instead of a dozen, stated in a standard place, with a public draft behind it. But it is a beginning, not a settlement. It is not yet live at three of the four companies backing it, it does not cover AI answers, it does not reach back into models already trained, and it does not bind companies that have not signed on. Add it if you want out of training, keep your existing rules until the rollouts land, decide the AI-answers question separately, and watch two dates: February 1, 2027, when Microsoft starts honoring the token, and May 3, 2027, when the draft itself expires unless renewed.

If your site is the kind of asset worth protecting — or the kind that needs to be found — it is worth getting the foundations right first. WebReOn builds fast, search-friendly websites through WebReOn Tools, and WebReOn Chat shows how AI can be put to work without giving away the content that makes your business yours. For students and small teams, our guide to the best free AI tools for students is a practical place to start.

Sources consulted: The Verge’s October 30, 2026 report (via subsequent coverage); Skimwire, Emergent, AI Weekly, RR Tech Pulse, Silicon Report, and Oossa coverage of the announcement; and the KimKJ AI Daily Briefing for October 31, 2026. Timelines and the draft’s status are as reported by those outlets; the IETF draft remains an individual submission, not a formal standard.