One robots.txt Rule to Opt Out of AI Training (2026)

Last updated: 5 November 2026

Google and Microsoft have both said they will read one shared line in robots.txt that tells AI companies not to train models on your pages. Microsoft starts honouring it from 1 February 2027, and Google aims to support it during 2027. Your normal search visibility does not change because of it, but the decision about AI training now deserves five calm minutes of your time.

We manage robots.txt files for a lot of small sites, and the most common mistake we see is panic editing. Someone reads a headline, pastes a block of rules from a forum, and quietly shuts out a crawler they actually needed. So this guide goes slowly. What changed, who it matters for, and exactly what we would do this week.

What actually changed

Until now, saying "no" to AI training meant writing a separate rule for every AI company. One line for this bot, another for that one, and a new line every time a new crawler showed up in your logs. Most site owners never kept up.

On 30 October 2026, Google announced on its Search Central blog that it plans to support a shared AI training control. It comes from a draft written inside the IETF, the group that maintains many internet standards. Microsoft published its own announcement on the same day, with the exact syntax.

The idea is simple. Instead of naming every AI crawler, you name one policy token. The token is not a real bot. It is a label that says "this is my preference about AI training", and every participating provider agrees to read it.

Here is what the rule looks like if you want to opt the whole site out of AI training:

User-Agent: __usage_ai-training
Disallow: /

You can also limit it to one folder, for example Disallow: /members/, if only part of your site needs protecting. If you say nothing at all, the default is allow.

How Google will read it

Google says that if your robots.txt opts out with this token and you have not written a separate rule for Google-Extended, it will treat that as if you had disallowed Google-Extended. Google-Extended is the control Google has offered since 2023 for Gemini model training and for grounding in Gemini Apps.

Two more details matter. First, company-specific rules keep working, so you can use the shared token for a broad "no" and still make an exception for one provider. Second, Google's own crawler documentation says Google-Extended does not affect whether your site appears in Google Search, and it is not used as a ranking signal. That is the part that should calm most bloggers down.

How Microsoft will read it

Microsoft went a step further. It reads the shared training token, and it also adds a separate page-level setting for grounding, which is when an AI assistant pulls your page in to help write an answer. Training and grounding become two separate choices. Microsoft starts applying the new preferences as it recrawls pages from 1 February 2027. Sites that relied on the older NoArchive or NoCache tags for this purpose should switch before 1 March 2027, because after that date those tags stop counting as training or grounding preferences.

Training, grounding and search are three different things

This is where most confusion starts. People say "AI bot" as if it were one thing. It isn't. A single company can run one crawler that builds a search index, another that collects training data, and a third that fetches a page only when a user asks a question. Blocking the wrong one has very different results.

Use of your page	What it means	What the new token covers
Search indexing	Your page can appear in normal search results	Not covered. Keep your search crawler rules as they are.
AI training	Your text may help teach future AI models	Covered. This is exactly what the shared token is for.
Grounding in answers	An assistant reads your page to support a live answer, often with a link	Partly. At Google it travels with Google-Extended. Microsoft has a separate page-level setting.

Keep this table in mind and most decisions get easier.

Who this affects

Bloggers and content publishers

If your income depends on original writing, recipes, tutorials, reviews or research, this is the group with the most to think about. You may not want your work feeding a model for free. But think about grounding separately. Being cited in an AI answer can still send a curious reader your way, and blocking it could reduce how often your brand shows up there.

Small businesses and local service sites

A dental clinic in Pune or a tile shop in Jaipur usually wants to be found everywhere. Your service pages are basically adverts. For most sites like this we would leave training allowed, or at least not rush. There is little to protect and a lot of visibility to keep.

WordPress and plugin sellers

Documentation pages are a special case. You probably want assistants to answer "how do I set up X" correctly, so blocking them may hurt your users more than it helps you. A paid course or members-only area is different, and the folder-level rule fits that nicely.

Agencies managing many client sites

Robots.txt used to be a technical file that only the developer touched. Now it carries a business decision. Write down each client's choice, get it approved, and keep that note next to the file. It saves a lot of awkward questions later.

What to do now: a short checklist

1. Open your current robots.txt. Type yourdomain.com/robots.txt in a browser and read every line. Many sites have rules nobody remembers adding.
2. Decide on training first. Ask one plain question: are we comfortable with our content helping train AI models? Yes, no, or only for certain folders.
3. Decide on grounding separately. Do we want assistants to read and cite our pages in answers? For most businesses the answer is yes.
4. Add the shared token only if your answer to training is no. Put it as its own group. Do not merge it into your User-agent: * group, because the draft says wildcard groups are not read as matching this token.
5. Leave search crawler rules alone. Never touch the lines for normal search crawlers while doing this. One wrong Disallow under the wrong user agent can remove a site from search.
6. Keep your sitemap line. It is easy to delete by accident when pasting new rules.
7. Test after saving. Reload the live file, check the formatting, and use a robots.txt tester to confirm your important pages are still crawlable.
8. If you used NoArchive or NoCache for AI reasons, plan the switch to Microsoft's new page-level tags before 1 March 2027.

What this rule cannot do

Let's be honest about the limits. Robots.txt is a request, not a lock. Responsible companies follow it. Others may not, and a badly behaved scraper can simply pretend to be an ordinary browser. If you need real enforcement, that happens at your hosting firewall or CDN, where you can block traffic by behaviour and not just by name.

Be careful with one-click "block all AI" switches too. Some security dashboards now offer them, and they can block assistants you actually want reading your pages. Turn such a setting on only after you have made the training and grounding decisions above.

Also remember that the shared token is still a draft. The IETF discussion continues, and the details could still shift before it becomes a final standard. Neither Google nor Microsoft has promised that your choice will bring more traffic or any payment. It is a preference, nothing more.

Our take

For most small sites we work with, the sensible move is boring. Leave search alone. Leave grounding on. Think properly about training, and if you do opt out, use the shared token in its own clean group. Then put a reminder in your calendar for early 2027 to check whether Google's support has gone live.

What will matter more over the next year is not the robots.txt line. It is whether search engines and assistants see your site as a trusted source in the first place. That still comes from useful content, a fast site and real mentions from relevant websites. If your pages are slow, our guide on converting images to WebP in WordPress is a quick technical win, and our lightweight WordPress plugins help with the speed side.

On the authority side, honest coverage on relevant sites is what we do every day. If you want help with that, have a look at our white hat guest post packages, or contact our team with your site and goals.

FAQ

Will blocking AI training hurt my Google rankings?

Google says Google-Extended, which the shared token maps to when you have no separate Google-Extended rule, does not affect inclusion in Google Search and is not a ranking signal. Your normal search crawler rules are what decide search visibility.

When does the shared AI training rule start working?

Microsoft says it will begin honouring the new preferences as it crawls from 1 February 2027. Google says it aims to support the control in 2027. Adding the line today does no harm, but expect it to be read only once each provider switches it on.

Do I still need separate rules for each AI crawler?

Not for a simple site-wide training preference. Company-specific rules keep working, though, so you can still use them for exceptions, such as allowing one provider while saying no to everyone else.

Is robots.txt enough to stop all AI scraping?

No. It only works with crawlers that choose to respect it. For firm blocking you need firewall or CDN rules on your hosting, and even those make scraping harder rather than impossible.

What is the difference between AI training and grounding?

Training means your content helps teach a future model. Grounding means an assistant reads your page at the moment someone asks a question, to support its answer, often with a link back. Many businesses say no to the first and yes to the second.