On 30 October 2026 Google said it plans a shared control in robots.txt (where a site instructs automated visitors) over whether AI companies may train their models on it. It aims for 2027 and calls it a draft, and the post doesn't give the name or exact wording. My reading is that it covers training, not whether you're cited in answers. So today's job is knowing what your robots.txt says and who can change it. Mine, as of 5 November 2026, has no GPTBot (an OpenAI crawler) or Google-Extended entry.
What Google announced on 30 October
John Mueller, Google's Search Advocate, announced the plan on the Google Search Central Blog on 30 October 2026. A website would use a shared 'policy token' in robots.txt to state its AI training preferences to several AI providers in one step. I read a policy token as a single agreed keyword in the file. The post says the idea is based on work from the AIPREF working group at the Internet Engineering Task Force, an internet standards group.
Three limits sit in the post itself. Google aims to support the control, alongside other AI providers, in 2027. It calls the plan a draft proposal. And it doesn't give the token's name or its syntax, meaning the exact wording a file would need.
Google-Extended, the AI training control Google has offered since 2023, is named as one that works alongside the new signal. The post says existing company-specific controls will continue to function. If a site uses the token to opt out of training and doesn't explicitly specify Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended. A disallow is the instruction that tells a visitor to stay out. Other AI providers can handle it similarly, the post says, but it names none of them.
What the control is for
The post describes a training control. Training means an AI company using a site's text to build or improve its models. The post says the mechanism brings the IETF's definition of AI training, while discussions continue on the rest of the AIPREF vocabulary, which I take to mean the other preferences the group is still working out.
So my reading is that the shared control, as described, is about training. The post doesn't say it covers whether a site is crawled (read by an automated visitor) for other purposes, or whether it's cited in an AI answer. That's a gap in the post, not a statement that it doesn't.
The one place the post touches answers is Google-Extended. It says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. I read grounding as Gemini drawing on web pages to build an answer, and the post doesn't define it.
My inference, from setting the disallow sentence above beside that one: a site that opts out through the token alone could end up with a Google-Extended disallow and no explicit allow. The post doesn't say what that does to a site's eligibility for grounding, and I'm not going to guess.
What my robots.txt says today
Every site can publish a robots.txt at its own address, such as rodfernandezdesign.com/robots.txt. In it, a site addresses automated visitors by name and says what it asks of each. The name a visitor sends to identify itself is its user agent, and a crawler is any automated visitor that reads pages, whether it belongs to a search engine or an AI company.
As of 5 November 2026, mine has no entry for GPTBot, which OpenAI's documentation lists among its crawlers, and none for Google-Extended. It does carry a comment line pointing to /llms.txt, a separate plain text file that describes the site for AI tools, and I'm making no claim about who reads it.
The post says sites that specifically allow Google-Extended are eligible for grounding. My file has no entry naming it, and the post doesn't say whether a catch-all entry counts as specifically allowing, so I can't tell you which side of that I'm on.
The whole file, as of 5 November 2026, in order:
- Four named visitors, Googlebot, Bingbot, Twitterbot and facebookexternalhit, each with an entry that allows everything.
- A catch-all entry for every other visitor, marked with an asterisk, that allows everything except a folder called /tmp/.
- A Sitemap line, which points to the list of the site's pages.
- A comment line pointing to /llms.txt.
What a request using GPTBot's name got
A robots.txt entry is addressed to a name, so I also looked at what my sites hand over to one. OpenAI's documentation describes GPTBot as crawling content that may be used in training its models. On 25 October and again on 5 November 2026 I fetched the home page of each of my two sites twice, once sending GPTBot as the user agent and once sending an ordinary browser's. Both sites returned the page's HTML (its underlying code) with the text already in it, identical either way. That was about 1,830 words on rodfernandezdesign.com and about 1,455 on Zeta Hybrid Tech, my Astro site, which I measured in Astro or Next.js for a business website?.
The limit: it was my machine sending that name, not OpenAI's network. It shows what my sites hand to a request that says it's GPTBot, not what OpenAI's crawler receives from its own network. And because my robots.txt has no GPTBot entry, the test says nothing about what a robots.txt rule does.
What this article can't tell you
This is a news explainer with my own file as the worked example. It isn't a test of what the control does, because I can't run one: the post doesn't give a name or syntax to try, and its aim is 2027. I haven't read the IETF working group's drafts, so everything here about the plan comes from Google's post, which describes a draft proposal.
Two more gaps. The post names no provider other than Google, so I can't tell you who else will take part. And I started logging automated visits to both my sites on 3 November 2026 but haven't analysed that yet, so I've got no figures on which crawlers visit.
How to sort your own situation
Two things are quick to establish today: what the file says and who can edit it. What the 2027 control will look like is beyond what the post tells us. Here's how I'd sort what you find.
- The file has entries for AI crawlers: you know what it says about them today, and the post says existing company-specific controls will continue alongside the new signal.
- The file has none for GPTBot or Google-Extended, as mine doesn't: it's silent on those by name, which is my position as of 5 November 2026.
- Nobody can say who edits the file or why it reads as it does: that's the gap to close first, by asking whoever has access to the site's files or hosting.
Where to start
Looking is free. Open your site's /robots.txt in a browser, or read the file in the site's code, and check two things: whether any AI company's crawler is named, and whether Google-Extended appears. Then read the catch-all entry, the one marked with an asterisk.
Then ask who can edit that file and who decides what it says about AI. If you maintain the site, that may be you, and the question becomes who signs off. Having that answer before the control arrives is the useful part.
What to read next
Custom-coded vs WordPress: a real-world speed comparison
The same local business, the same services, one website replaced by another, both measured with Google PageSpeed Insights on the same day. Not a laboratory test, but the closest the real world gets.
Read articleAstro or Next.js for a business website?
My rule for choosing between Astro and Next.js, and what two of my own sites send on first load, measured with PageSpeed Insights on 25 October and 5 November 2026.
Read articleHow much of my Next.js article pages' JavaScript is my own code?
On 5 November 2026 I ran the Next.js bundle analyser on a copy of my site. On an article page, about 81 percent of the JavaScript loaded up front is Next.js and about 3 percent is code I wrote.
Read articleQuestions local business owners usually ask next
Want me to look at your site?
Send me the address and I'll review the site you've got now. The Free Website Review looks at what's working, what's getting in the way and the clearest next step, and it's free.