robots.txt AI Training Opt-Out: We Scanned 332 Egyptian Sites Before the 2027 Shared Signal
Short answer: On 30 October 2026 Google said it will support, in early 2027, a shared signal in robots.txt that lets a site owner refuse the use of its content for generative AI training across several companies with just two lines, and Microsoft said it will honor it from 1 February 2027. On 3 November 2026 we scanned the robots.txt files of 332 Egyptian domains from the Tranco list: only 152 serve a valid file, only 17 block any AI crawler, and none uses the new shared token yet. For most Egyptian businesses the right move is not a blanket block but separating training from visibility in AI answers.
The robots.txt AI training question is now a marketing decision, not just a technical one: this small file at the root of your site decides whether your pages are used to train models and whether your brand appears in AI assistant answers. In this article we explain what changed in October 2026, share the results of an original scan of Egyptian websites by the Smart One Group team, and give a decision framework plus a 7-step plan before February 2027.
What did Google and Microsoft announce on 30 October 2026?
Google published a Search Central post titled "Supporting a shared AI training control for websites". It says site owners will be able to use a shared policy token in robots.txt to set training preferences across participating AI providers, building on the IETF AIPREF working group. When a site opts out of training with this token and does not explicitly name Google-Extended, Google will treat it as a Google-Extended disallow, while each company's own controls keep working.
The Google crawler documentation puts the date at early 2027 and confirms that Google-Extended does not affect inclusion in Google Search and is not a ranking signal, but it does control training of future Gemini models and grounding in Gemini Apps and Vertex AI.
On the same day Microsoft Bing said it will adopt the shared training signal and added a separate page-level grounding preference with three values: Allow, Disallow and DisallowExtended. Microsoft will start honoring the new preferences on 1 February 2027, and after 1 March 2027 it will no longer read NoArchive and NoCache as training or grounding preferences.
What is the __usage_ai-training token?
The __usage_ai-training token is a virtual user-agent name written in a robots.txt User-Agent line; no real crawler sends it, and participating companies read it to learn your stance on generative model training. It comes from an IETF draft submitted by Kevin Kelley of Anthropic on 30 October 2026, an individual -00 draft that expires on 3 May 2027 unless updated.
- Opt the whole site out of training: a User-Agent: __usage_ai-training line followed by Disallow: /.
- Exceptions are possible for a path, for example allowing a press folder with Allow: /press-kit/.
- The wildcard group User-Agent: * does not express this preference according to the draft.
- The token is not a crawl directive; blocking a specific crawler still needs a group with its own name.
The IETF vocabulary draft defines AI training as using an asset to modify the learned parameters of a generative AI model, excluding training that is exclusively for non-generative models.
robots.txt and AI on Egyptian websites: our scan results
On 3 November 2026 at 09:34 Cairo time we fetched robots.txt for all 332 .eg domains in the Tranco top-million list (list 262N9), parsed user-agent groups according to RFC 9309, and also checked for an llms.txt file. A file counted as valid only if it returned status 200 as plain text rather than an HTML page.
| Indicator | Count | Share |
|---|---|---|
| Domains that responded to the robots.txt request | 290 of 332 | 87.3% |
| Valid robots.txt on an Egyptian host | 152 | 45.8% of total |
| Return an HTML page instead of the file | 50 | 15.1% |
| Return a 404 error | 66 | 19.9% |
| Name at least one AI crawler | 19 | 12.5% of valid files |
| Fully block at least one AI crawler | 17 | 11.2% of valid files |
| Block Google-Extended | 11 | 7.2% of valid files |
| Use a Content-Signal line with ai-train=no | 9 | 5.9% of valid files |
| Use the shared __usage_ai-training token | 0 | 0% |
| Declare a Sitemap in the file | 95 | 62.5% of valid files |
| Publish an llms.txt file | 26 | 7.8% of total |
The most blocked crawlers in the sample were CCBot on 15 sites, GPTBot on 14 and ClaudeBot on 13. By zone, only 3 of 87 .gov.eg domains, 3 of 92 .edu.eg domains and 7 of 90 .com.eg domains block an AI crawler.
Five notable patterns
- Blocking usually comes from a default: 8 of the 17 blocking sites use the managed robots.txt generated by Cloudflare, and all of them carry the Content-Signal line with search=yes and ai-train=no.
- Training and search are already separated: 12 of the 14 sites blocking GPTBot do not block OAI-SearchBot, so they refuse training while leaving ChatGPT search visibility open.
- Explicit allow stances: Vodafone Egypt explicitly allows agents such as OAI-SearchBot, Google-Extended, ClaudeBot and PerplexityBot, and Jumia Egypt explicitly allows Google-Extended while blocking only CCBot.
- Probably unintended full blocks: two university sites block every crawler through the wildcard group and allow only Googlebot and Facebook's crawler, which shuts out Bingbot and every other AI search crawler.
- The basics are missing: more than half of the domains do not serve a valid robots.txt at all, and 50 return a full HTML page when the file is requested, a common misconfiguration in some CMS setups.
robots.txt AI training vs visibility: why a blanket block hurts your marketing
The most common mistake we see at Egyptian companies is mixing up three different uses: crawling for classic search indexing, model training, and grounding, which means fetching your pages at answer time with a link back. Refusing training does not hide your site from Google Search, but at Google a Google-Extended block covers both training and grounding in Gemini Apps, while Microsoft separates the two. Appearance in AI Overviews and AI Mode is governed by a separate Search Console control.
For an Egyptian clinic, service company or online store, a mention with a link in an AI assistant answer is now a real discovery channel, as we explained in our article on the official sites section in Google AI Mode. Copying a ready-made file that blocks every agent may protect your content from training, but it also hides your brand from answers.
A three-stance decision framework
| Stance | Best for | What to write in robots.txt |
|---|---|---|
| Fully open | Service firms, clinics and stores that want maximum visibility | Nothing extra, or explicit allows for key agents |
| Refuse training, allow answers | Most companies that publish original content | A __usage_ai-training group with Disallow: / while AI search agents stay allowed |
| Broad lockdown | Paid, sensitive or licensed content | The shared token plus named agent blocks, and password protection rather than the file alone |
A 7-step plan before February 2027
- Open your robots.txt URL and confirm it returns plain text with status 200, not an HTML page or an error.
- List every AI agent named in the file and check whether a plugin or protection service added them without your knowledge.
- Write down your stance using the framework above, with marketing and legal involved.
- Add a __usage_ai-training group as a separate block, never inside a block that also names Googlebot or Bingbot, so search is not blocked.
- Keep existing controls such as Google-Extended and GPTBot, because they are what actually works today until support starts in 2027.
- If you rely on NoArchive or NoCache, review your Bing settings before 1 March 2027.
- Declare your sitemap in robots.txt, review server logs monthly, and re-check after every theme or hosting change.
robots.txt is a published preference, not a security lock; compliant crawlers respect it, while confidential content needs protection at the server and hosting level.
Frequently asked questions
Will opting out of AI training remove my site from Google Search?
No. Google's documentation states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and the shared token will be treated as Google-Extended when Google-Extended is not named explicitly.
When will companies start honoring __usage_ai-training?
Google targets early 2027, and Microsoft set 1 February 2027 to start honoring the new preferences. Until then, each company's own controls are what apply.
Is the wildcard User-Agent: * group enough to refuse training?
No. The draft says the wildcard group must not be read as matching the policy token, and a file without a __usage_ai-training group expresses no preference.
What is the difference between training and grounding?
Training permanently changes a model's parameters using your content, while grounding fetches your content at the moment a user asks a question and can cite it with a link, which is why grounding is tied directly to your brand's visibility.
Does robots.txt protect content from copying?
No. The file is a public statement of preferences that only compliant crawlers follow; sensitive content needs a login or server-level protection.
Do I need an llms.txt file?
llms.txt is an optional plain-text summary of your site aimed at language models; we found one on 26 of 332 Egyptian domains. It does not replace robots.txt, it is not a training preference, and there is no published evidence that it improves rankings.
If you want your robots.txt reviewed and the right balance between protection and visibility chosen, the Smart One Group digital marketing and visibility and web development teams can set up and test the file as part of a monthly follow-up plan. Contact us for an initial review.
