A shared AI-training policy token is gaining provider support. Learn what `__usage_ai-training` means, how it differs from crawler access and grounding, and how to roll it out without accidentally blocking search discovery.
Crawler access, model training, and AI grounding are different content-use decisions and should be configured separately.
As of November 4, 2026, `__usage_ai-training` is a proposed shared policy token for expressing an AI-training preference in `robots.txt`. It is described in an Internet-Draft, not a final IETF standard. The token is separate from crawler-access rules: it does not itself block a request, protect private data, or guarantee that every provider will honor the preference. Anthropic has already started implementing support according to the draft author, OpenAI says implementation is planned for the coming months, Google targets early 2027, and Microsoft says it will begin honoring the new preferences from February 1, 2027 as it recrawls pages.
Substantive update — November 4, 2026: This guide corrects the earlier crawler classifications and adds the shared training-preference work announced by OpenAI, Google, and Microsoft on October 30, 2026.
OpenAI announced that it will honor a common AI-training preference signal in `robots.txt`. It says it aims to implement the signal in the coming months and will publish a further update when its implementation is ready. OpenAI also says the shared signal will not replace its existing crawler controls. Read OpenAI's announcement.
Google announced plans to support the shared policy token in early 2027. Google says that if a site expresses an opt-out through the shared token and does not explicitly specify `Google-Extended`, it plans to interpret the choice as if `Google-Extended` were disallowed. Google also says `Google-Extended` does not affect inclusion in Google Search or act as a Google Search ranking signal. Read Google's Search Central announcement and its crawler documentation.
Microsoft announced a shared AI-training preference for `robots.txt` plus separate page-level controls for AI training and AI grounding. Microsoft says it will begin honoring the new preferences from February 1, 2027 as it crawls. The March 1, 2027 transition deadline is specifically important for sites that still rely on the older `NoArchive` or `NoCache` preferences. Read Microsoft's publisher guidance.
Anthropic standards lead Kevin Kelley, the draft author, also wrote to the IETF AI Preferences mailing list that Anthropic had already started implementing support and that rollout across services would take time. The note did not give a completion date. See the IETF mailing-list announcement. These are provider commitments and rollout plans, not proof that the shared token is already enforced consistently across every crawler today.
The current Internet-Draft uses an ordinary robots.txt group keyed by a policy token:
```text User-agent: __usage_ai-training Disallow: / Allow: /press-kit/ ```
In the draft's example, the preference is against AI training for the whole site except `/press-kit/`, where the preference is in favor. The token is not a crawler name and does not need to appear in an HTTP request's `User-Agent` header. The draft says a consumer checks the policy token and path in the published robots.txt file.
A specific `User-agent: __usage_ai-training` group is required. The draft says a wildcard `User-agent: *` group must not be interpreted as matching this policy token. If no group for the token exists, the draft says no shared training preference has been expressed.
The signal linked by OpenAI is `draft-kelley-aipref-training-preference-signal-00`, an individual Internet-Draft last updated October 30, 2026. The IETF Datatracker explicitly says the document is not endorsed by the IETF and has no formal standing in the IETF standards process; it is listed as an active individual draft and is due to expire on May 3, 2027 unless updated or replaced. The wider AI Preferences working-group work is continuing. Treat the syntax as work in progress and re-check the draft before deploying it. Read the current draft status and text.
These terms describe different operations. A publisher should decide what it wants for each one instead of treating every AI-related user agent as the same kind of bot.
| Purpose | What it means | Examples and controls | | --- | --- | --- | | Search crawling | A provider fetches public URLs for search discovery or indexing. | OpenAI `OAI-SearchBot`; Anthropic `Claude-SearchBot`; Perplexity `PerplexityBot`; Google `Googlebot`. Use the provider-specific crawl groups. | | Model training | Content may be used to modify the learned parameters of a generative model. | OpenAI `GPTBot`; Anthropic `ClaudeBot`; the proposed shared `__usage_ai-training` preference for participating providers. | | AI grounding | Content is used as input to help form a generated response. | Google documents `Google-Extended` for both Gemini training and grounding. Microsoft documents separate page-level AI-grounding preferences. | | User-triggered fetch | A service fetches a page because a user asks for it or invokes an action. | OpenAI `ChatGPT-User` and Anthropic `Claude-User`. These are not equivalent to automatic search crawlers; robots.txt treatment may differ. |
OpenAI explicitly distinguishes `OAI-SearchBot` (ChatGPT Search) from `GPTBot` (model training). It also documents `ChatGPT-User` for certain user-initiated actions and says robots.txt rules may not apply to those requests. See the OpenAI crawler reference.
Anthropic's public crawler documentation distinguishes `ClaudeBot` (training), `Claude-SearchBot` (search), and `Claude-User` (user-initiated retrieval). See Anthropic's crawler guidance.
Google's `Google-Extended` is a product token, not a separate HTTP crawler user-agent. Google says it controls whether content it crawls may be used for future Gemini model training and for grounding in Gemini Apps and related experiences. It does not affect Google Search inclusion or ranking.
| Provider | Published position | Practical implication today | | --- | --- | --- | | OpenAI | Announced October 30 that it will honor the shared signal; implementation is planned for the coming months. | Keep using `GPTBot` for training preference and `OAI-SearchBot` for ChatGPT Search until OpenAI publishes implementation details. | | Google | Announced October 30 that it aims to support the shared token in early 2027. | Keep `Googlebot` rules separate. Understand that `Google-Extended` covers both Gemini training and grounding; the shared opt-out is planned to map to `Google-Extended` disallow when no explicit rule is present. | | Microsoft | Announced October 30 that shared training preferences in robots.txt and page-level training/grounding controls will be honored from February 1, 2027 as pages are crawled. | Use the transition guidance if the site currently relies on `NoArchive` or `NoCache`; the March 1, 2027 deadline is not a universal action for every site. | | Anthropic | The draft author stated on the IETF AI Preferences mailing list that Anthropic had started implementing support; rollout across services will take time. | Keep the existing `ClaudeBot`, `Claude-SearchBot`, and `Claude-User` controls while the shared implementation rolls out. |
| Your goal | Safer starting point | Important caveat | | --- | --- | --- | | Keep ChatGPT Search discovery while opting out of OpenAI model training | Allow `OAI-SearchBot`; disallow `GPTBot`. | `ChatGPT-User` is user-triggered and is not the Search crawler. | | Keep Claude Search discovery while opting out of Anthropic training | Allow `Claude-SearchBot`; disallow `ClaudeBot`. | `Claude-User` is a separate user-initiated fetcher. | | Keep Google Search eligibility | Keep `Googlebot` crawl access available for pages intended to appear in Search. | `Google-Extended` does not control Google Search inclusion or ranking. | | Opt out of Google-Extended uses | Disallow `Google-Extended` or, when supported, use the shared training preference with no explicit `Google-Extended` rule. | Google documents `Google-Extended` as covering both Gemini training and grounding; do not assume the two can be separated through this token. | | Separate Microsoft training and grounding choices | Use the shared training preference and Microsoft's page-level `ai-training` / `ai-grounding` controls when they become effective. | Microsoft says the new preferences begin taking effect February 1, 2027 as pages are crawled. | | Keep private or customer data inaccessible | Use authentication, authorization, and server-side access controls. | `robots.txt` is public and is not an access-control or confidentiality mechanism. |
This example illustrates the separation. It is not a universal provider policy and should be adapted to the site's rights, contracts, and desired product visibility:
```text # Search crawlers — allow only the public content intended for discovery User-agent: Googlebot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Provider-specific model-training control User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / # Shared AI-training preference (Internet-Draft; provider support varies) User-agent: __usage_ai-training Disallow: / ```
Google's documented mapping matters: when the shared token expresses a training opt-out and `Google-Extended` is not explicitly specified, Google plans to treat that as disallowing `Google-Extended`. Because Google-Extended also covers grounding, a publisher who wants Gemini grounding should review that trade-off rather than blindly copy this example.
For Microsoft page-level controls, the published examples include `<meta name="Bingbot" content="ai-training:disallow">` and `<meta name="Bingbot" content="ai-grounding:disallow">`. Microsoft also documents `ai-grounding:disallowextended` to limit grounding to Microsoft-operated experiences rather than third-party grounding services. There is no site-wide Microsoft grounding directive in robots.txt.
1. Inventory content and rights. Separate public marketing pages, licensed material, user-generated content, customer data, and content with contractual restrictions. Do not assume one preference is appropriate for every path.
2. Define the business outcome. Decide independently whether each content class should be crawlable for search, usable for model training, or eligible for AI grounding.
3. Keep search crawlers distinct. Identify each provider's search crawler and training crawler from its current official documentation. Do not use `GPTBot` as a proxy for ChatGPT Search or `ClaudeBot` as a proxy for Claude Search.
4. Keep existing provider-specific rules. The shared signal does not replace the existing controls, and provider rollout dates differ. Do not remove working rules just because the new token has been published.
5. Test syntax and precedence. Validate path matching, `Allow` exceptions, explicit provider groups, and the exact policy-token spelling. Re-check the current draft and each provider's interpretation before publishing.
6. Protect private content technically. Use authentication, authorization, signed URLs, and server-side access checks. A robots.txt disallow is a request to compliant crawlers, not a way to hide a URL.
7. Version and monitor changes. Keep the file in source control, review diffs, monitor verified crawler logs, and re-check provider documentation as implementations ship.
Measure each layer separately. Search crawler requests show that a provider's crawler fetched a URL; they do not prove that the page was indexed, retrieved for a specific answer, or cited. Training-crawler requests show crawler activity, not that a model was trained on the content. Grounding preferences affect a different use case again.
For GEO measurement, combine provider-specific crawler logs with controlled prompt tests, source/citation observations, Search Console and Bing Webmaster Tools data, and attributable referral sessions. Record the provider, product surface, model or mode where known, prompt, locale, date, source URL, and whether the answer actually cited the page. Do not turn crawler hits into a citation-share metric.
Use the Robots.txt Generator & Validator to prepare provider-specific crawl rules and preview the proposed shared training token. The token is clearly labeled as a draft preference and is not evaluated by the URL crawl-rule tester.
Use the AI Visibility Checker to review the site's current OAI-SearchBot crawlability and on-page clarity checks. It does not certify provider compliance or guarantee AI citations.
For an implementation audit covering crawl health, structured data, sitemap hygiene, and GEO page patterns, see HiMat's Technical SEO & GEO service.
For measurement methodology, read Measuring GEO Visibility & AI Search Citations. For server-rendered content and structured data guidance, read GEO Engineering for Next.js and React. For enterprise access policy, see GEO Agentic Context Governance.
No. In the current draft it is a content-use preference token, not a crawl directive. A provider may use the preference to inform its crawling, but the token itself does not replace provider-specific crawl rules.
No. The linked document is an individual Internet-Draft. The IETF Datatracker says it is not endorsed by the IETF and has no formal standing in the standards process. It may change before any final standard.
No. OpenAI documents `GPTBot` for model-training collection and `OAI-SearchBot` for ChatGPT Search. They are separate controls. `ChatGPT-User` is used for certain user-initiated actions and is not the Search crawler.
No. Google says `Google-Extended` does not affect Google Search inclusion or ranking. However, Google plans to map a shared training opt-out to disallowing Google-Extended when no explicit Google-Extended rule is present, and Google-Extended covers both Gemini training and grounding.
It depends on the provider. Microsoft documents separate page-level AI Training and AI Grounding preferences. Google currently documents `Google-Extended` as covering both Gemini training and grounding, so the same separation is not available through that token alone. OpenAI's current crawler controls distinguish training from Search crawling, while the shared signal is specifically about training.
No. robots.txt is a public preference file for compliant crawlers. It is not authentication, access control, a confidentiality boundary, or a licensing agreement. Keep private data behind server-side authorization.
No. OpenAI says the shared signal will not replace its existing crawler controls, and provider support is rolling out at different times. Keep provider-specific rules until each provider documents the behavior you need.
The shared AI-training preference is a useful step toward simpler publisher controls, but it is not a universal switch and it does not collapse crawling, training, grounding, and user-triggered retrieval into one decision. Keep search access and training preferences separate, understand each provider's exceptions, protect private content with real access controls, and update your robots.txt only after checking the current provider documentation.
Explore other service pillars