Your developer asks whether to block AI bots. Sales wants the company to appear in AI search. Someone else wants to keep the website out of model training. A single “allow AI” switch can't express all three concerns.
Start with the specific provider and purpose. Some providers offer separate controls for search and model training; others bundle uses differently. Write down the outcome you want before changing robots.txt or a firewall rule. This guide explains the distinctions and a review process—it doesn't change your website settings.
Search access and training preferences are different decisions
Search access concerns whether a system can retrieve your pages for a search experience. A training preference concerns whether collected material may be used to develop models. Neither is the same as a person clicking through to your site.
OpenAI documents independent controls for OAI-SearchBot and GPTBot. OAI-SearchBot supports ChatGPT search; GPTBot crawls material that may be used for training foundation models. A publisher can allow the former while disallowing the latter. That distinction makes “we blocked OpenAI” too vague for a useful change log.
Record the exact user-agent token and the reason for the decision. Don't extend one provider's terminology to every assistant. Product documentation changes, so check the current source before implementation.
Which control matches your goal?
| Goal | What to review | Important limit |
|---|---|---|
| Keep public pages eligible for ChatGPT search | OAI-SearchBot rules and actual request access | Access doesn't guarantee selection, a citation or a recommendation. |
| Express an OpenAI training opt-out | GPTBot rules | This is separate from the search setting. |
| Control Google Search crawling, including its AI search features | Googlebot and relevant Search controls | Don't use Google-Extended as a substitute for Search controls. |
| Manage certain Gemini training and grounding uses | Google-Extended's documented scope | It covers more than training; read the affected-products description. |
| Protect customer or confidential information | Authentication and appropriate access controls | robots.txt is not a security boundary. |
Google describes Google-Extended as a product token governing specified training and grounding uses in Gemini products. It isn't a separate HTTP user agent and doesn't determine inclusion or ranking in Google Search. Blocking it is therefore not simply a training-only choice.
For AI Overviews and AI Mode, Google's website guidance points to Googlebot for crawling and Search preview/indexing controls for limiting what appears. Review the effect on ordinary search as well as AI features before making a change.
A small example your developer can review
The following is an illustrative OpenAI-specific policy, not a complete replacement for your site's existing file:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /It expresses a preference to allow search crawling and disallow GPTBot crawling. It doesn't grant access through a firewall, undo historical collection or guarantee a particular search answer. OpenAI separately documents ChatGPT-User for user-initiated actions, where robots.txt rules may not apply; don't assume the two groups above control every possible visit.
Before using an example like this, have the site owner inspect the existing groups, wildcard rules and policies on the exact hostname. A marketing site, application and help centre may have different requirements. Keep a backup and a clear rollback plan.
Test delivery, not just the text of a rule
A file can say “Allow” while another layer rejects the request. Ask your developer to check the public page, response status and relevant CDN or firewall controls. Confirm what happened in logs rather than assuming the rule worked because it saved successfully.
Google's robots.txt introduction explains that the file manages crawler requests and is not a reliable way to keep material private or remove a page from search. Sensitive content belongs behind suitable access controls. If your actual goal is de-indexing, ask for an indexing review rather than treating a crawl block as equivalent.
For a fictional example, imagine ExampleCo allows the search crawler on its marketing domain but its security service still presents a challenge. The configuration file alone doesn't prove the page was fetched. The useful next step is checking the rejected request with the developer, not publishing ten more articles.
Use the provider's current verification guidance when evaluating traffic. A user-agent name in a log is a clue; don't automatically label every matching request genuine. Document the verification method and keep uncertain traffic separate.
Record four separate outcomes
- Policy: which provider-specific preference did you save, on which hostname and date?
- Access: did a verified request actually reach the intended public page?
- Answer observation: did a later sampled answer mention, recommend or link the brand?
- Business result: did measurable relevant visits or useful enquiries follow?
These observations answer different questions. A crawl isn't a visit, a citation isn't an endorsement, and a subsequent answer change doesn't establish that your settings caused it. Use our AI visibility measurement guide for the answer-side definitions and our ChatGPT tracking routine for recording comparable checks.
Your next step is a short policy note: the public content you want discoverable, the uses you want to limit, the exact controls involved and the person who will verify the result. That gives marketing and engineering a shared decision to review, rather than a vague argument about blocking “AI.”
