In a review of 38 complete SaaS robots.txt snapshots, 31 (81.6%) had no group naming OAI-SearchBot. None of those 31 blocked that crawler at the homepage path through the robots directives we evaluated.
For an SEO or AEO audit, that's a useful distinction. Searching a robots file for an AI crawler's name tells you whether it has a named group. It doesn't tell you whether the crawler is allowed to fetch a particular URL.
The reverse mistake is possible, too. Nine files named GPTBot. Eight of those nine still permitted it at the homepage path. A bot appearing in robots.txt isn't automatically a block.
This study examined a deliberately selected sample of 50 SaaS website hosts across ten categories. We obtained complete text for 38 and left 12 unclassified. These are observations about the retrieved files, not estimates for all SaaS websites. The complete source table and method are below.
What the audit found
| Observation | Count | Share |
|---|---|---|
| No group naming OAI-SearchBot | 31 of 38 complete files | 81.6% |
| OAI-SearchBot permitted at the homepage path | 38 of 38 complete files | 100.0% |
| Group naming GPTBot | 9 of 38 complete files | 23.7% |
| GPTBot permitted at the homepage among files naming it | 8 of 9 files | 88.9% |
| GPTBot blocked but OAI-SearchBot permitted at the homepage | 1 of 38 complete files | 2.6% |
Source: original analysis of public robots.txt text retrieved November 11, 2026, Pacific time. The tested path was / on each listed HTTPS host. Source crawl labels ranged from “today” to “2 weeks ago.” These were retrieved snapshots, not a simultaneous live crawl. Twelve unavailable files are excluded from the percentages.
Of the 31 files without a named OAI-SearchBot group, 30 supplied a wildcard group. The remaining file, for Okta's listed host, had neither a matching named group nor an active wildcard group. Under the evaluation rules, no restriction applied to that bot. Commented-out lines were ignored.
Why the crawler's name isn't the answer
Robots rules are evaluated for a crawler and a path together. Under RFC 9309, a matching named group takes precedence. If there isn't one, the crawler uses the wildcard group when available. With no applicable group, no rules apply. A path without a matching restriction is permitted.
That means a missing OAI-SearchBot entry can be entirely consistent with allowing its crawl. It also means a named GPTBot entry can contain an allowance, partial restrictions, or a full block. The effective directives determine the result.
In this sample, a rule that flagged every missing OAI-SearchBot name as “search blocked” would have produced 31 false positives for the homepage test. A rule that treated every GPTBot name as a block would have misclassified eight of the nine files that named it. Those are results for these two hypothetical audit shortcuts, not a claim about any particular commercial audit tool.
Three files show the distinction
Loom separated the two OpenAI crawlers. Its retrieved file contained a GPTBot-specific Disallow: /. OAI-SearchBot wasn't named and fell back to a wildcard group with Allow: /. The homepage result was blocked for the training crawler and permitted for the search crawler.
Postman named both crawlers in a permissive group. In its retrieved file, GPTBot and OAI-SearchBot shared a group that allowed the homepage while restricting specific URL patterns. Finding GPTBot in that file wasn't evidence of a homepage block.
Slack used the wildcard route. Its retrieved file didn't name either OpenAI crawler. It restricted paths such as /messages, but those rules didn't match /. Both crawlers were permitted at the tested path.
These examples describe the snapshots we examined. They aren't judgments about those companies' wider AI policies, and their files can change.
Search and training are separate decisions
OpenAI's crawler documentation distinguishes OAI-SearchBot, which supports ChatGPT search, from GPTBot, which gathers content that may be used for model training. It says the settings are independent. A site can permit search crawling while disallowing the training crawler.
That distinction belongs in an AEO audit. “AI bots allowed” is too broad a field to capture the decision. Record the specific crawler, the URL being checked, the applicable group, and the resulting permission.
Also keep robots permission separate from actual access. A CDN challenge, firewall rule, authentication requirement, or server error can prevent retrieval even when the file permits it. A permitted homepage tells you nothing conclusive about a product page or article. It doesn't establish indexing, citation eligibility across every system, or improved rankings.
For examples of how to configure each policy, see our guide to separating AI search access from model training.
A better check for your next client audit
- Choose the URLs that matter. Test the actual product, comparison, documentation, and article URLs on their correct hosts. Keep the homepage as one observation.
- Evaluate each crawler separately. Identify its applicable group, then resolve the matching Allow and Disallow directives. Preserve the winning rule or record that no restriction matched.
- Check access outside robots.txt. Review relevant CDN controls and verified crawler logs. OpenAI publishes its crawler IP ranges alongside its documentation. A request made with a copied user-agent string alone doesn't prove that the real crawler can get through.
- Report permissions and outcomes in separate fields. An allowed URL, a successful crawl, and an observed citation are different findings. Record the evidence and date for each.
Be careful when adding a named bot group to an existing file. Don't assume that restrictions from the wildcard group will carry over. Check the resulting rules against the pages you intend to protect and the pages you intend to make crawlable.
Methodology and limits
Sample. Fifty intentionally selected SaaS and software-platform website hosts, five in each of ten editorial categories. This is a convenience sample, not a traffic-ranked list or random sample. Some sites share a corporate parent, and categories overlap. The unit is a host's robots file, not an independent company. No WildSEO performance data or customer records were used.
Retrieval. We requested each listed HTTPS /robots.txt URL through a web retrieval service on November 11, 2026, Pacific time. We retained its returned text and source crawl-age label. For long files, we retrieved additional line ranges and confirmed that every reported line was present before classification. Thirty-eight files were complete. Twelve requests returned retrieval errors or access restrictions, so we recorded them as unknown. A retrieval-service failure is not evidence that a website blocks a particular crawler. Cached source text may differ from the live file, and relative crawl labels aren't precise collection timestamps.
Classification. We checked active User-agent groups for the exact product tokens GPTBot and OAI-SearchBot, case-insensitively. We ignored comments, combined repeated matching groups, and used the wildcard group only when no named match existed. If neither existed, no rules applied. We evaluated Allow and Disallow path patterns against /, including wildcard and end-of-path syntax, using the most specific matching rule and allowing on an equivalent tie. Empty directives didn't restrict access. The protocol basis is RFC 9309's path rules. The implementation was checked against explicit blocks, wildcard fallback, repeated groups, empty directives, comments, and allow/block ties.
Scope. “Permitted” means the evaluated robots directives did not block the root path. It is not a test of the actual homepage response, redirect destination, full-site crawlability, or a provider's compliance. We didn't measure rankings, traffic, citations, or training use. Other mechanisms, including Content-Signal preferences, were not converted into Allow/Disallow rules. This is therefore not a complete assessment of a site's AI content-use policy.
Reproduction. Use the 50 source URLs below, keep unavailable responses unknown, select each bot's applicable group, and evaluate /. Count the named-group and path-permission fields separately. Percentages use the 38 complete files, except the 8-of-9 statistic, which uses only files naming GPTBot. A fresh retrieval can produce different results because files change.
Source data: all 50 selected hosts
“Named” means an active User-agent group names that exact bot. “Permitted” refers only to the homepage path under the evaluated robots directives. The age column reproduces the retrieval service's source crawl label, relative to the retrieval date. Each host links to its public robots file.
| Host / source | Category | OAI-SearchBot group | OAI root | GPTBot group | GPT root | Source age |
|---|---|---|---|---|---|---|
| www.salesforce.com | CRM | Wildcard | Permitted | Wildcard | Permitted | today |
| www.hubspot.com | CRM | Wildcard | Permitted | Wildcard | Permitted | today |
| www.pipedrive.com | CRM | Unknown | Unknown | Unknown | Unknown | Unavailable |
| www.zoho.com | CRM | Wildcard | Permitted | Wildcard | Permitted | today |
| www.close.com | CRM | Unknown | Unknown | Unknown | Unknown | Unavailable |
| asana.com | Work management | Wildcard | Permitted | Wildcard | Permitted | today |
| monday.com | Work management | Named | Permitted | Wildcard | Permitted | today |
| clickup.com | Work management | Wildcard | Permitted | Wildcard | Permitted | 3 days ago |
| www.notion.com | Work management | Unknown | Unknown | Unknown | Unknown | Unavailable |
| www.airtable.com | Work management | Unknown | Unknown | Unknown | Unknown | Unavailable |
| slack.com | Collaboration | Wildcard | Permitted | Wildcard | Permitted | today |
| www.zoom.com | Collaboration | Unknown | Unknown | Unknown | Unknown | Unavailable |
| miro.com | Collaboration | Wildcard | Permitted | Wildcard | Permitted | today |
| www.loom.com | Collaboration | Wildcard | Permitted | Named | Blocked | 3 days ago |
| www.figma.com | Collaboration | Unknown | Unknown | Unknown | Unknown | Unavailable |
| www.zendesk.com | Customer support | Wildcard | Permitted | Wildcard | Permitted | last week |
| www.intercom.com | Customer support | Wildcard | Permitted | Wildcard | Permitted | last week |
| www.freshworks.com | Customer support | Wildcard | Permitted | Wildcard | Permitted | today |
| www.helpscout.com | Customer support | Unknown | Unknown | Unknown | Unknown | Unavailable |
| www.gorgias.com | Customer support | Unknown | Unknown | Unknown | Unknown | Unavailable |
| mailchimp.com | Marketing | Wildcard | Permitted | Wildcard | Permitted | 3 days ago |
| www.klaviyo.com | Marketing | Named | Permitted | Named | Permitted | 5 days ago |
| www.activecampaign.com | Marketing | Wildcard | Permitted | Wildcard | Permitted | 2 weeks ago |
| www.braze.com | Marketing | Wildcard | Permitted | Wildcard | Permitted | 3 days ago |
| www.brevo.com | Marketing | Wildcard | Permitted | Wildcard | Permitted | today |
| www.shopify.com | Commerce and websites | Wildcard | Permitted | Wildcard | Permitted | today |
| www.bigcommerce.com | Commerce and websites | Wildcard | Permitted | Wildcard | Permitted | last week |
| webflow.com | Commerce and websites | Wildcard | Permitted | Wildcard | Permitted | today |
| www.squarespace.com | Commerce and websites | Wildcard | Permitted | Named | Permitted | last week |
| www.wix.com | Commerce and websites | Wildcard | Permitted | Wildcard | Permitted | today |
| github.com | Developer tools | Named | Permitted | Named | Permitted | today |
| about.gitlab.com | Developer tools | Named | Permitted | Named | Permitted | 3 days ago |
| www.atlassian.com | Developer tools | Wildcard | Permitted | Wildcard | Permitted | yesterday |
| www.postman.com | Developer tools | Named | Permitted | Named | Permitted | 3 days ago |
| sentry.io | Developer tools | Wildcard | Permitted | Wildcard | Permitted | 6 days ago |
| www.datadoghq.com | Infrastructure | Unknown | Unknown | Unknown | Unknown | Unavailable |
| newrelic.com | Infrastructure | Wildcard | Permitted | Wildcard | Permitted | 5 days ago |
| www.cloudflare.com | Infrastructure | Wildcard | Permitted | Named | Permitted | today |
| www.mongodb.com | Infrastructure | Wildcard | Permitted | Wildcard | Permitted | last week |
| www.elastic.co | Infrastructure | Unknown | Unknown | Unknown | Unknown | Unavailable |
| www.okta.com | Security | No applicable group | Permitted | No applicable group | Permitted | yesterday |
| auth0.com | Security | Wildcard | Permitted | Wildcard | Permitted | 4 days ago |
| 1password.com | Security | Wildcard | Permitted | Wildcard | Permitted | today |
| bitwarden.com | Security | Unknown | Unknown | Unknown | Unknown | Unavailable |
| www.vanta.com | Security | Wildcard | Permitted | Wildcard | Permitted | 2 weeks ago |
| www.workday.com | HR and payroll | Wildcard | Permitted | Wildcard | Permitted | yesterday |
| www.bamboohr.com | HR and payroll | Named | Permitted | Named | Permitted | 2 days ago |
| gusto.com | HR and payroll | Unknown | Unknown | Unknown | Unknown | Unavailable |
| www.rippling.com | HR and payroll | Wildcard | Permitted | Wildcard | Permitted | yesterday |
| www.deel.com | HR and payroll | Named | Permitted | Named | Permitted | today |
How to cite this study
Tim White / WildSEO. “AI Search Access: 31 of 38 SaaS Sites Didn’t Name OAI-SearchBot.” The Trailhead, November 12, 2026. Retrieval date: November 11, 2026, Pacific time. Cite the source table and retain the sample, root-path scope, and retrieved-snapshot qualification.
In 38 complete SaaS robots.txt snapshots retrieved on November 11, 2026, 31 (81.6%) had no OAI-SearchBot-specific group. All 31 permitted the homepage path under the evaluated robots directives. The sample was purposive, and 12 additional selected hosts could not be classified.
