Skip to content
    AI Search & GEO

    How to Verify an AI Crawler in Your Server Logs

    By Tim White•Published •3 min read
    How to Verify an AI Crawler in Your Server Logs

    TL;DR

    • Find the trusted client IP, match it to the provider’s list for that bot, and check the response before changing access rules.

    Your server log says OAI-SearchBot hit the pricing page and got a 403. Before you ask engineering to change the firewall, check whether the request actually came from OpenAI. Anyone can put a crawler's name in a user-agent string.

    Start with an IP address from a trusted log. Match it against the provider's published ranges for the specific bot. Then check what the request got back. Those three steps tell you whether you have enough evidence to investigate a delivery problem.

    1. Which IP should you trust?

    Pick one request to a public URL your team cares about. Record the timestamp and time zone, hostname, path, HTTP status, and user-agent string. Save a request ID if your logs provide one. It's useful for finding the same request in both your edge and application logs.

    If your site sits behind a CDN or proxy, the IP in your application logs may belong to that intermediary. Use the client IP recorded at the edge, or ask your developer which origin field your trusted proxy sets. A forwarded header only helps if you know which trusted proxy stamped it and how the origin accepts it. If you can't establish where the IP came from, mark the request unverified and move on to a better log source.

    2. Does that IP match the right bot?

    OpenAI's crawler documentation links to separate published IP lists for OAI-SearchBot, GPTBot, and ChatGPT-User. Start with the list for the user agent you're investigating. OAI-SearchBot is the search crawler. GPTBot serves a different purpose, and ChatGPT-User can represent a user-initiated visit.

    Use a tool that checks whether an address falls inside a CIDR range, the block of addresses represented by an entry such as 203.0.113.0/24. Record the provider list you used, the time you checked it, and the matching block if there is one. For an older request, today's list may differ from the list in place when it happened. Keep the result uncertain if you lack the historical evidence.

    Use the provider’s own verification method

    If the request claims to be from Google, use Google's verification instructions. Google documents both matching the relevant published ranges and a manual reverse DNS check followed by a forward lookup back to the original IP. Follow the provider's own method for other bots.

    3. What does the response tell you?

    What you foundWhat to do
    Trusted client IP matches the correct list, response is 403 or 429Use the request ID to check whether a firewall rule, rate limit, or origin response explains the result.
    Verified request received a 200Check the response if available. A 200 can still serve a challenge or the wrong page. Keep citation and indexing checks separate.
    No match, missing IP, or uncertain historical rangesLabel it unverified. Keep it out of verified crawler totals and gather better evidence before changing rules.

    A fictional request from the edge logs

    A CDN row shows 203.0.113.42 requesting /pricing, claiming to be OAI-SearchBot, and receiving a 403. The IP comes from a range reserved for documentation. If a real request's trusted client IP matched the search bot's published list, you'd use that row to find the matched security rule. If the only address available belonged to your CDN, you'd ask for the edge log first.

    Give engineering one request they can trace

    • Request: timestamp with time zone, hostname, path, status and request ID.
    • Verification: claimed bot, provider list and list-check date, then matched or unresolved.
    • Question: which rule or service returned this response, and was that outcome intended?

    Send the record through an approved internal channel. Remove client IPs and sensitive URL parameters from anything shared externally. If the request was verified and the rule was intentional, use the guide to choosing crawler controls to confirm whether that policy still fits the site. If engineering adjusts a rule, write down the exact hostname and path it covers. Check for a later verified request to that same URL and compare its response. If no verified request turns up, log the test as pending. A manual visit can show whether the page works in your browser, but it cannot confirm how the server treats that crawler.


    Know where you stand.Know what to do next.

    Start a free trial and see which AI answers leave you out, who shows up instead, and what to fix first.