OAI-SearchBot blocked: what it may mean for AI search visibility
OAI-SearchBot is the crawler that builds OpenAI's search index. If you block it, ChatGPT's search-backed answers will struggle to find you.
Your robots.txt disallows OAI-SearchBot, either under an explicit User-agent: OAI-SearchBot group or via a blanket User-agent: * Disallow rule. OAI-SearchBot honors robots.txt, so OpenAI's search-style index cannot fetch and re-fetch your pages for inclusion.
OAI-SearchBot is distinct from GPTBot. It powers the live search index ChatGPT consults when answering questions, not future model training. Blocking OAI-SearchBot is effectively opting out of being cited by ChatGPT's search-backed answers, which is a separate traffic surface from classic Google Search.
- A broad "block all AI" policy grouped OAI-SearchBot with GPTBot and ClaudeBot without distinguishing training from retrieval.
- A User-agent: * Disallow: / rule sweeps up every crawler including OAI-SearchBot.
- A CDN bot-management rule treats OAI-SearchBot as unwanted traffic and returns 403 at the edge.
- A staging robots.txt with Disallow: / was promoted to production.
- The policy was set deliberately but without weighing the cost of AI citations.
- Open AI Crawler Checker with a representative URL on your site.
- Find the OAI-SearchBot row in the crawler matrix.
- Check whether the match comes from OAI-SearchBot specifically or from a catch-all group.
- Confirm that the page also returns 200 with real text and no noindex, so OAI-SearchBot could actually use it if allowed.
- 1
Add an explicit Allow group for OAI-SearchBot
Add User-agent: OAI-SearchBot with Allow: / above or alongside your catch-all group so there is no ambiguity about what is allowed.
- 2
Decide training vs. search once, globally
If you want to opt out of training only, keep GPTBot disallowed and allow OAI-SearchBot. If you want to opt out of AI entirely, disallow both, but accept that ChatGPT will not cite you.
- 3
Verify the CDN is not overriding robots.txt
Check that your CDN or WAF is not blocking the OAI-SearchBot user-agent at the edge. Edge blocks override robots.txt and return 403.
- 4
Re-test with the AI Crawler Checker
After updating robots.txt and any WAF rules, re-run the checker against the same URL. OAI-SearchBot should now show as allowed.
Is OAI-SearchBot the same as GPTBot?
No. GPTBot is OpenAI's training crawler. OAI-SearchBot powers the search index that ChatGPT consults at answer time. They are controlled independently in robots.txt.
Should I allow OAI-SearchBot?
If you want your pages to be citable inside ChatGPT's search-backed answers, yes. If you explicitly want to stay out of AI search, disallowing OAI-SearchBot is a valid choice, it just has a real visibility cost.
How do I test it?
Use the AI Crawler Checker. The per-crawler matrix tells you, for a given URL, whether OAI-SearchBot is allowed or blocked and which robots.txt group applied.
Ready to diagnose your URL?
AI Crawler Checker runs the exact checks discussed above.