LocalIs your robots.txt hiding your business from AI search? A plain-English check
Many small business websites block AI crawlers without the owner knowing — sometimes because a plugin or hosting "bot protection" setting was switched on, sometimes because someone copied a robots.txt that blocks everything with "AI" in the name. That can be a reasonable choice for training bots. It's a costly one if it also blocks the bots that let assistants cite and link to you.
Step 1: Look at your robots.txt (2 minutes)
Open https://yourwebsite.com/robots.txt in a browser. It's a public text file. You'll see groups like this:
User-agent: *
Disallow: /wp-admin/User-agent says which crawler the rule applies to (* means all). Disallow lists paths it shouldn't crawl. Disallow: / means "nothing on this site".
No robots.txt file at all (a "not found" page) generally means crawlers are allowed.
Prefer not to read it by hand? The free AI Crawler Checker fetches your robots.txt and lists, bot by bot, what it allows. If robots.txt looks fine but bots still can't reach you, see how to unblock AI crawlers at the firewall or CDN.
Step 2: Know which bots do what
The key idea: AI companies now separate search bots (which find pages to show and link in answers) from training bots (which collect content to train models). You can allow one and block the other.
| robots.txt token | Company | What the company says it's for |
|---|---|---|
OAI-SearchBot | OpenAI | Surfacing websites in ChatGPT search. Sites opted out "will not be shown in ChatGPT search answers" (may still appear as navigational links). |
GPTBot | OpenAI | Crawling content that may be used to train OpenAI's foundation models. |
ChatGPT-User | OpenAI | Visits triggered by a user's request in ChatGPT; OpenAI says robots.txt rules may not apply, and it isn't used to decide search inclusion. |
Googlebot | Google Search, including its AI features (AI Overviews, AI Mode). | |
Google-Extended | A control token (not a separate crawler) for whether Google may use your content to train future Google AI models and for grounding in Google's AI assistant apps and Vertex AI. Doesn't affect Google Search inclusion or ranking. |
Sources: OpenAI, Google, read October 2026.
Step 3: Spot the common problems
Problem A — blocking everything
User-agent: *
Disallow: /This asks every well-behaved crawler, including Googlebot, to stay out. Sometimes it's left over from when the site was being built. Fix it unless you truly want to be invisible.
Problem B — blocking search bots along with training bots
User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /If your goal was only "don't train on my content", this goes further than needed: per OpenAI, blocking OAI-SearchBot keeps your site out of ChatGPT search answers.
Step 4: A sensible starting point
If you want to be eligible for AI search citations but prefer not to contribute to model training, something like this expresses that (adapt the paths to your site):
# Search crawlers: allowed
User-agent: Googlebot
User-agent: OAI-SearchBot
Allow: /
# Training: opted out
User-agent: GPTBot
Disallow: /
# Everyone else: allowed except admin
User-agent: *
Disallow: /wp-admin/
Sitemap: https://yourwebsite.com/sitemap.xmlAbout Google-Extended: blocking it is a trade-off. By Google's description it opts you out of training Google's AI models and grounding in Google's AI assistant apps. A local business that wants to be named in those answers may prefer to leave it allowed. This is a business decision — there's no single right answer.
Changes aren't instant: OpenAI says updates can take about 24 hours to be reflected on its side.
Step 5: Check your firewall too
robots.txt is only a request. Your hosting company, CDN, or a security plugin can block bots outright, regardless of robots.txt. Google lists making sure crawling is allowed "by any CDN or hosting infrastructure" among its fundamentals, and some AI companies publish the IP ranges their bots use so you can allow them in a firewall. If you use a "block AI bots" toggle, check whether it distinguishes search bots from training bots.
What this won't do
Allowing search crawlers makes you eligible to be read and cited. It doesn't make an assistant recommend you. Content, accurate listings, and reputation still matter — see how AI assistants choose local businesses.
Sources
- OpenAI, "Overview of OpenAI Crawlers" — platform.openai.com/docs/bots
- Google, "List of Google's common crawlers" — developers.google.com
- Google Search Central, "AI features and your website" — developers.google.com