How to unblock AI crawlers: robots.txt, your CDN, and your firewall

October 2026 · By the LocalCited team · 6 min read

If an AI assistant can't fetch your website, it can't read your services, hours or service area, and it is less likely to cite you. A block can sit in three places: your robots.txt file, your CDN or hosting "bot protection", and a security plugin or firewall. This guide walks through each one, in the order we'd check them.

First, decide what you want to allow

AI companies run separate bots for separate jobs. OpenAI, for example, says OAI-SearchBot is used to show websites in ChatGPT search, and that sites which opt out of it "will not be shown in ChatGPT search answers". GPTBot is the one that collects content that may be used for model training, and OpenAI says each setting is independent. So you can allow search bots and still block training bots. Our robots.txt guide has a table of the main bots and what each company says they do.

For most local service businesses, the useful goal is: let search and assistant bots in, and make a deliberate choice about training bots.

Layer 1: robots.txt

Open https://yourwebsite.com/robots.txt. Look for Disallow: / under User-agent: * or under a named AI bot. To allow the main AI search bots explicitly, a group like this works (keep your existing rules for other paths):

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /

One rule trips people up: a bot that finds a group with its own name follows only that group and ignores the User-agent: * group. If you add a named group, repeat any Disallow lines (such as /wp-admin/) that you still want to apply.

The quickest way to read the result is the free AI Crawler Checker, which lists each AI bot as allowed or blocked.

Layer 2: your CDN (Cloudflare and similar)

A CDN can block a bot before it ever reads robots.txt. Cloudflare documents two separate features:

AI bot policies with Bot Preference Sync. In the Cloudflare dashboard you can set policies for Search, Agent and Training crawlers. When Bot Preference Sync is on and a policy is set to block (or, for Training, Disallow), Cloudflare adds managed lines to the top of your robots.txt. So the robots.txt visitors see may not match the file on your server.
AI Crawl Control. This is the enforcement side: it can block selected AI crawlers outright, whatever robots.txt says.

What to do: in Cloudflare, go to the Security settings, filter by bot traffic, and open the AI bot policies. Check that Search and Agent are not set to block if you want AI assistants to read your site. Then check AI Crawl Control for any crawler-specific blocks. Afterwards, reload /robots.txt in a browser: Cloudflare says managed content appears or disappears within seconds.

Other CDNs and hosts have their own "bot fight" or "block AI scrapers" switches. If you use one, look for whether it separates search bots from training bots, and ask your host if you can't find the setting.

Layer 3: security plugins and firewalls

WordPress security plugins, hosting firewalls and rate limiters can return errors or challenges to bots they don't recognise. Two clues: your robots.txt allows a bot but a checker reports it can't fetch pages, or your server logs show the bot getting 403 or 429 responses.

The reliable fix is to allow the bot by its published IP addresses rather than by user-agent name alone, because names can be faked. OpenAI publishes IP lists for its bots (for example openai.com/searchbot.json for OAI-SearchBot), and other AI companies document theirs in their crawler docs.

Check that it worked

Re-run the AI Crawler Checker and confirm the search bots show as allowed.
Run the AI-Readiness Check on your homepage to confirm it loads and your business details can be read.
Give it time. OpenAI says it can take about 24 hours for its systems to reflect a robots.txt change.

What this won't do

Unblocking makes your site readable. It doesn't make any assistant recommend you. Accurate listings, clear service pages and other sites mentioning you still matter; see how AI assistants choose local businesses.

Sources

OpenAI, "Overview of OpenAI Crawlers" — platform.openai.com/docs/bots (read October 26, 2026)
Cloudflare, "robots.txt setting" (Bot Preference Sync, AI Crawl Control) — developers.cloudflare.com (read October 26, 2026)
Google, "List of Google's common crawlers" — developers.google.com

Want to see where you stand? Run the free AI Crawler Checker and AI-Readiness Check on your own site, or get the $49 audit: we ask ChatGPT about 28 questions a customer might ask about your type of business and city, report whether and how it mentions you, check your website's readiness, and send a prioritized fix list within 3 business days of getting your details. Results vary, and we can't guarantee any AI mention, ranking or recommendation.