Lesson 1 of 28 · AI crawler access
Let AI search bots read your site
What it is
AI assistants send their own crawlers — like OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-User — to read web pages when someone asks a question. Your robots.txt file decides whether they're allowed in.
Why it matters
If a bot can't read your site, the assistant has nothing to quote and will recommend a competitor whose site it can read.
✓ Good example
robots.txt only blocks private folders like /wp-admin/, and every search and browsing bot can read service pages.
✗ Common mistake
A security plugin added 'User-agent: GPTBot / Disallow: /' and a copy-pasted block list that also caught OAI-SearchBot and PerplexityBot.
Try it: block a bot
User-agent: * Disallow: /wp-admin/ User-agent: OAI-SearchBot Disallow: /
ChatGPT search can read your site: No
You pass when: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User and Googlebot are not blocked.
Check it now with our free AI crawler checker.