Lesson 2 of 28 · AI crawler access
No accidental 'block everything' rule
What it is
Two lines — 'User-agent: *' followed by 'Disallow: /' — tell every crawler to stay away from your whole site. It's often left behind when a site moves from a test address to the real one.
Why it matters
This single mistake removes you from Google and every AI tool at once, and nothing on the site looks broken to you.
✓ Good example
The live site's robots.txt says 'Allow: /' or only hides specific folders.
✗ Common mistake
The web designer launched the site but forgot to remove the staging rule, so it has been invisible for months.
You pass when: There is no 'User-agent: * / Disallow: /' on the live site.
Check it now with our free Robots.txt generator.