Lesson 1 of 7 · Crawling
Block only what should stay private
What it is
robots.txt tells crawlers which folders to skip. A 'noindex' tag tells search engines not to list a page. Use them for admin, login and thank-you pages, not for real content.
Why it matters
One wrong rule can hide your whole site from Google. Note: robots.txt stops crawling, not listing — use noindex for pages you never want in results.
✓ Good example
robots.txt hides /wp-admin/; the thank-you page has a noindex tag.
✗ Common mistake
'Disallow: /' left over from the test site.
You pass when: Service pages are crawlable and private pages are hidden.
Check it now with our free Robots.txt generator.