Lesson 5 of 12 · Module 5 · Remove obstacles
Check technical SEO and AI access
What it is
Discovery, crawling, rendering, indexing and ranking are separate. HTTP 200 means served—not guaranteed indexed. Review five URLs: homepage, product, collection, guide and another important page. Record status, sitemap, robots, canonical, intended indexing, mobile issues and unknowns. Sitemaps: inspect /sitemap.xml; Shopify generates them. Verify important public URLs rather than replacing the system casually. Robots: Shopify defaults suit most stores. Do not paste a generic generated file over them. Customize robots.txt.liquid only for a specific understood need and test it. Blocking crawling does not reliably remove a URL from Google's index. A supported noindex directive must be fetched to take effect. Use real access controls for private information. Noindex implementation may need theme customization; unlisted storefront products are not interchangeable with a general indexing control. Canonical: a preferred URL signal for duplicate/similar content, not a command. Google can choose differently. Evaluate variant, collection-path and filter URLs rather than indexing every variation. Mobile: test buying tasks and representative templates in PageSpeed Insights. Review images, unnecessary scripts, layout shifts and app overhead. Lab scores differ from real-user Core Web Vitals; field data may be missing on low-traffic pages. Product data: compare visible price, currency, availability, images and policies with Product/Offer markup and Merchant Center data. AI bots: OAI-SearchBot is for ChatGPT search; GPTBot relates to potential training. Controls are independent. Google-Extended is purpose-specific, not Google Search indexing. Access does not guarantee citations. llms.txt is an experimental convention, not a Google indexing requirement.
Why it matters
Helpful resources • Free: Sitemap auditor, Redirect checker, AI crawler checker below. A robots observation is not proof of indexing or bot activity. • Free external: PageSpeed Insights, Rich Results Test, Merchant Center, Search Console URL Inspection. • Internal: Website Audit, Content Update Log. The log needs configured Search Console access. Sign-in/client access required; check any displayed credit cost. • Sources: Shopify robots, Noindex, Canonicals, OpenAI bots.
Free exercise workbooks
This lesson: URL Diagnostics.
PDF: print and write by hand. Spreadsheet: edit in Excel or import into Google Sheets. Keep completed copies private; answers are not uploaded.
✓ Good example
Google chose a different canonical; the owner investigates duplication and links before changing the page.
✗ Common mistake
Using robots as privacy protection, adding several schema apps or promising llms.txt guarantees placement.
You pass when: Five URLs have notes and owners. Check two product/feed samples where available, assign fixes and document an access policy.
Check it now with our free AI crawler checker.