Commerce Website AI Crawler Readiness Guide

1. Define the public and private boundary

Public product pages, cases, guides, legal pages and machine files should return stable canonical responses. Account data, admin consoles, payment confirmation, uploaded evidence, webhook operations and privileged reports should require authentication or return a non-indexable response.

Do not rely on robots.txt to protect secrets. Robots directives control cooperative crawling; authentication and authorization protect data.

Inventory every route as public-indexable, public-noindex, authenticated or not served.
Keep private and transactional routes out of sitemap, llms and public navigation.
Return X-Robots-Tag or meta noindex on public operational pages when needed.

2. Establish one canonical URL system

Choose one URL for each public concept. Internal links should use that URL, duplicate legacy addresses should permanently redirect to it, and the same address should appear in canonical, hreflang, Open Graph, structured data, sitemap and machine files.

Canonical pages return 200 without a redirect chain.
Legacy public .html pages return one permanent redirect to the clean URL.
Sitemap lists only canonical, indexable, 200 pages.

3. Allow legitimate crawlers deliberately

Googlebot and Bingbot cover conventional search. OAI-SearchBot is used for OpenAI search discovery; PerplexityBot is used for Perplexity search. Follow current official user-agent and IP verification guidance instead of trusting a user-agent string alone.

Cloudflare WAF, bot protections and rate limits should be reviewed with logs. Broad allow rules are unsafe; verified crawler traffic should receive access to public pages within reasonable limits.

Test robots.txt and a representative public URL with each target crawler policy.
Log crawler UA, URL, status and timestamp without sensitive query payloads.
Review 403, 429 and challenge responses before changing WAF rules.

4. Align visible pages and machine-readable files

Structured data must reflect visible content. Organization, Brand, Service, Offer, Breadcrumb and FAQ nodes should use current names, URLs, prices and dates. Do not publish hidden reviews, ratings or outcomes.

llms.txt, product feeds, GEO indexes, knowledge manifests and well-known files should point to the same canonical entity and public evidence pages.

Parse every JSON and JSON-LD block in CI.
Reject machine-file URLs that resolve to 404, duplicate or private routes.
Keep dateModified tied to a real content change.

5. Measure discovery and release safely

Use Google Search Console and Bing Webmaster Tools for indexing, queries and crawl errors. Use server logs for crawler status. Track AI and organic referrals separately from demo, audit, checkout and completed-order events.

IndexNow can notify participating search engines after a verified production deployment. A dry-run file and URL set should be reviewed before sending a real notification.

Record the deployed version and canonical URL inventory.
Run desktop, mobile, bilingual, link, schema and HTTP-status QA.
Keep a rollback version and do not treat crawler submission as indexing success.

Primary sources and references

These links define platform behavior or security guidance. They are sources, not endorsements of TC SiteOS.

Make the public product easy to crawl without exposing private operations.