How to audit your website for AI crawlability

What the Website page checks

Whether your site is reachable, parse-able, and cite-able for AI systems. Think of it as a crawlability MOT for AI search before you judge Monitoring or Sources.

What it tests

• Domain reachable
• Pages discovered
• robots.txt, sitemap.xml, and llms.txt status
• Metadata and machine-readable signals
• Pass/fail checks per page

When to run a scan

• After any meaningful site change — redesign, migration, CMS update.
• When monitoring results look weak or inconsistent.
• Before assuming a visibility problem is a content problem.

How to use the results

1. Fix foundational blockers first — reachability, wrong domain, missing files.
2. Then check whether crawlable pages are also cited pages in Sources.
3. Re-run Monitoring to see if visibility moves.

This sequence stops you from optimizing pages AI still can't reach and helps the rest of your workflow stay grounded in real site conditions.

Keep these files healthy

• robots.txt — don't accidentally block AI crawlers.
• sitemap.xml — fresh and complete.
• llms.txt — describe the pages you want AI systems to prioritize.

Next steps

• Check citation coverage in Sources.
• Re-run Monitoring after fixes.