Free tool
Can AI crawlers read your site?
Enter a page and see which of 36 AI crawlers your robots.txt lets in, from GPTBot and ClaudeBot to PerplexityBot. We also request the page as each bot, which shows when a firewall or CDN turns it away even though robots.txt says yes.
What it checks
Your robots.txt, read the way RFC 9309 tells crawlers to: a bot follows the group that names it, and only falls back to User-agent: * when none does. The longest matching rule wins. We show the line that decided each bot.
Then the page itself, once as a normal browser and once as each AI bot that publishes its user agent. If the bot gets a 403, a challenge page or a different answer, something in front of your site is treating it differently.
What it can't tell you
Whether the bots really visit. Only your server sees that, and analytics that runs in the browser never does, because crawlers don't run its script.
It also can't see rules that check a bot's IP address. Google, Bing and Apple verify theirs that way. We leave them out of the live test. Every bot here has a page in our AI crawler list with its IP ranges and how to block it.