Claude-SearchBot

What user agent does Claude-SearchBot use?

Anthropic does not publish a full user agent string for Claude-SearchBot, only the name.

How do I block Claude-SearchBot?

Anthropic says its bots honour robots.txt, including the Crawl-delay extension, and that each subdomain needs its own rules. Blocking Claude-SearchBot stops it indexing your pages, which Anthropic says may make your site less visible in Claude’s search results.

Add this to the robots.txt file at the root of your site:

User-agent: Claude-SearchBot
Disallow: /

How do I check that a Claude-SearchBot visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Anthropic publishes one list of IP addresses for all its bots, and says a crawler whose address is on it is from Anthropic.

Spreelytics runs this check for you. Every crawl that claims to be Claude-SearchBot is compared with the IP ranges Anthropic publishes, refreshed every 12 hours, and one from outside them counts as spoofed.

How do I see Claude-SearchBot visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “Claude-SearchBot”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises Claude-SearchBot by its user agent and counts each visit on the Bots page under Indexing, apart from your visitor numbers.

Sources: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler