ClaudeBot

What user agent does ClaudeBot use?

Anthropic does not publish a full user agent string for ClaudeBot, only the name.

How do I block ClaudeBot?

Anthropic says blocking ClaudeBot tells it to leave your site’s future pages out of its training data. It also honours Crawl-delay if you would rather slow it down, and each subdomain needs its own rules. Anthropic warns that blocking its IP addresses instead may not opt you out, because it then cannot read your robots.txt.

Add this to the robots.txt file at the root of your site:

User-agent: ClaudeBot
Disallow: /

How do I check that a ClaudeBot visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Anthropic publishes one list of IP addresses for all its bots, and says a crawler whose address is on it is from Anthropic.

Spreelytics runs this check for you. Every crawl that claims to be ClaudeBot is compared with the IP ranges Anthropic publishes, refreshed every 12 hours, and one from outside them counts as spoofed.

How do I see ClaudeBot visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “ClaudeBot”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises ClaudeBot by its user agent and counts each visit on the Bots page under Training, apart from your visitor numbers.

Sources: support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler