AI crawlers·Common Crawl
CCBot
Common Crawl’s crawler. Common Crawl is a non-profit that builds a free, open copy of the web for research and analysis, and says its data, filtered by others, has been a main source of training data for many large language models.
- Operator
- Common Crawl
- robots.txt token
CCBot- Follows robots.txt
- Yes
- IP ranges
- Published
- In Spreelytics
- Training tab
What user agent does CCBot use?
Common Crawl publishes these user agent strings for CCBot.
The second is the older bot’s string, and Common Crawl says it may raise the version number.
CCBot/2.0 (https://commoncrawl.org/faq/)
CCBot/1.0 (+https://commoncrawl.org/bot.html)How do I block CCBot?
Common Crawl says CCBot checks robots.txt before it fetches a page, honours Crawl-delay and nofollow, and backs off when a server returns errors.
Add this to the robots.txt file at the root of your site:
User-agent: CCBot
Disallow: /How do I check that a CCBot visit is real?
Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.
Common Crawl warns that other crawlers pretend to be CCBot. It publishes CCBot’s IP ranges, and a reverse DNS lookup of a genuine IPv4 request gives a host name ending in crawl.commoncrawl.org, which resolves back to the same address. IPv6 addresses have no reverse DNS yet, so check those against the list.
Spreelytics runs this check for you. Every crawl that claims to be CCBot is compared with the IP ranges Common Crawl publishes, refreshed every 12 hours, and one from outside them counts as spoofed.
How do I see CCBot visits on my site?
Your server’s access logs record the user agent of each request, so you can search them for “CCBot”.
Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises CCBot by its user agent and counts each visit on the Bots page under Training, apart from your visitor numbers.
Sources: commoncrawl.org/ccbot, commoncrawl.org/faq, commoncrawl.org/about
See your own numbers today.
Free up to 100k events a month · no credit card