AI crawlers
Every AI crawler, by name.
The 62 crawlers Spreelytics recognises, from 21 companies. Each one its company documents has a page: what it does, its user agent, how to block it and how to tell a real visit from a fake, from the company’s own documentation. Spreelytics counts their visits with a small middleware on your server.
17 bots
AI answers
An assistant fetches the page live, to answer what someone just asked it.
15 bots
Indexing
A search or answer engine crawls the page so it can show or cite it later.
19 bots
Training
A crawler collects pages as training data for AI models.
11 bots
Other
A known AI crawler with no clearer purpose. Counted under Training.
The list.
By company. A bot with no page is one its company has not documented.
OpenAI
Anthropic
Perplexity
- Google-Agent
- GoogleAgent
- Google-NotebookLM
- Google-GeminiNotebook
- Google-Read-Aloud
- Googlebot
- Google-InspectionTool
- GoogleOther
- Google-Extended
- Google-CloudVertexBot
Microsoft
Mistral
Apple
Amazon
DuckDuckGo
Meta
xAI
- xAI-SearchBot
- Grok-DeepSearch
- GrokBot
- xAI-Bot
- xAI-Grok
- xAI-Web-Crawler
- Grok
Moonshot AI
ByteDance
- TikTokSpider
- Bytespider
- Doubaobot
Baidu
- Baiduspider
- ERNIEBot
- YiyanBot
Alibaba
- Qwen-User
- QwenBot
- TongyiBot
- AliyunBot
Zhipu AI
- ChatGLM-Spider
DeepSeek
Cohere
- cohere-ai
- cohere-training-data-crawler
Allen AI
You.com
Common Crawl
Opting out of AI training.
These are the bots and tokens whose companies say what they collect may be used to train AI models. One robots.txt group can name them all. Read each one’s page before you block it: some control more than training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: KimiBot
User-agent: Bytespider
User-agent: DeepseekBot
User-agent: CCBot
Disallow: /FAQ
About AI crawlers.
What people ask before they block one, or start counting them.
What is an AI crawler?
A bot that fetches web pages for an AI company. Some fetch a page live because someone asked an assistant a question, some crawl for the index an AI search engine answers from, and some collect pages as training data for models.
How do I block an AI crawler?
Add a group for its token to the robots.txt file at the root of your site, with Disallow: /. Each bot’s page gives its token and what its company says about robots.txt. Some bots that fetch a page because a person asked for it do not follow robots.txt, or have no token at all, and their pages say so.
Can a bot pretend to be GPTBot or ClaudeBot?
Yes. A user agent is text the client chooses, so anyone can claim any name. Many AI companies publish the IP ranges their crawlers use, and some document a reverse DNS check as well: each bot’s page says how to check it. Spreelytics checks crawls from most of the bots with published ranges against them, and each bot’s page says whether it does.
Why do AI crawlers not show up in my analytics?
Most analytics tools count visits with a script that runs in the browser, and a crawler that reads your HTML without running that script never triggers it. Your server still sees every request, which is where the Spreelytics middleware, @spreelytics/ai-crawl, counts them.
See your own numbers today.
Free up to 100k events a month · no credit card