AI crawlers·Allen AI
AI2Bot
Ai2’s crawler. Ai2 says it explores certain domains to collect web content that is used to train its open language models.
- Operator
- Allen AI
- robots.txt token
- None published
- Follows robots.txt
- Not stated
- IP ranges
- Not published
- In Spreelytics
- Training tab
- Source
- Allen AI’s documentation
What user agent does AI2Bot use?
Allen AI publishes this user agent string for AI2Bot.
Ai2 says the crawler always sends this user agent.
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)How do I block AI2Bot?
Ai2’s crawler page says nothing about robots.txt. It says you can filter or turn away AI2Bot by its user agent string, at your server or CDN.
How do I check that a AI2Bot visit is real?
Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.
Allen AI does not publish IP ranges or a DNS check for AI2Bot, so there is no way to confirm a request really came from it.
How do I see AI2Bot visits on my site?
Your server’s access logs record the user agent of each request, so you can search them for “AI2Bot”.
Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises AI2Bot by its user agent and counts each visit on the Bots page under Training, apart from your visitor numbers.
Sources: allenai.org/crawler
See your own numbers today.
Free up to 100k events a month · no credit card