AI crawlers·Google
GoogleOther
Google’s general-purpose crawler. Google’s product teams use it to fetch public content, for example in one-off crawls for internal research and development, and Google does not tie it to any one product.
- Operator
- robots.txt token
GoogleOther- Follows robots.txt
- Yes
- IP ranges
- Published
- In Spreelytics
- Training tab
- Source
- Google’s documentation
What user agent does GoogleOther use?
Google publishes these user agent strings for GoogleOther.
Google writes W.X.Y.Z where the Chrome version goes, because it changes: match that part with a wildcard. GoogleOther-Image and GoogleOther-Video are separate crawlers that also follow the GoogleOther token.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36How do I block GoogleOther?
Google says its common crawlers, GoogleOther among them, always obey robots.txt when they crawl automatically.
Add this to the robots.txt file at the root of your site:
User-agent: GoogleOther
Disallow: /How do I check that a GoogleOther visit is real?
Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.
Google publishes the IP ranges its common crawlers use. A reverse DNS lookup of a genuine request’s IP address gives a host name under googlebot.com, and a forward lookup of that name returns the same address.
Spreelytics runs this check for you. Every crawl that claims to be GoogleOther is compared with the IP ranges Google publishes, refreshed every 12 hours, and one from outside them counts as spoofed.
How do I see GoogleOther visits on my site?
Your server’s access logs record the user agent of each request, so you can search them for “GoogleOther”.
Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises GoogleOther by its user agent and counts each visit on the Bots page under Training, apart from your visitor numbers.
Sources: developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#googleother, developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
See your own numbers today.
Free up to 100k events a month · no credit card