Google-Agent

What user agent does Google-Agent use?

Google publishes these user agent strings for Google-Agent.

The first is the mobile agent, the second the desktop one. Google writes W.X.Y.Z where the Chrome version goes, because it changes: match that part with a wildcard.

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

How do I block Google-Agent?

Google publishes no robots.txt token for Google-Agent, and says that because a person asked for the fetch, its user-triggered fetchers generally ignore robots.txt.

How do I check that a Google-Agent visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Google publishes the IP ranges its user-triggered fetchers use. A reverse DNS lookup of a genuine request’s IP address gives a host name under google.com or gae.googleusercontent.com, and a forward lookup of that name returns the same address. Google has also begun signing some Google-Agent requests with Web Bot Auth, as https://agent.bot.goog, but not all of them, so keep the IP check as well.

Spreelytics runs this check for you. Every crawl that claims to be Google-Agent is compared with the IP ranges Google publishes, refreshed every 12 hours, and one from outside them counts as spoofed.

How do I see Google-Agent visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “Google-Agent”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises Google-Agent by its user agent and counts each visit on the Bots page under AI answers, as Google, apart from your visitor numbers.

Sources: developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers#google-agent, developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth, developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests