Googlebot

What user agent does Googlebot use?

Google publishes these user agent strings for Googlebot.

The first is Googlebot Smartphone and the second Googlebot Desktop. Google says the last two are rare. Google writes W.X.Y.Z where the Chrome version goes, because it changes: match that part with a wildcard.

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)

How do I block Googlebot?

Google says its common crawlers, Googlebot among them, always obey robots.txt when they crawl automatically. One token covers both versions. Google’s AI features in Search, such as AI Overviews and AI Mode, follow Googlebot’s rules and the nosnippet and noindex controls: Google-Extended does not change them.

Add this to the robots.txt file at the root of your site:

User-agent: Googlebot
Disallow: /

How do I check that a Googlebot visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Google publishes the IP ranges its common crawlers use. A reverse DNS lookup of a genuine request’s IP address gives a host name under googlebot.com, and a forward lookup of that name returns the same address.

Spreelytics runs this check for you. Every crawl that claims to be Googlebot is compared with the IP ranges Google publishes, refreshed every 12 hours, and one from outside them counts as spoofed.

How do I see Googlebot visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “Googlebot”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises Googlebot by its user agent and counts each visit on the Bots page under Indexing, apart from your visitor numbers.

Sources: developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#googlebot, developers.google.com/search/docs/appearance/ai-features, developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests