Amazonbot

What user agent does Amazonbot use?

Amazon publishes this user agent string for Amazonbot.

Amazon writes W.X.Y.Z where the Chrome version goes, because it changes.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

How do I block Amazonbot?

Amazon says its crawlers follow robots.txt allow and disallow rules but not Crawl-delay, that each of its bots is controlled separately, and that a change can take about 24 hours. Each subdomain needs its own robots.txt. Amazonbot also honours the noarchive meta tag, which Amazon reads as do not use the page for model training.

Add this to the robots.txt file at the root of your site:

User-agent: Amazonbot
Disallow: /

How do I check that a Amazonbot visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Amazon publishes a separate list of IP addresses for each of its bots, this one for Amazonbot.

How do I see Amazonbot visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “Amazonbot”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises Amazonbot by its user agent and counts each visit on the Bots page under Training, apart from your visitor numbers.

Sources: developer.amazon.com/amazonbot