Baiduspider

What user agent does Baiduspider use?

Baidu publishes these user agent strings for Baiduspider.

The first is the desktop agent, the second the mobile one.

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

How do I block Baiduspider?

Baidu says it strictly follows robots.txt and that a change usually takes about 48 hours. A Baiduspider rule covers all of Baidu’s product crawlers, which also have tokens of their own such as Baiduspider-image. Baiduspider-cpro and Baiduspider-ads ignore robots.txt, because they work under customer agreements.

Add this to the robots.txt file at the root of your site:

User-agent: Baiduspider
Disallow: /

How do I check that a Baiduspider visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Baidu publishes no IP list. A reverse DNS lookup of a genuine request’s IP address gives a host name ending in baidu.com or baidu.jp, such as baiduspider-123-125-66-120.crawl.baidu.com, and a forward lookup of that name returns the same address.

How do I see Baiduspider visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “Baiduspider”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises Baiduspider by its user agent and counts each visit on the Bots page under Indexing, apart from your visitor numbers.

Sources: help.baidu.com/question?prod_id=99&class=476&id=2996, help.baidu.com/question?prod_id=99&class=0&id=3001