msnbot

What user agent does msnbot use?

Bing’s current documentation publishes no msnbot user agent string. Its 2010 announcement names the last version, msnbot 2.0b.

How do I block msnbot?

Bing says Bingbot still honours robots.txt rules written for msnbot, so a group that blocks msnbot also blocks Bingbot, unless Bingbot has a group of its own.

Add this to the robots.txt file at the root of your site:

User-agent: msnbot
Disallow: /

How do I check that a msnbot visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Bing’s Verify Bingbot tool checks requests that call themselves Bingbot or msnbot against Bingbot’s published IP addresses. A genuine request reverse-resolves to a host name ending in search.msn.com, such as msnbot-157-55-33-18.search.msn.com.

Spreelytics runs this check for you. Every crawl that claims to be msnbot is compared with the IP ranges Microsoft publishes, refreshed every 12 hours, and one from outside them counts as spoofed.

How do I see msnbot visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “msnbot”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises msnbot by its user agent and counts each visit on the Bots page under Indexing, apart from your visitor numbers.

Sources: blogs.bing.com/webmaster/2010/6/Bing-crawler-bingbot-on-the-horizon/, blogs.bing.com/webmaster/2012/5/To-crawl-or-not-to-crawl,-that-is-BingBot-s-questi/, www.bing.com/webmasters/help/verify-bingbot-2195837f