Meta-WebIndexer

What user agent does Meta-WebIndexer use?

Meta publishes these user agent strings for Meta-WebIndexer.

Meta says the user agent in your logs will be similar to these.

meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-webindexer/1.1

How do I block Meta-WebIndexer?

Meta says to block its crawlers by disallowing them in robots.txt, and that they may keep a cached copy of the file for up to 24 hours.

Add this to the robots.txt file at the root of your site:

User-agent: Meta-WebIndexer
Disallow: /

How do I check that a Meta-WebIndexer visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

Meta does not publish IP ranges or a DNS check for Meta-WebIndexer, so there is no way to confirm a request really came from it.

How do I see Meta-WebIndexer visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “Meta-WebIndexer”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises Meta-WebIndexer by its user agent and counts each visit on the Bots page under Indexing, apart from your visitor numbers.

Sources: developers.facebook.com/documentation/sharing/webmasters/web-crawlers