Bytespider

What user agent does Bytespider use?

ByteDance publishes this user agent string for Bytespider.

ByteDance gave this as an example in a 2019 Toutiao Search article, and has published no newer string.

Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML,like Gecko)Chrome/41.0.6633.1032 Mobile Safari/537.36;Bytespider;https://zhanzhang.toutiao.com

How do I block Bytespider?

ByteDance’s AI training summaries say Bytespider is designed to respect robots.txt and to avoid overloading websites. ByteDance publishes no robots.txt example for it.

Add this to the robots.txt file at the root of your site:

User-agent: Bytespider
Disallow: /

How do I check that a Bytespider visit is real?

Anyone can put a crawler’s name in a user agent, so the name alone proves nothing.

ByteDance publishes no current IP list or DNS check for Bytespider. The only addresses it has given are in the 2019 Toutiao Search article, so a request cannot be reliably confirmed.

How do I see Bytespider visits on my site?

Your server’s access logs record the user agent of each request, so you can search them for “Bytespider”.

Or add the Spreelytics server middleware, @spreelytics/ai-crawl. It recognises Bytespider by its user agent and counts each visit on the Bots page under Training, apart from your visitor numbers.

Sources: seed.bytedance.com/transparency, www.toutiao.com/article/6759451726496940557/, zhanzhang.toutiao.com/