Software Development:
Individual Programming

Symptom

My website is frequently being visited by undesireable bots. The access_log shows that sources like Majestic12, HTTrack, Semrush and more request the same files over and over again.

Solution

The requesting services are “bad bots”, meaning programs and website systems that check and analyze other websites. Visits of such bots are mostly useless for website owners. They only cause CPU load and decrease the speed of your server. A special software on the Bitpalace hosts recognizes many of these bots and automatically blocks them. But you can also add security to your site by adding these lines to the beginning of your .htaccess file:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} (seobility|PetalBot|UptimeRobot|seocompany|LieBaoFast|SEOkicks| Uptimebot|Cliqzbot|ssearch_bot|domaincrawler|AhrefsBot|spot|DigExt|Sogou| MegaIndex.ru|majestic12|80legs|SISTRIX|HTTrack|Semrush|MJ12|MJ12bot|MJ12Bot|Ezooms|CCBot|TalkTalk|Ahrefs|BLEXBot) [NC]
RewriteRule .* - [F]

This will block all frequently found bad bots from visiting your website.

Please note that the line containing the Bad Bot names (SEOkicks|Uptimebot|...) must be written in a single line without spaces. Copies from this article may contain space characters due to line breaks. Remove these spaces before saving.

Overview of frequently asked questions