Most bots visiting this site already know that they are not welcome, but they continue to try to intrude. All bots will be banned unless they are known safe indexers named Duck Duck Go or BingBot and/or can show a positive reason for visiting.
This is addition to IP addresses that blatantly disregard our robots.txt requirements. They, in effect, automatically self-blacklist.
If your bot is banned and you believe that it should not be, contact me with your reasoning and I will consider your case.
As long as I’ve been building and maintaining websites (since 1996) and running Content Management Software websites seriously (since 2001) I’ve noticed higher and higher traffic rates on my sites: lots of traffic from bots and other unusual web auto-surfers that serve no apparent purpose except to use bandwidth for which I pay.
Since bandwidth uses my money, it’s seems illogical for a webrobot to just visit a website without as much as a “By Your Leave”. Most accept this rude behavior. I do not.
I view all bots as thieves or pirates, but some are acceptable since they help people find this blog.
Right now the only bots that I accept are three indexers. They are Bing.com and DuckDuckGo.com*and Google.
My blacklist ban includes all bots especially those that do not have a title or a recognized IP address or most AWS bots that appear have no defined purpose or browser description, I consider them to be malicious.
In addition, all visits reported to be from Russia and China are automatically banned since their countries have such significant reputations for maliciousness.
I hope other site owners and webmasters agree. I am receptive to constructive feedback. Also, I will no longer publish the IP addresses of blocked and blacklisted “visitors”; there’s just too many!
Those blocked by me specifically do not include many other IP addresses that are auto-blocked by SpamCop and Project HoneyPot and other software aids installed that block more than 4000 known malicious bots.
You think the Web is honest? I know a bridge in NYC for sale; interested?
Yes, I am blocking entire ranges of IP addresses that I find to appear to be part of a Botnet or are loosely affiliated with a bot.
I think I am in good company since Project Honeypot has identified 1,039,715 bad web hosts, 865,963 harvesters, 374,796 search engines, 103,962,208 spam servers, 1,266,791 comment spammers, 27,873,066 dictionary attackers and 32,997 Robot.txt rule breakers as of September 23, 2019., from its beginning in 2004.
Did you know that some countries have a poor reputation for hosting spamming and phishing websites? Check https://www.projecthoneypot.org/statistics.php & https://www.netcraft.com/anti-phishing/phishing-site-feed/ to learn more who they are and what can be done.
_______________
* DuckDuckGo is an Internet search engine that doesn’t track you like Google and most others do.