I’ve recently started paying greater attention to my 404 errors in order to clean up what I can and improve my site’s SEO and ranking, and have noticed something that I don’t understand.
In my 404 error log, I’m seeing quite a few searches conducted by user agents that look like this:
And several that are similar…..but they are all requesting files that don’t exist.
What are the python searches? Are they like bad bots? How do I block or prevent them?
I have a lot of bad bots too, and I found an older (2017) resource with some code to block them by User-Agent in my .htaccess file, which I implemented but it does not seem to be working – I still see logs of those bad bots also requesting mostly non-existent resources as well as a lot of posts with the /email or /print appended….. is there any truly effective way to block bad user-agents?