Categories
CMS Mastering Development Wordpress

Site searches by Python for non-existent assets

I’ve recently started paying greater attention to my 404 errors in order to clean up what I can and improve my site’s SEO and ranking, and have noticed something that I don’t understand.

In my 404 error log, I’m seeing quite a few searches conducted by user agents that look like this:

python-requests/2.23.0python-requests 2.23.0

And several that are similar…..but they are all requesting files that don’t exist.

What are the python searches? Are they like bad bots? How do I block or prevent them?

I have a lot of bad bots too, and I found an older (2017) resource with some code to block them by User-Agent in my .htaccess file, which I implemented but it does not seem to be working – I still see logs of those bad bots also requesting mostly non-existent resources as well as a lot of posts with the /email or /print appended….. is there any truly effective way to block bad user-agents?

Leave a Reply

Your email address will not be published. Required fields are marked *