Join Free
+ Reply to Thread
Page 3 of 3 FirstFirst 1 2 3
Results 21 to 24 of 24

Thread: Block Bad Bots

  1. #21

  2. #22
    Senior Member
    Join Date
    Apr 2011
    Blog Entries
    Quote Originally Posted by AcidRaZor View Post
    i've been reading up on iptables and such (I use csf which handles auto-blocking of failed attempts to login and what not) and apparently iptables can scan the incoming headers of a request. I had an issue with DFind and I used:

    iptables -I INPUT -d -p tcp --dport 80 -m string --to 70 --algo bm --string 'GET /' -j DROP

    to block it. I'm sure you can modify it to check for user-agent info?
    I found this. Might be interesting. (reference: ).

    Also, here's a guide on how to download and install mod_security (reference: .

    Chirpy from recommended using mod_security (reference: ) to achieve bad bot blocking. There is even a script that integrates with WHM, available at:
    This is an exclusive! and free! add-on product for cPanel/WHM. The product provides you with an interface to the cPanel mod_security implementation from within WHM.
    With ConfigServer ModSecurity Control you can:
    • Disable mod_security rules that have unique ID numbers on a global, per cPanel user or per hosted domain level
    • Disable mod_security entirely, also on a global, per cPanel user or per hosted domain level
    • Edit files containing mod_security configuration settings in /usr/local/apache/conf
    • View the latest mod_security log entries
    Anyhow, this is what I found on Google during a few minutes of searching. I have a lot of other things on my plate, but I'd be interested to see where this discussion goes.

    Even if a person isn't a fan of cPanel, there are a lot of individuals who still rely on it. A lot of VPS and dedicated server plans offer it, and not many people have achieved a level of knowledge to admin a server without it. If the goal in the end is to reduce the hammering of Prosperent's API, it makes sense to collaborate in a positive manner.

    "The greatest limitation in coding is imagination."
    -- Amazon Browse Node Database: .
    -- Create a Free Publisher Account

  3. #23
    After implementing this for a few days I thought I would post a short list of actual bots that have hit my sites and now get blocked...

    RewriteCond %{HTTP_USER_AGENT} Baiduspider [OR]
    RewriteCond %{HTTP_USER_AGENT} DotBot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Java [OR]
    RewriteCond %{HTTP_USER_AGENT} libcurl [OR]
    RewriteCond %{HTTP_USER_AGENT} ^magpie-crawler [OR]
    RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
    RewriteCond %{HTTP_USER_AGENT} ^PostRank [OR]
    RewriteCond %{HTTP_USER_AGENT} sindice [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Sogou [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Sosospider [OR]
    RewriteCond %{HTTP_USER_AGENT} YandexBot [OR]
    RewriteCond %{HTTP_USER_AGENT} YodaoBot [OR]

  4. #24

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
coupons | coupons and deals