CSF Installation Problem: Solve Blocking Google Bot

September 21, 2011 / Web Hosting

“CSF” and “Mod security” both are very important for your server. If any user wants to install them then they might see it blocking Google bot for some specific rules.

Here is what you should be doing to make Google Bots crawl through your sites again.

You have to restart Apache after running the below rules to mod_security configuration file :

# GoogleBot by user-agent…
SecRule HTTP_USER_AGENT “Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot” nolog,allow
SecRule HTTP_USER_AGENT “GoogleBot” nolog,allow
SecRule HTTP_USER_AGENT “googlebot” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image” nolog,allow
##
SecRule HTTP_USER_AGENT “AdsBot-Google” nolog,allow
SecRule HTTP_USER_AGENT “Googlebot-Image/1.0″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Googlebot/Test” nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google/2.1″ nolog,allow
SecRule HTTP_USER_AGENT “Mediapartners-Google*” nolog,allow
SecRule HTTP_USER_AGENT “msnbot” nolog,allow

Leave a Reply

Your email address will not be published. Required fields are marked *