Referrer spam

Referer spam is a kind of search engine-targeted spam. The technique involves making repeated web site requests using a fake referer url pointing to a spam-advertised site. Sites that publicize their referer statistics will then also link to the spammer's site. This benefits the spammer because of the free link, and also gives the spammer's site improved search engine link placement due to link-counting algorithms that search engines use.

As with email spam, web site operators who receive unwanted referer spam may respond using filtering. Some web sites receive so many referer spam hits that they amount to a denial of service attack on the server because there are not enough resources left on the server to handle legitimate traffic.

An example configuration fragment for filtering using the Apache server is as follows:

 RewriteEngine On
 #a known spamming site
 RewriteCond %{HTTP_REFERER} ^http://(.*\.)?egolddomain.(com|net)(/.*)?$ [NC,OR]
 #various "porno" words when separated by line breaks are probably spam
 RewriteCond %{HTTP_REFERER} ^http://.*(\b)porn(o(graph(y|er))?)?(\b).* [NC,OR]
 #referer url's with excessive - characters are probably spammers
 RewriteCond %{HTTP_REFERER} ^(http://www.)[a-z]+-[a-z]+- [NC,OR]
 #can add as many rules as desired following the pattern of the previous line
 #set an environment variable "BAD_GUY" so we can send their logs to a different file
 RewriteRule ^(.*) %{HTTP_REFERER} [R=301,E=BAD_GUY:1,L]
 #because we have set E=BAD_GUY above, we can do this in our log file:
 CustomLog /var/log/apache/access.log combined env=!BAD_GUY
 CustomLog /var/log/apache/access_bad.log combined env=BAD_GUY

The "fake" web site hits will go to access_bad.log, whereas normal traffic goes to access.log. The "RewriteCond" lines contain Regular expressions that can be used to match any undesirable traffic, thereby excluding it.

Another Solution If most of the spam is coming from a few IP addresses, or is requesting a certain page (that may no longer exist on the server) the Apache server may also be configured to deny access via the configuration file, (often named httpd.conf), based on either IP address or the name of the requested file by adding lines like the following:

 #Deny access based on the filename or path of the requested file
 <Location /links>
  Deny from all
 </Location>
 #
 #Deny access based on the IP address or host name of the offending site
 <Directory /usr/local/etc/httpd/htdocs>
   Deny from 72.36.244.166
   Deny from big-bad-spammer-blah-blah.com
 </Directory>

A good statistics analysis program will allow you to target the worst offenders.

A Third Solution is to install the ModSecurity module for Apache which allows you to deny requests based on any variable from the server environment, such as referer, request, IP, host, etc.

External links

• A script for filtering referer spam with Apache web server
• ReferrerCop - A searchable database of known referrer spam with tools for filtering referrer spam from Apache log files and AWStats data files.